What Is a GPU (and Why Is It Different From a CPU)?

Quick answer
A GPU is a chip built for throughput: thousands of simple cores running the same operation on a large block of numbers simultaneously.
A CPU is built for latency: a handful of complex cores that each finish one difficult task quickly, including unpredictable branches.
AI models are almost entirely matrix multiplication, which is the ideal shape for a GPU because every output value can be computed independently.
Tensor cores are specialised units inside modern GPUs that do these small matrix operations in one step instead of many.
The GPU’s own memory (VRAM) is usually the limit, not its compute: a model that does not fit will not run, however fast the chip.

What is a GPU?

A graphics processing unit is a processor designed around a different bet than a CPU. A CPU is optimised to complete a single stream of varied work as quickly as possible, with large caches and clever circuitry that guesses which branch the code will take next. A GPU is optimised to apply the same simple operation to an enormous number of values at once — originally to colour millions of pixels, which is embarrassingly parallel work. It achieves this with thousands of small cores, each individually far weaker than a CPU core, sharing a wide memory system and a scheduler that likes nothing better than uniform work.

The consequence is that a GPU is not a faster CPU, it is a differently shaped one. It cannot run your web server or your build script well, because those are full of branches and dependencies where each step depends on the last. It is outstanding at the one pattern AI is made of: taking two arrays of numbers and combining them into a third, over and over, at a scale where the parallelism never runs out.

Throughput versus latency, and why branches are fatal

The cleanest way to hold the difference is: a CPU maximises the speed of one task, a GPU maximises the number of tasks finished per second. A CPU core with a deep pipeline and a big cache might finish a complex operation in nanoseconds while a GPU thread takes many times longer — but the GPU runs tens of thousands of them concurrently. If you have one calculation, the CPU wins. If you have a hundred million independent calculations, the GPU finishes while the CPU is still getting started.

This also explains the GPU’s weakness. When a block of threads hits a branch, the hardware runs both sides of it in sequence with the inactive threads masked off, so code full of conditionals wastes most of the machine. Memory access has the same character: threads that read from scattered addresses serialise, while threads that read a contiguous block are served at full speed. GPUs therefore reward code that is uniform, regular, and predictable, which is a precise description of a matrix multiplication and a poor description of business logic.

Why AI is a GPU workload

A transformer model, in the end, is a long sequence of matrix multiplications with a few simple element-wise operations in between. Multiplying a matrix by another matrix means computing each output value independently from a row of one and a column of the other — there is no dependency between results, so the work can be spread across every core the hardware has. This is the property that made GPUs the default substrate for deep learning, and it is why the field’s cost curve is tied to graphics hardware rather than general-purpose processors.

Modern AI accelerators add specialised units for exactly this pattern. Tensor cores, as NVIDIA calls them, execute a small matrix multiply-and-accumulate in a single instruction rather than a sequence of individual multiplies, which is a large part of why each GPU generation moves the same workload faster. The industry also runs these operations in lower precision — 16-bit, 8-bit, and 4-bit numbers instead of the 32-bit floats of scientific computing — trading a little accuracy for much higher throughput and much less memory use. That trade is the basis of almost every technique for making large models affordable.

# What GPU is actually installed, and how much of it is in use
nvidia-smi

# Live utilisation and memory, once per second
nvidia-smi --query-gpu=name,utilization.gpu,memory.used,memory.total --format=csv -l 1

The memory is usually the real limit

Buying a GPU is largely a memory decision. The chip sits next to its own high-bandwidth memory — GDDR for consumer cards, HBM for data-centre parts — which is far faster than system RAM because it is closer and wired much more widely. It is also expensive and soldered on, so you cannot add more later. Every model’s weights, its intermediate activations, and its cache of conversation state must all fit in that fixed pool at the same time, and if they do not, the model does not run at all. Optimisations like quantization exist mostly to shrink the memory footprint.

For AI work the practical question is therefore never ‘how fast is this GPU’ in isolation, but ‘how much VRAM does this model need, and does the card have that much’. A modest GPU with enough memory will run a model comfortably while a much faster card with less will fail outright. The VRAM guide works through the arithmetic; the short version is that capacity decides whether the job is possible and bandwidth and compute decide how quickly it finishes.

Do you need one?

Three honest cases. Training a model, fine-tuning one, or serving a model at scale: yes, and there is no realistic alternative. Running a small model occasionally for a personal project: often no — a modern CPU can generate text slowly, and an API call to a hosted model is usually cheaper than any hardware you would buy. Working on ordinary application code that happens to call a model’s API: no, and the GPU lives on the provider’s side of that boundary, not yours.

The economics point one way for most small projects. Cloud GPUs are billed by the hour, so a few hours of experimentation costs a few dollars, while the card itself costs as much as a used car and depreciates. Unless you are using it continuously, renting beats owning — and if you are using it continuously, that is a strong signal you have a real workload and should be comparing prices carefully. For everything else, the API is the answer, and the hardware becomes somebody else’s problem.

Where this bites vibecoders
The GPU is where AI builders most often spend money they did not need to. Asked to ‘add AI’, an assistant may propose a local model, which means a GPU with enough memory, which means a machine far more expensive than the API call would have been. The reasoning in the generated plan is rarely stated, so the constraint — 14 GB of weights for a 7B model in half precision, plus cache — never appears until the model fails to load. Knowing that the requirement is a memory figure rather than a speed figure is what turns ‘I need a GPU’ into ‘I need 8 GB of VRAM’ or, more often, ‘I need an API key’.

Where AI coding assistants get this wrong

Recommending a local model without checking whether the target machine has a GPU or enough VRAM.
Choosing a GPU by compute speed when capacity is what determines whether the model loads.
Assuming a GPU speeds up the whole application rather than only the tensor operations.
Running inference in 32-bit precision and then wondering why memory is exhausted.
Buying hardware for a workload that an hourly cloud instance or an API would serve more cheaply.
Expecting CPU-bound application code to benefit from a GPU at all.

Checklist

Work out the minimum memory the model needs before choosing any hardware.
Confirm whether the task is training, fine-tuning, or inference — the requirements differ enormously.
Compare the cost of a cloud GPU hour and an API against the price of owning the card.
Check the runtime actually supports the card and its architecture.
Decide the precision you will run at, since it changes memory by up to 4x.
Measure utilisation: an idle GPU is money burning.

FAQ

Why are GPUs better than CPUs for AI?

Because AI models are mostly matrix multiplications, and every output value of a matrix multiply can be computed independently. That lets thousands of simple GPU cores work in parallel, while a CPU’s few complex cores would process the same work a step at a time. CPUs are better at sequential logic with branches, which is what ordinary application code is made of.

Can I run AI models on a CPU?

Yes, and for small models it works. Generation will be slow — often several times slower than a mid-range GPU — but a quantized small model can produce usable output on a modern laptop CPU. It is a reasonable choice for occasional use and a poor one for anything interactive or high-volume.

What are tensor cores?

Specialised units inside modern GPUs that perform a small matrix multiply-and-accumulate in a single instruction instead of a sequence of individual operations. They are much faster than the general-purpose cores for exactly the operation neural networks are built from, and they are why the same chip performs so differently on AI workloads than on other kinds of maths.