GPU vs CPU for AI: When Is a GPU Actually Worth It?

Quick answer
Training and fine-tuning are GPU work. Inference — running a model someone else trained — is a spectrum where CPUs are often adequate.
The decision is a memory calculation first: if the model does not fit in VRAM it does not run, however much compute you have.
For occasional use, an API call to a hosted model is cheaper than any hardware, because you pay per token instead of per month.
If you need a GPU continuously, renting by the hour beats owning; owning only wins at sustained utilisation.
A CPU is fine for small quantized models, batch jobs that are not latency-sensitive, and anything where the model is a small part of the request.

Split the problem before comparing hardware

There are three different jobs, and they have different requirements. Training builds a model from data and is the most demanding: it needs to hold the model, its gradients, and the optimiser state in memory — several times the model’s own size — and run for hours or days. Fine-tuning adapts an existing model and is lighter, especially with parameter-efficient methods that only update a small fraction of the weights. Inference runs a model that already exists, needs the weights plus some working space, and lasts as long as the request.

Most people asking ‘do I need a GPU for AI’ mean inference, and the honest answer there is often no. Training and fine-tuning, on the other hand, are GPU work in any realistic sense: on a CPU the same job takes so long that it stops being practical. Sorting out which of the three you are actually doing removes most of the confusion from the hardware discussion before any spec sheet is opened.

The memory calculation comes before the speed comparison

A GPU either can or cannot host a model, and that binary comes first. Weights take roughly parameters times bytes per parameter, so a 7-billion-parameter model is around 14 GB in 16-bit precision and around 4 GB quantized to 4 bits, with additional room needed for the key-value cache and activations. If the arithmetic does not fit in the card’s memory, the options are a smaller model, a lower precision, or an API — not a faster card.

Once it fits, the speed question is mostly about memory bandwidth, because generating text requires reading the whole model for every token. That is why two GPUs with similar compute figures can differ substantially in tokens per second, and why a CPU that seems to have plenty of gigahertz produces tokens slowly: its memory bandwidth is a fraction of a discrete card’s. In practice, a CPU with enough system memory can run a small quantized model at a few tokens per second, which is fine for a background task and painful for a chat interface.

What each option actually costs

Compare four options rather than two. An API provider charges per token — no capital cost, no operations, someone else’s scaling problem, and per-call prices that keep falling. A rented cloud GPU bills by the hour, often from around a dollar or two per hour for a mid-range accelerator and more for the newest parts, which is trivial for a few hours of experimentation and expensive if it runs all month. Owned hardware has a large upfront cost and a resale value that drops quickly. And a CPU-only instance is the cheapest of all if the workload can tolerate it.

The break-even is unforgiving. A card costing a few thousand dollars is equivalent to roughly a thousand hours of rented access — some weeks of continuous use — and the rented one is upgraded out from under you for free. Unless you are running the hardware most of the day, every day, the arithmetic favours renting or an API. There is also an operational cost that never appears in the comparison: drivers, CUDA versions, thermal issues, and a machine that is idle most of the time still drawing power.

# Break-even: hours of rented GPU before owning becomes cheaper
card_price = 2500      # mid-range accelerator, roughly
rental_per_hour = 1.5
print(round(card_price / rental_per_hour), 'hours of rental')   # ~1667 hours
print(round(card_price / rental_per_hour / 24), 'days of continuous use')

When a CPU is genuinely enough

Four cases where skipping the GPU is the right engineering decision. First, small models: a quantized few-billion-parameter model on a modern multi-core CPU produces usable output for summarisation, classification, and extraction, just not quickly. Second, batch work that is not interactive: if a nightly job can spend an hour processing a queue, the latency of CPU inference does not matter. Third, workloads where the model is a small fraction of the request — embedding a document, calling a model once per submitted form — so the surrounding code dominates. Fourth, and most commonly, anything that can call a hosted API.

The signal that you have outgrown the CPU is concrete rather than aspirational: requests queue behind each other, users notice the delay, or the same job runs all day. At that point you are no longer guessing about demand and can compare renting against an API with real numbers. Buying hardware to discover whether you have demand is the expensive way to find out.

Where this bites vibecoders
The default plan an assistant produces for ‘add AI to my app’ is to load a model locally, which quietly implies a GPU. The plan is usually right in principle and wrong in cost: it turns a project with no capital budget into one that needs a machine with enough memory, plus the maintenance of drivers and versions. The alternative — one API key and a per-token bill — costs nothing until people use it. Asking ‘what does this cost to run for a hundred users a day?’ before accepting the generated architecture is the whole of the decision.

Where AI coding assistants get this wrong

Proposing local inference as the default when an API would be simpler and cheaper.
Recommending a GPU without checking that the model’s weights fit in its memory.
Confusing fine-tuning with inference and applying the heavier requirements to both.
Choosing hardware before measuring whether there is any demand to serve.
Ignoring the ongoing cost of running an always-on accelerator.
Assuming CPU inference is unusable, when it is adequate for small quantized models.

Checklist

Decide whether the task is training, fine-tuning, or inference.
Compute the model’s memory requirement in bytes before anything else.
Compare API pricing against rented GPU hours for your expected volume.
Calculate the break-even hours before buying hardware.
Check whether the workload is latency-sensitive or merely throughput-sensitive.
Re-evaluate once real usage exists, not before.

FAQ

Can I fine-tune a model without a GPU?

For very small models and very small datasets, technically yes, but it is slow enough to be impractical. Parameter-efficient fine-tuning reduces memory substantially and is the usual way to fine-tune on modest hardware. In practice fine-tuning is the clearest case where renting a GPU for a few hours is worth more than any alternative.

Is it cheaper to own a GPU than to rent one?

Only at high sustained utilisation. A mid-range card costs roughly a thousand hours of rented access, and ownership adds power, cooling, driver maintenance, and rapid depreciation. If you are not using it most days, renting or calling an API costs less and gives you better hardware when you need it.

How slow is CPU inference really?

For a small quantized model on a modern multi-core processor, expect a few tokens per second — several times slower than a mid-range GPU, but usable for background work and non-interactive tasks. The gap widens with model size, because memory bandwidth becomes the limiting factor.