What Is VRAM (and Why Does a Model Not Fit)?

Quick answer
VRAM is the GPU’s own memory pool, separate from system RAM. It holds the model’s weights and everything the model is working on.
Capacity decides whether a model runs at all; bandwidth decides how fast it generates.
Rough weight arithmetic: parameters × bytes per parameter. A 7B model needs about 14 GB in 16-bit precision, or around 4 GB quantized to 4 bits.
Two things above the weights matter: the key-value cache that grows with context length and concurrency, and the activations for the batch you are processing.
Apple Silicon and integrated graphics share system memory instead of having a separate pool, which trades bandwidth for a much larger ceiling.

What is VRAM?

VRAM (video RAM) is the memory attached to the graphics processor — a separate pool from the system RAM the CPU uses. It holds the data the GPU is working on: for a game, the textures and frame buffers; for an AI model, the model’s parameters plus everything the current computation needs. It is soldered to the card and cannot be upgraded, so the amount you buy is the amount you live with. When people say a GPU has 8 GB, this is the number they mean.

The reason it is separate rather than shared is bandwidth. A GPU needs to feed thousands of cores at once, so its memory is physically much closer and connected by a far wider bus than a memory slot on a motherboard allows. That is why video memory is expensive and why the distinction between capacity and bandwidth, which seems academic on a CPU, has direct consequences here: capacity sets the ceiling on the models you can load, and bandwidth sets how quickly each token comes out.

Bandwidth is the number that decides speed

Generating text with a large language model is memory-bound rather than compute-bound. Producing each token requires reading essentially the entire model once — every parameter participates in the calculation — so the ceiling on tokens per second is roughly the memory bandwidth divided by the size of the model. That single relationship explains a lot of confusing benchmark results: a card with 5% more compute but the same memory speed will not generate noticeably faster, while a card with faster memory will, even if its compute figure is lower.

This is why data-centre accelerators use HBM (high bandwidth memory), a technology that stacks memory dies and wires them to the chip across an extremely wide interface, reaching bandwidth an order of magnitude above desktop memory. It is also why the practical question about a GPU is rarely its teraflops. When you compare two machines that can host a model, bandwidth predicts the tokens per second far more reliably than any compute specification on the box.

Why a model does not fit

The arithmetic is simple enough to do in your head, which is why it is worth doing before buying anything. Each parameter is a number, and the precision you store it in decides how many bytes: 16-bit (half precision) is 2 bytes per parameter, 8-bit is 1, and 4-bit is half a byte. So a 7-billion-parameter model needs roughly 14 GB at 16-bit, 7 GB at 8-bit, and about 4 GB at 4-bit — before any overhead. A 70B model at 16-bit needs around 140 GB, which is why it is normally quantized and often split across several cards.

Then add the parts that catch people out. The key-value cache stores attention state for the conversation so the model does not recompute it each token; it grows with context length and with how many requests you handle at once, and for long conversations it can rival the weights. Activations for the current batch take more room. Framework overhead and fragmentation take some too. A model that squeezes into a card at exactly its weight size will fail the moment you give it a long prompt or a second concurrent user, which is the usual story behind ‘it loaded, then crashed’.

# Rough VRAM needed for the weights alone
params = 7e9
for bits, label in [(16, 'fp16'), (8, 'int8'), (4, 'int4')]:
    print(label, round(params * bits / 8 / 1e9, 1), 'GB')
# fp16 14.0 GB, int8 7.0 GB, int4 3.5 GB - plus cache and activations

Unified memory and shared graphics

Not every machine has a separate pool. Apple Silicon puts CPU and GPU on the same package sharing one pool of unified memory, so an application can address far more than a typical consumer card offers — which is why a laptop with 64 GB of unified memory can load models that would need a very expensive discrete GPU. Integrated graphics on ordinary laptops work the same way, borrowing from system RAM.

The trade is bandwidth and contention. Unified memory is much slower than a discrete card’s dedicated pool, so token generation is slower even when the model fits comfortably; and because the pool is shared, a model that fills it leaves nothing for the rest of the machine, which can make the whole system sluggish. This is why the same model can be usable on a laptop with less total memory than a discrete card that cannot load it at all.

How to make a model fit

Five levers, in the order worth trying. Quantization first: running in 4 or 8 bits cuts memory to a half or a quarter, with a quality cost that is real but often acceptable, and it is the single most effective change. Shorter context next: the key-value cache grows with prompt length, so limiting how much history you keep reduces both memory and cost. Smaller batches and a single concurrent request reduce activation memory. Offloading moves some layers to the CPU, which lets a model run that would not otherwise load but at a substantial speed penalty. Multiple GPUs are the last resort, since splitting a model across cards works but adds complexity and does not multiply speed the way people hope.

If none of those gets you there, the model is the wrong size for the hardware, and the two honest options are a smaller model or an API. Choosing a smaller model is often correct anyway: the difference in capability between a large and a small model is frequently smaller than the difference between a well-designed prompt and a careless one, and a small model running locally costs nothing per token. The decision is a memory calculation followed by a quality judgement, not a hardware purchase.

Where this bites vibecoders
The failure mode is always the same. An assistant recommends a current small model, you buy or rent a GPU that seems generous, and the process dies with a memory error during load or — worse — after twenty messages when the cache has grown. Nothing in the generated code mentions VRAM, because the model is working from what is technically possible rather than what fits on your machine. Doing the parameters-times-bytes calculation, then adding room for the cache and concurrency, takes two minutes and prevents an afternoon of confusing OOM errors.

Where AI coding assistants get this wrong

Choosing a model without computing its memory requirement in bytes.
Forgetting the key-value cache, which grows with context length and concurrent requests.
Loading a model at 16-bit precision on hardware that could only host the quantized version.
Assuming more compute on the same memory bandwidth will generate tokens faster.
Splitting a model across GPUs and expecting throughput to double.
Filling unified memory completely and making the whole system unusable.

Checklist

Compute weight memory as parameters times bytes per parameter.
Add headroom for the key-value cache, activations, and framework overhead.
Decide on quantization before choosing hardware.
Cap context length, since it is a memory cost as well as a token cost.
Check bandwidth, not just capacity, when predicting tokens per second.
Leave room for concurrent requests if more than one person will use it.
Fall back to a smaller model or an API when the arithmetic does not fit.

FAQ

How much VRAM do I need to run a 7B model?

About 14 GB at full 16-bit precision, roughly 7 GB at 8-bit, and around 4 GB at 4-bit, before adding the key-value cache and activations. In practice a 4-bit quantization on a card with 8 GB runs a 7B model comfortably for single requests, while a 16-bit version needs a card with 16 GB or more.

Does more VRAM make a model faster?

Not directly. Capacity determines whether the model loads; speed during generation is governed mostly by memory bandwidth and by how much of the model has to be recomputed. More capacity only makes things faster indirectly, by letting you keep a longer context in memory instead of offloading or discarding it.

Why does it fit alone but fail with two users?

Because each concurrent request needs its own key-value cache and activations. A model using most of the available memory for weights leaves no room for the second request, so the runtime either queues it, offloads, or fails. Size for the concurrency you expect, not for a single user.