The VRAM Cliff
Every local-LLM buyer hears it within a week: "make sure the model fits in VRAM." Here is the mechanism behind that advice and how to plan around it.
As of 2026-10-07. The sizing figures below are planning heuristics, not per-model measurements; measured VRAM, engine and context live on the board (see Sources).
What the cliff is
A GPU can only run a model at full speed while every weight it needs sits in its own memory (VRAM). GPU memory bandwidth is on the order of hundreds of GB/s to TB/s; system RAM bandwidth is roughly an order of magnitude lower, and CPU compute is far weaker still.
When a model fits entirely in VRAM, every token generation step streams weights from VRAM — fast enough to feel instant.
When it does not fit, the inference engine falls back to one of two things:
- Partial offload: some layers live in system RAM and are copied to the GPU per pass. Each such transfer runs at system-memory speed — the classic 10x-class throughput drop.
- Full CPU inference: everything runs on the CPU and RAM, which is fine for small models and terrible for large ones.
That discontinuity — tiny VRAM shortfall, huge speed loss — is the VRAM cliff. Models do not degrade gracefully as they outgrow a card; they fall off it.
Why it matters more than raw GPU speed
A $2,000-MSRP flagship (32GB) and a pair of used 24GB cards (48GB total) are different tools because of where the cliff sits:
- 27B Q4 (~21–24GB with context): fits a 24GB card — great.
- 70B Q4 (~40–48GB with context): does not fit 32GB; does fit 48GB.
So a 48GB rig can run models a 32GB flagship simply cannot run at GPU speed — which is exactly why the used 24GB and 48GB market is the center of gravity of 2026's local-AI hardware economy.
Planning math (rough)
For Q4 quantization, plan on roughly 0.7–0.8 GB of VRAM per billion parameters, then add context overhead (KV cache grows with context length; long-context work can add several GB). Examples:
| Target model class | Q4 size approx. | Comfortable VRAM |
|---|---|---|
| 7B–9B | 6–8 GB | 12 GB (lots of context headroom) |
| 13B–14B | 10–12 GB | 16–24 GB |
| 27B–32B | 21–26 GB | 24 GB (32 GB for long context) |
| 70B | 40–48 GB | 48 GB (dual 24GB or one 48GB) |
Quantization choice moves the cliff: lower quant (Q3/Q4) shrinks weights but hurts quality; higher quant (Q6/Q8) improves quality but pushes the model off the cliff — run your own quality check at your target context before trusting either direction.
Practical rules
- Size the rig for the biggest model you will run in 12 months, not the model you are playing with this week.
- Buy VRAM headroom for context: long-context agents eat GBs of KV cache on top of weights.
- Beware "it technically runs": engines will run anything with offload — watch tokens/s, not "did it load".
- Benchmark on your own workload: our benchmark methodology pages show what to record (engine, quant, context, tokens/s) so comparisons are honest.
FAQ
What is the "VRAM cliff" exactly?
The sharp performance collapse that happens when a model's weights plus KV cache stop fitting fully in VRAM and spill to system RAM. Throughput can drop 10x or more instead of degrading linearly — which is why two "similar" GPUs can feel completely different on the same model.
My model loads — why is it so slow?
"It loaded" usually means the engine offloaded layers to CPU RAM. The moment weights spill, decode speed collapses. Check the GPU/CPU layer split the engine prints at startup, and measure tok/s on your own workload rather than trusting "it runs".
How much VRAM do I need beyond the weights?
Plan on 20–30% over the quantized weight size for KV cache and context. A 40 GB Q4 model wants ~48 GB of VRAM to run comfortably — that gap is why 48 GB cards dominate "what can I run" decisions, and why long-context workloads tip borderline GPUs over the cliff.
Is partial offloading ever acceptable?
As a last resort, yes — a few spilled layers is slower than it feels "worth". Plan for a full fit at your target context, and treat partial offload as emergency behavior, not a setup you live with.
Sources
- Q4 sizing heuristic (0.7–0.8 GB per billion parameters, 20–30% KV overhead): DiggTop benchmark methodology — a planning approximation, not a per-model measurement.
- Per-model measured VRAM, engine, quant and context: benchmark database · compatibility matrix.
- Current market picks by VRAM tier: local LLM GPU buying guide.
Figures follow the standard Q4 sizing heuristic. Per-model measured numbers live in the benchmark database — see the buying guide for current market picks.