VRAM & Memory Limits
Why memory, not compute, decides what you can run — and what to do about it.
Local inference is memory-bound: model weights, the KV cache and the runtime all live in VRAM, and crossing the capacity line changes throughput by an order of magnitude. These pieces cover the arithmetic, the tiers worth buying, and the offload strategies that keep a model running when it does not fit.
The VRAM Cliff: Why It Gets 10x Slower
The single most important concept in local LLM hardware: what happens when weights spill out of VRAM, and how to plan around it.
- Sep 7, 2026 · 6 min readMoE Expert Cache: VRAM Stops Being the CeilingExpert caching moves cold MoE experts to system RAM — FreeToken paper plus llama.cpp RFC #24528. Community: 90+ tps on a 16GB 5070 Ti. As of 2026-09-07.
- Sep 11, 2026 · 9 min readMoE Expert Offload: Same Weights, 26× ApartSame weights, same box: 3.3 tok/s with whole-layer expert offload vs 150 with row-selective UVA reads, and what decides big-MoE speed on consumer GPUs.
- Sep 10, 2026 · 4 min readWhat $1,000 buys for local LLMs (Sep 2026)What $500-$1,000 buys in VRAM for local LLMs, from the 2026-09-28 price snapshot: 12GB, 16GB, 24GB and now 32GB.
- Sep 4, 2026 · 8 min readLocal LLM GPU Buying Guide (2026)What to buy to run local LLMs in September 2026, chosen by VRAM tier with current US prices, used-market alternatives and honest tradeoffs.
- Sep 17, 2026 · 10 min readX local-model bench digest — Sep 17, 2026Local-LLM throughput measured on X: 30 posts and 68 runs, with hardware, quantization, engine, context and source links.
- Sep 5, 2026 · 5 min readRTX 5090 vs RTX 3090 for Local LLMsRTX 5090 vs used RTX 3090 for running LLMs locally: community tok/s data, VRAM math, street prices and which card actually wins.
- Sep 4, 2026 · 3 min readllama.cpp Optimization Cheat SheetThe settings that actually move llama.cpp throughput: quant choice, offload, flash attention, context, batch — with the flags and the reasoning.
- Oct 8, 2026 · 10 min readX local-model bench digest — Oct 8, 2026Local-LLM throughput measured on X: 25 posts and 69 runs, with hardware, quantization, engine, context and source links.
- Sep 5, 2026 · 3 min readThe 48GB VRAM Local LLM BuildHow to get 48GB of VRAM for 70B-class local LLMs: dual RTX 3090 vs used A6000 48GB — power, cooling, PCIe and real community tok/s.
- Sep 5, 2026 · 3 min readBest GPU for Qwen3: VRAM Tiers per ModelWhich GPU runs which Qwen3 model — 0.5B to 180B mapped to VRAM tiers, with real community tok/s from 143 sourced DiggTop benchmark runs.
- Sep 5, 2026 · 4 min readQuantization for Local LLMs: Q4_K_M vs NVFP4How quantization cuts VRAM for local LLM inference: GGUF K-quants, IQ quants and NVFP4 compared with community data from real GPU runs.
- Oct 17, 2026 · 9 min readX local-model bench digest — Oct 17, 2026Local-LLM throughput measured on X: 29 posts, 67 runs — hardware, quantization, engine and tok/s with a source link for every number.
- Oct 16, 2026 · 10 min readX local-model bench digest — Oct 16, 2026Local-LLM throughput measured on X: 27 posts, 54 runs — hardware, quantization, engine and tok/s with a source link for every number.
- Oct 10, 2026 · 9 min readX local-model bench digest — Oct 10, 2026Local-LLM throughput measured on X: 29 posts, 74 runs — hardware, quantization, engine and tok/s with a source link for every number.
- Oct 2, 2026 · 11 min readX local-model bench digest — Oct 2, 2026Local-LLM throughput measured on X: 47 posts, 137 runs — hardware, quantization, engine and tok/s with a source link for every number.
- Sep 29, 2026 · 11 min readX local-model bench digest — Sep 29, 2026Local-LLM throughput measured on X: 37 posts, 111 runs — hardware, quantization, engine and tok/s with a source link for every number.
- Sep 24, 2026 · 8 min readX local-model bench digest — Sep 24, 2026Local-LLM throughput measured on X: 15 posts and 44 runs, with hardware, quantization, engine, context and source links.
- Sep 20, 2026 · 9 min readX local-model bench digest — Sep 20, 2026Local-LLM throughput measured on X: 21 posts and 85 runs, with hardware, quantization, engine, context and source links.
- Sep 18, 2026 · 10 min readX local-model bench digest — Sep 18, 2026Local-LLM throughput measured on X: 30 posts and 77 runs, with hardware, quantization, engine, context and source links.
- Sep 16, 2026 · 9 min readX local-model bench digest — Sep 16, 2026Local-LLM throughput measured on X: 18 posts, 48 runs — hardware, quantization, engine and tok/s with a source link for every number.
- Sep 14, 2026 · 8 min readX local-model bench digest — Sep 14, 2026Local-LLM throughput measured on X: 12 posts, 25 runs — hardware, quantization, engine and tok/s with a source link for every number.
- Oct 12, 2026 · 9 min readX local-model bench digest — Oct 12, 2026Local-LLM throughput measured on X: 40 posts and 109 runs, with hardware, quantization, engine, context and source links.
- Sep 30, 2026 · 10 min readX local-model bench digest — Sep 30, 2026Local-LLM throughput measured on X: 32 posts, 90 runs — hardware, quantization, engine and tok/s with a source link for every number.
- Sep 28, 2026 · 8 min readQwen3.8-27B on 22 GPUs: 44 Sourced RunsOne model, 22 GPUs, 44 sourced runs, 16.8 to 435 tok/s. The card matters less than the quant, the engine and the context you run it at.
- Sep 28, 2026 · 9 min readOne 180B MoE, Five Formats: 41 Sourced RunsQwen3.8-Flash-Next's split entities merged into one page: 41 sourced rows across three checkpoints, 20.3 to 474 tok/s, and what fits a 24GB card.
- Sep 26, 2026 · 9 min readX local-model bench digest — Sep 26, 2026Local-LLM throughput measured on X: 27 posts, 74 runs — hardware, quantization, engine and tok/s with a source link for every number.
- Sep 23, 2026 · 8 min readX local-model bench digest — Sep 23, 2026Local-LLM throughput measured on X: 20 posts and 56 runs, with hardware, quantization, engine, context and source links.
- Sep 22, 2026 · 10 min readX local-model bench digest — Sep 22, 2026Local-LLM throughput measured on X: 29 posts, 79 runs — hardware, quantization, engine and tok/s with a source link for every number.
- Sep 15, 2026 · 7 min readContext Is Not a Cliff: 790× the Window, +18%Ten context points from 327 to 258,790 tokens on one DGX Spark: decode throughput barely notices context length. What actually moves the number.
- Sep 15, 2026 · 7 min readTwo RTX 3060s: 16 tok/s Dense, 94 tok/s MoEThe same dual-3060 rig runs a 30B dense model at 16 tok/s and a 30B mixture-of-experts at 94. That 6x spread decides whether the build is worth it.
- Sep 12, 2026 · 9 min readX local-model bench digest — Sep 12, 2026Local-LLM throughput measured on X: 25 posts, 81 runs — hardware, quantization, engine and tok/s with a source link for every number.
- Sep 7, 2026 · 6 min readQwen3.8-27B Uncensored: MTP Kept, Cost MeasuredHeretic abliteration on Qwen3.8-27B: refusals 98 to 12 per 100 prompts, ~0.5 capability points cost, MTP tensors kept. PPL, sizes and tok/s, linked.
- Sep 6, 2026 · 3 min readDual RTX 3090 70B Build (48GB, 2026)Two used 24GB cards give 48GB for 70B-class local inference. What it costs at the 2026-09-28 snapshot, and sourced tok/s.
- Sep 5, 2026 · 3 min readRyzen AI Max+ 395 (Strix Halo) 128GBRyzen AI Max+ 395 Strix Halo mini PCs put 128GB unified memory behind a 40-CU iGPU — real llama.cpp tok/s for 30B, 80B, 120B and 228B models.
- Sep 4, 2026 · 2 min read$1,500 Sweet-Spot Build (Used RTX 3090)The best tokens-per-dollar in 2026: one used RTX 3090 24GB runs 32B models interactive and 70B at patient speeds. Prices as of 2026-09-28.
- Sep 4, 2026 · 2 min read$800 Entry Build for Local LLMsThe cheapest sensible local-LLM machine: a 12GB card that runs 7B–14B models well. Prices as of 2026-09-28.
- Sep 4, 2026 · 2 min readRun 70B Models on Two Used RTX 3090sThe 48GB route: running 70B-class models at usable speeds on a pair of used RTX 3090s for around $2,600. Pricing as of 2026-09-28.
- Sep 4, 2026 · 3 min readState of Local AI Hardware — Q3 2026The VRAM shortage has flipped the 2026 GPU market: 5090s at $9,000, the 3060 resurrection, 48GB as the new anchor, and what it means for local-LLM buyers.