DiggTop
Menu
⌕

MoE Expert Cache: VRAM Stops Being the Ceiling

A widely shared thread this week made a specific claim: a ~23GB MoE model (Qwen3.6-35B-A3B-NVFP4) running at 90+ tps on an RTX 5070 Ti with 16GB of VRAM — a model that does not fit on the card by a standard measure. The trick behind it is not a new quantization. It is expert caching: cold experts live in system RAM, hot experts are cached in VRAM, and cache misses are handled by a bandwidth-adaptive mix of PCIe transfers and CPU compute. Two concrete implementations now exist — a research system with a paper (FreeToken) and an RFC in llama.cpp's official repository.

As of 2026-09-07. All numbers below are reported by their respective authors or community members — none of it has independent replication yet; see Sources.

Why MoE models broke the VRAM budget

Mixture-of-Experts models activate only a small fraction of their parameters per token — Qwen3.8-Flash-Next activates ~3B of its 19.5B — but the standard inference requirement is that every expert is resident in VRAM, because the router can pick any of them on the next token. When a model overflows VRAM, the existing fallback is static offloading (--cpu-moe / -ot .ffn_.*_exps.=CPU in llama.cpp): experts that do not fit stay on the CPU, and every decode step pays the cost of reading those expert weights from system RAM at memory bandwidth while the GPUs sit idle. That is the ceiling the new work removes.

The enabling observation, measured in the llama.cpp RFC, is that decode-time expert routing is heavily skewed: on Qwen3.5-122B, the top 10% of experts take ~80% of routing hits. A small, hot subset of experts does most of the work — so a dynamic cache of the hot subset in VRAM is a high-hit-rate bet.

FreeToken: the research system

FreeToken is an "edge-native MoE serving" system from a Berkeley-led author list (Shuo Yang, Xiaoze Fan, Melissa Pan, …, Kurt Keutzer, Song Han, Matei Zaharia, Chenfeng Xu, Ion Stoica; arXiv:2608.16157). Instead of committing to a fixed split of the model between VRAM and RAM, it keeps the complete expert set in CPU RAM as the source of truth and uses the GPU's spare memory as a shared LRU cache for recently used experts. When a routed expert is not in the cache, a bandwidth-adaptive policy decides per miss whether to stream it over PCIe or compute it on the CPU — whichever is cheaper on that machine, re-measured as hardware conditions change.

The paper's headline results (author-reported, single runs):

Setup FreeToken llama.cpp
GLM-5.2 (753B) on one RTX PRO 6000 14.9 tok/s 7.3 tok/s (2.0×)
Model reach 35B on an 8GB laptop GPU; 284B on a gaming desktop; 753B on one workstation GPU —

It supports 20+ MoE models, exposes OpenAI- and Anthropic-compatible APIs (PyPI: freetoken v0.1.2, repo at FlashML-org/FreeToken), and ships a one-click app at flashml.ai. It is NVIDIA-first — the miss-path kernel is CUDA.

llama.cpp RFC #24528: the same idea, inverted

The llama.cpp side (RFC "MoE expert cache, VRAM caching of hot CPU-resident experts with hybrid hit/miss execution", opened by leloch as the follow-up to the closed PR #24524) takes a different, deliberately conservative architecture: MUL_MAT_ID stays on the CPU. Inside the CPU kernel, thread 0 dispatches the cached (hit) rows as one batched matvec on the GPU while the other threads compute the miss rows on the CPU as usual. Consequences:

Two quirks worth noting: the RFC is transparent that the document and its code were "predominantly generated by an AI (Claude Fable 5), working under my direction," and it is still an RFC — ~1,700 lines in a new CUDA file pair plus a backend-agnostic API table, not yet merged.

An independent community implementation of the same pattern for Vulkan (dual Radeon RX 6950 XT, Qwen3.8-Flash-Next UD-Q2_K_XL, experts forced to CPU) reported warm-cache decode going from 8.2 t/s to 18.9–19.1 t/s at ~93% hit rate (freenode.net writeup).

Community numbers (no methodology — not on the leaderboard)

Per site policy these are attributed claims, not benchmark runs:

None of these posts document the model revision, cache size, hit rate, context length, or full command line, so none is eligible for benchmark_runs under the "no methodology, no leaderboard" rule. If any author publishes the full setup, the ingestion path is documented in the site's runbook.

What this does not mean

Bottom line

The constraint "the model must fit in VRAM" was an implementation accident of dense expert placement, not a law of physics — decode routing is skewed enough that a small hot expert set does most of the work, and both a research lab and llama.cpp's community converged on the same fix within weeks. For giant MoE models on a consumer box, the practical question shifts from "does it fit?" to "how hot are its experts, and what is your RAM bandwidth?"

Sources

All measurements reported by their respective authors or community members as of 2026-09-07; no independent replication yet.

↑