DiggTop
Menu
⌕

MoE Expert Offload: Same Weights, 26× Apart

On September 10, 2026, J&M Labs published a day-one test of DeepSeek-V4.1-Flash on a single DGX Station GB300 — a machine in the ~$93,000 class, roughly fifteen times the top of our weekly price snapshots and not something most of our readers will ever own. The part worth keeping anyway: the same checkpoint, on the same box, on the same day, ran at 3.3 tok/s under one offload design and 85–150 tok/s under another. The 26× gap comes down to one question — how many bytes does the engine move per decode step, and does it move rows or layers?

As of 2026-09-11. Every performance number below is an author-reported first-hand measurement from that one Station; we did not run these tests, so they are not on our benchmark board (see the note at the end).

The setup: a 748B model that does not fit

DeepSeek-V4.1-Flash is a 552B backbone plus a 196B Engram table, shipped as a mixed FP4/FP8/BF16 checkpoint of 510,313,343,553 bytes (~475 GiB) with no re-quantization. One GB300 Station exposes 250.7 GiB of HBM, with 494 GB of Grace LPDDR5X behind it. Even with the Engram tables placed in host memory — which is what DeepSeek's design intends — the resident set overshoots HBM before any KV cache is allocated. A second offload tier is mandatory: the author ended up pinning 189 GiB of Engram in Grace RAM and moving 60–70 GiB of routed experts to host memory.

That second tier is the entire story.

3.3 tok/s: copy every layer, every token

SGLang's OffloaderV2 in cpu mode copies the entire expert tensor of each offloaded layer on every token: 15 layers × 6.72 GiB ≈ 100 GiB per decode step across the C2C link. At roughly 330 GB/s that is ~300 ms of transfer — exactly the 3.3 tok/s measured. The GPU reports 100% utilization the whole time because it is stalled on the copy stream, not computing. Batching hides the problem almost perfectly: 13.0 tok/s at C4, 25.8 tok/s at C8. That is why a batched benchmark can look respectable while single-stream is unusable.

85–150 tok/s: read only the rows the router picked

vLLM's UVA backend maps the pinned host tensor into GPU address space and lets the MoE kernel read only the rows the router selected: 6 of 384 experts per layer, about 1.6 GB per token instead of ~100 GiB. Same weights, same fixtures, same box:

Fixture (131K ctx, temp 0) SGLang OffloaderV2 vLLM UVA
C1 single stream 3.3 tok/s 85.2 tok/s
C4 aggregate 13.0 tok/s 172.7 tok/s
C8 aggregate 25.8 tok/s 183.5 tok/s
300-word paragraph (~313 tok) 94.9 s 3.5 s
Count 1–60 (120 tok) 36.9 s 1.5 s

It is not a quirk of one model, either. The same box had already shown the same swap on full GLM-5.3: 1.4 tok/s with SGLang OffloaderV2, 33 tok/s with vLLM UVA.

Why this matters if you will never buy a $93k box

The bandwidth ratio is the whole argument. That Station's C2C link runs at roughly 330 GB/s. A consumer PCIe 4.0 x16 slot delivers about 25–32 GB/s in practice, and PCIe 5.0 x16 about 50–64 GB/s. A design that copies whole layers yields 3.3 tok/s at 330 GB/s; scale that to PCIe 4.0 and you are well under 1 tok/s — unusable for a single user, never mind two. A row-selective design moves bytes proportional to the active experts rather than to the size of the offloaded layer, so it degrades gracefully as the link gets slower instead of collapsing.

That is the same conclusion consumer-side work has been converging on: FreeToken's LRU expert cache with a bandwidth-adaptive miss path, llama.cpp's RFC #24528 (experts stay on the CPU, hot rows are served by the GPU — +25% on GLM-5.1 754B across 4× RTX 3090), and Vulkan dual-card reports. We covered that thread in MoE Expert Cache: VRAM Stops Being the Ceiling. The GB300 test is the same lesson measured at the top of the market: what decides viability is the selectivity of the reads, not how clever the quantization is.

RAM and NVMe are part of the rig. The Station runs with 189 GiB of Engram pinned in host memory plus 70 GiB of offloaded experts, and host memory sits at 429 of 494 GB. The consumer analogue is the 4× RTX 3090 box that had to keep a 47.7 GB FP8 n-gram table on NVMe with 31 GB of system RAM. Once a model exceeds VRAM, system RAM capacity and even storage placement are first-class parameters, not afterthoughts.

The 150 tok/s is a fixture. The day is 88.

The article's most useful number is not the headline. With DSpark speculative decoding (the drafter ships inside the checkpoint), single-stream decode depends entirely on what the model is generating:

Workload (C1, temp 0) No spec DSpark k=5 k=5 acceptance
Shell command sequences 93 tok/s 144–150 tok/s 90.8%
Code (Python, bash) 93 tok/s 139–146 tok/s 63.8%
Tool-call JSON (real tools=) 93 tok/s ~130 tok/s 87.8%
Structured text (tables, bullets) 93 tok/s 111–116 tok/s 49.5%
Prose 93 tok/s 89–93 tok/s 29.8%

On a prose fixture, k=5 was 10–18% slower than no speculation: over 7,998 draft steps, acceptance by position was 59.5 / 30.4 / 13.9 / 6.1 / 2.8% — 22.5% overall, 2.13 tokens per verify step. The reason is specific to offloaded MoE: a verify step pushes k+1 tokens through every MoE layer, and each of those tokens fetches its own expert rows across the offload path. Six tokens per step means six times the expert traffic, so landing two of them is a loss.

Replaying 24 real agent turns from the author's session (12 ending in a tool call, 12 text; temperature 0, 400-token cap) gave 88.1 tok/s at 49.1% draft acceptance — a wash against the 93 tok/s no-spec baseline — with the turns that genuinely emitted a tool call running at 138–171 tok/s. If you use MTP or a draft model on an offloaded MoE, measure acceptance on your own prompts before assuming a speedup: the fixture decides. It is the same pattern we saw on consumer MTP heads, where prose gained 1.19× and code 1.28×.

Three more things worth stealing

  1. Prefill is boringly linear. Cold prefill held roughly 18K tok/s from 6K to 200K tokens, eased to 16.7K at 415K, and reached 11.4K tok/s at 972,435 tokens (85 s to first token). In the 1M configuration, a 10.05 GiB KV pool held 4.79M tokens — about 4.5 concurrent full-length requests.
  2. A long prefill starves everyone behind it. With vLLM's defaults, a 480K-token prompt blocked short requests for 3–18 s (18.2 s TTFT for a request fired 5 s in). Capping the long request's per-step chunk at 6144 tokens (--long-prefill-token-threshold 6144) cut that to about 1 s, at a 16% cost to the long request. If a GPU pool is shared by more than one user or agent, chunked-prefill behavior is a first-class tuning knob. Note it is interleaved, not isolated: short requests still decode at only ~2.5 tok/s while they overlap the long chunk.
  3. The autotune cache is keyed to your launch flags. The first MXFP4 MoE autotune on that box took 74 minutes (168 tuned configs). Changing --max-num-seqs, the speculative config, or --max-model-len produces a new engine-config hash and an empty cache directory, so the tuner starts over — seeding the new directory with an existing autotune_configs.json before launch turns 74 minutes into minutes. Two smaller traps from the same page: the model's chat template rejects reasoning_effort: medium outright, and the offload weight matcher is routed_experts.*, not experts.*.

What this does not say

Bottom line

When a model does not fit in VRAM, the question is not "how much did we offload" but "how many bytes per token does the engine read, and does it read rows or layers". Row-selective reads are why expert offload works at all. Everything else — quantization, launch flags, speculative decoding, KV layout — tunes a number that only exists if the read pattern is already right.

FAQ

Why is expert offload slow in some engines?

Because the engine moves whole expert tensors instead of the routed rows. On the GB300 test, SGLang's OffloaderV2 copied 15 layers × 6.72 GiB ≈ 100 GiB per decode step to produce 3.3 tok/s, while vLLM's UVA path read only the 6-of-384 experts the router selected — about 1.6 GB per token — for 85–150 tok/s on the same weights.

Does speculative decoding help on a MoE model with offloaded experts?

Only when acceptance is high on your actual text. Each verify step runs k+1 tokens through every MoE layer, and each token fetches its own expert rows over the offload path, so the cost scales with the draft block. The author measured 144–150 tok/s on shell commands but 89–93 on prose, where k=5 was 10–18% slower than no speculation; on real agent transcripts the average was 88.1 tok/s.

Can a 500B-class MoE run on consumer GPUs?

Not this one — the test machine pairs 288 GB of HBM with 494 GB of host RAM. Consumer-scale giant-MoE work (for example 4× RTX 3090 running a 754B model) depends on expert caching and row-selective reads over PCIe, which is slower and more sensitive to RAM bandwidth; see our MoE expert cache coverage and the full benchmark board.

Sources

Author-reported third-party measurements as of 2026-09-10. DiggTop did not run these tests, so they are not eligible for benchmark_runs under our methodology rule; they appear here as an engineering result with the source linked.

↑