Llama-3.3-70B-Instruct
1recorded run
20.1best tok/s
1verified
What the Llama-3.3-70B-Instruct numbers mean
The fastest verified result on our board is 20.1 output tok/s on an CMP 170HX at NVFP4 with a 32768-token context. That is about 49.7 ms per generated token — 500 tokens takes roughly 24.9 seconds. Prompt processing on the same record runs at 626.2 tok/s, so a 2,000-token prompt is read in about 3.2 seconds. Peak VRAM was measured at 57.4 GB.
How much VRAM does Llama-3.3-70B-Instruct need?
Llama-3.3-70B-Instruct has 71.0B parameters. These estimates use the same formula as the VRAM calculator: weights, plus about 15% for a moderate-context KV cache and runtime overhead, plus a small engine floor.
| Quantisation | Estimated VRAM | Smallest card that fits |
|---|---|---|
| Q4_K_M (4-bit) | 46.7 GB | 64 GB |
| Q8_0 (8-bit) | 87.6 GB | 141 GB |
| FP16 (16-bit) | 164.1 GB | multi-GPU or CPU offload |
Fastest GPUs for this model (best tok/s)
- CMP 170HX20.1
Models with a similar footprint
| Model | Params | Q4_K_M weights |
|---|---|---|
| Qwen3-Coder-Next | 80B | 45.0 GB |
| Kimi-Linear-48B-A3B-Instruct | 49B | 27.6 GB |
| Ornith-1.5-35B-A3B | 36B | 20.2 GB |
| Ornith-1.5-35B-A3B-GPTQ-Int4-sym-G128-MTP-BF16-MixedCal-v2 | 36B | 20.2 GB |
| Ornith-1.5-35B-A3B-MTP | 36B | 20.2 GB |
| Qwen3.6-35B-A3B | 36B | 20.2 GB |