Only with CPU offload

DeepSeek V4.1 Flash 552B-A16B at Q4_K_M needs 311.0 GB but only 139.4 GB is addressable, so about 55% of the layers would stream from system RAM at roughly 60 GB/s. Expect around 2.8 tokens per second — usable for batch work, painful for chat.

45% on GPU 8K context Q4_K_M · 310.4 GB MIT Released 10 Sep 2026 New this month Vision Not in the Ollama library

Twice V4 Flash's backbone (552B, 16B active per generated token) plus 196B of Engram lookup tables (189 GiB at FP8) that DwarfStar streams from SSD, so they are not counted here. A 512 GB Mac model, and as of October 2026 no mainline llama.cpp or Ollama build: DwarfStar on a Mac, vLLM across four GPUs. DeepSeek puts the cache at 890 bytes/token; modelled conservatively as 4 latent layers plus a 128-token window.

What hardware do I need for DeepSeek V4.1 Flash 552B-A16B? →

The VRAM budget

weights 310.4 GB

Weights 310.4 GB KV cache @ 8K 0.04 GB Runtime overhead 0.6 GB Over budget 171.6 GB past 139.4 GB

Quantisation ladder

Quant	Weights	Total @ 8K	Max context	Tok/s	Quality	Fit
Q8_0	546.2 GB	546.9 GB	—	~1.2	−0.1% ppl	407.5 GB over
Q6_K	421.6 GB	422.2 GB	—	~1.7	−0.4% ppl	282.8 GB over
Q5_K_M	364.4 GB	365.0 GB	—	~2.1	−0.8% ppl	225.6 GB over
Q4_K_M	310.4 GB	311.0 GB	—	~2.8	−1.9% ppl	171.6 GB over
Q3_K_M	251.3 GB	251.9 GB	—	~4.2	−5.4% ppl	112.5 GB over
Q2_K	215.3 GB	215.9 GB	—	~6.2	−15% ppl	76.5 GB over

Quality is the published perplexity delta against f16 weights. Max context assumes an f16 KV cache; q8_0 roughly doubles it. This model interleaves sliding-window layers (128 tokens, 4 of 40 layers global), which is why its cache barely grows with context. This model uses multi-head latent attention, so its cache is a compressed latent rather than full K and V.

How to run it

vLLM

terminal

# in the vllm/vllm-openai:deepseekv41-flash-0909 image
$ vllm serve nvidia/DeepSeek-V4.1-Flash-NVFP4 \
    --tensor-parallel-size 4 \
    --tokenizer-mode deepseek_v41 \
    --reasoning-parser deepseek_v41 \
    --language-model-only \
    --max-model-len 8192

The command NVIDIA tested on four GB300 GPUs with its NVFP4 checkpoint, text only. No single card holds it.

A serving engine, not a chat app. Built for many requests at once. More on vLLM.

01The NVFP4 checkpoint is 491 GiB, split across four GPUs by tensor parallelism.

02Close anything else holding VRAM. A browser with hardware acceleration can sit on 1–2 GB.

03If it falls back to CPU silently, drop the context first, then step down a quantisation.

See all models for this rig Compare with another model