LLM VRAM calculator: how much VRAM does a model need on your GPU?

VRAM needed = weights + KV cache + overhead, checked against usable memory, which is the advertised size less what the display and driver keep (about 1.6 GB on a discrete card; a quarter of a Mac's unified memory).

• Weights = parameters × bits per weight ÷ 8. At Q4_K_M (4.83 effective bits) that is about 0.6 GB per billion parameters; at Q8_0, 1.06 GB. For a mixture-of-experts model every parameter counts, not just the active ones — they all have to be resident.
• KV cache = layers × 2 × KV heads × head dimension × context × bytes per element. It is linear in context, so 128K costs sixteen times 8K — unless the model uses sliding-window, latent or hybrid attention, which the calculator models exactly from each config. Its own calculator.
• Overhead = a flat 0.6 GB for the CUDA or Metal context and compute buffers. The least precise number here.

A fit with at least 15% of usable memory spare is "runs great"; under that it is "tight"; past it, the overflow streams from system RAM at a few tokens per second. The method, in full.