KV cache calculator: how much memory does context cost?

bytes per token = caching layers × 2 × KV heads × head dim × bytes per element — then multiply by the context length. At f16 that is 2 bytes per element; a q8_0 cache is 1.0625 and roughly halves it for a negligible loss; q4_0 quarters it and degrades long-context recall. Three families cache far less than the formula suggests and the table says which: sliding-window models keep only a few hundred tokens in most layers, latent-attention models (DeepSeek, GLM, Kimi) cache a compressed vector, and the 2026 hybrids (Qwen3.5+, Nemotron 3, LFM2.5) cache in only a fraction of their blocks.

GLM-4.7-Flash 30B-A3B has 47 blocks, and caches a compressed latent of 576 elements per layer instead of full keys and values. Per token, per caching layer: 576 elements × 2 bytes.