The single-file model format llama.cpp reads, and therefore Ollama, LM Studio and most local runtimes. It holds the quantised weights, tokenizer and metadata together, and the quantisation is in the file name.

What does Q4_K_M mean?

4-bit nominal width (about 4.8 effective bits per weight), a k-quant that keeps the embedding and output tensors at higher precision, and the medium mix of higher-precision tensors. It is the default: roughly a 2% perplexity cost for a 70% size reduction from 16-bit.

Which quantization should I download?

Q4_K_M unless you have memory to spare, then Q5_K_M, Q6_K or Q8_0. Drop to Q3_K_M only to afford a longer context; below that, a smaller model at Q4 beats a larger one at Q2.

Is Q8_0 better than Q4_K_M?

Measurably, by about 0.1% perplexity against about 2%, and it is nearly twice the size. In use the difference is rarely visible; Q6_K is the sensible top step.

How big is a 27B model in GGUF?

Qwen3.8 27B: 27.5 GB at Q8_0, 21.2 GB at Q6_K, 15.6 GB at Q4_K_M, 12.7 GB at Q3_K_M — weights only, before the KV cache.

Explainer

GGUF and LLM quantization explained: Q4_K_M, Q8_0 and which file to download

A model's download page lists a dozen files with names like Q4_K_M, Q5_K_S, Q8_0 and IQ3_XXS. They are the same model at different precisions, and the one you pick decides whether it fits your card and how much quality you lose for it. This page is the decoder ring, with the sizes computed for real models.

What GGUF is

GGUF is the file format llama.cpp reads — and so, underneath, Ollama, LM Studio, Jan, Kobold and most other local runtimes. One file holds the weights, the tokenizer and the metadata the runtime needs (layer count, attention shape, context length), so a .gguf is self-contained in a way a Hugging Face safetensors folder is not. It replaced the older GGML format in 2023. The weights inside it are usually quantised, and the quantisation is in the file name.

What quantization does

A model is trained in 16-bit floating point: two bytes per parameter, so a 27B model is 54 GB of weights. Quantisation stores each weight in fewer bits — 8, 6, 5, 4, 3 or 2 — and keeps a small per-block scale factor so the original range survives. The cost is rounding error, which shows up as a slightly higher perplexity (how surprised the model is by text) and, at the aggressive end, as visible damage: worse instruction following, worse maths, repetition.

The effective bits per weight sit above the nominal number because k-quants (the _K families) keep the embedding and output tensors at higher precision, and the block scales take a little room too. Those effective figures, from llama.cpp's own table, are what the sizes on this site are computed with:

Quant	Bits/weight	Qwen3.5 9B (9.65B)	Qwen3.8 27B (27.8B)	Llama 3.3 70B Instruct (70.6B)	Quality	Notes
F16	16	18.0 GB 24 GB card	51.8 GB 64 GB card	131.5 GB 192 GB card	Reference	Unquantised weights
Q8_0	8.5	9.5 GB 16 GB card	27.5 GB 40 GB card	69.9 GB 141 GB card	−0.1% ppl	Near-lossless
Q6_K	6.56	7.4 GB 11 GB card	21.2 GB 32 GB card	53.9 GB 80 GB card	−0.4% ppl	Indistinguishable in practice
Q5_K_M	5.67	6.4 GB 10 GB card	18.4 GB 32 GB card	46.6 GB 64 GB card	−0.8% ppl	Safe if it fits
Q4_K_M default	4.83	5.4 GB 10 GB card	15.6 GB 24 GB card	39.7 GB 64 GB card	−1.9% ppl	Recommended
Q3_K_M	3.91	4.4 GB 8 GB card	12.7 GB 20 GB card	32.1 GB 48 GB card	−5.4% ppl	Lossy — use to reach longer context
Q2_K	3.35	3.8 GB 8 GB card	10.8 GB 16 GB card	27.5 GB 40 GB card	−15% ppl	Visible damage. Prefer a smaller model.

Weights only; add the KV cache for your context and 0.6 GB overhead for the memory bill. "Card" is the smallest discrete class that runs it with headroom at an 8K context. Quality figures are published perplexity deltas, not our measurements.

Reading the name: Q4_K_M

Q4 — the nominal bit width. Q8 is near-lossless, Q4 is the default, Q2 is for emergencies.
_K — a k-quant: weights grouped in super-blocks with their own scales, and the important tensors kept at higher precision. Prefer these to the legacy Q4_0/Q4_1, which are the same size and worse.
_S / _M / _L — small, medium, large: how many of the attention and feed-forward tensors get a higher-precision quant than the headline number. _M is the usual balance; _S saves a few hundred megabytes, _L spends a few hundred for slightly better output.
IQ (as in IQ4_XS, IQ3_M, IQ2_XXS) — importance-matrix quants, which use a calibration pass to decide which weights can afford the fewest bits. At 3 bits and below they hold up noticeably better than the plain k-quant of the same size; at 4 bits the difference is small. They are slower on CPU and on older GPUs.
Q8_0 / F16 / BF16 — the reference tiers: 8-bit with a scale per block, and the unquantised halves.

Which one to download

Q4_K_M by default. About 4.8 bits per weight, a ~2% perplexity cost, and the point past which each extra bit buys the least. It is what Ollama's library tags are, and what this site sizes with.
Go up if it fits with room: Q5_K_M, Q6_K, then Q8_0. Q6_K is indistinguishable from the original in nearly every use; Q8_0 is for when you have the memory and want to stop thinking about it.
Go down to buy context, not to squeeze a model in. Q3_K_M is a legitimate choice when it is the difference between 8K and 32K of context on your card. Q2_K is not: at that point a smaller model at Q4 is better than a larger one at Q2.
Native formats are not quantisations. gpt-oss ships in MXFP4 — its 4-bit weights are the reference weights, not a lossy conversion of something larger — and DeepSeek V4.1 Flash in NVFP4. Take the native file; a GGUF "Q8" of it is bigger with nothing gained.

Every model page here prints the whole ladder against your card — size, max context and verdict at each quantisation — so the decision is a lookup. Try Qwen3.8 27B on a 4090 or Gemma 4 12B on a 3060.

Quantising the KV cache too

The weights are only half the bill. Every token in the context window is stored as a key and a value per layer, in 16-bit by default, and at long context that cache can outweigh the weights. llama.cpp can quantise it (-ctk q8_0 -ctv q8_0), which is a separate decision from the weight quantisation:

Cache type	Bytes/element	Qwen3 32B @ 32K
f16	2 B	8.00 GB	Default. No quality loss.
q8_0	1.0625 B	4.25 GB	Roughly halves the cache for a negligible loss.
q4_0	0.5625 B	2.25 GB	Quarter size. Degrades long-context recall.

q8_0 is nearly free and roughly halves the cache; q4_0 degrades long-context recall and is a last resort. The KV cache calculator prices it per model and context.

Where the files are

Model authors publish safetensors; the GGUF conversions come from the community — bartowski, unsloth, lmstudio-community and others — within a day or two of a release, as repositories named ORG/MODEL-GGUF with one file per quantisation. llama.cpp and Ollama can pull straight from one: llama-server -hf ORG/MODEL-GGUF:Q4_K_M or ollama run hf.co/ORG/MODEL-GGUF:Q4_K_M. On a Mac the equivalent for MLX is an mlx-community/MODEL-4bit repository, where 4-bit and 8-bit are the usual steps.

Paste any model id into the VRAM calculator and the page lists the GGUF conversions it found, sized against your device.