Only with CPU offload
Qwen2.5-Coder 32B at Q4_K_M needs 21.0 GB but only 10.6 GB is addressable, so about 57% of the layers would stream from system RAM at roughly 60 GB/s. Expect around 3.3 tokens per second — usable for batch work, painful for chat.
The first local code model that felt competitive with hosted assistants. Qwen3.8 27B is smaller and far ahead on agentic work.
Hardware requirements What hardware do I need to run Qwen2.5-Coder 32B? →At Q4_K_M with an 8K context: a 32 GB GPU (RTX 5090) or a 36 GB Mac (M4 Max). Minimum, recommended, high-quality and long-context tiers, with all 114 devices checked at every quantisation.
Fits instead: Qwen2.5-Coder 14B (10.4 GB) · Qwen2.5-Coder 7B (5.3 GB)
The VRAM budget
Quantisation ladder
| Quant | Weights | Total @ 8K | Max context | Tok/s | Quality | Fit |
|---|---|---|---|---|---|---|
| Q8_0 | 32.5 GB | 35.1 GB | — | ~1.5 | −0.1% ppl | 24.5 GB over |
| Q6_K | 25.0 GB | 27.6 GB | — | ~2.1 | −0.4% ppl | 17.0 GB over |
| Q5_K_M | 21.7 GB | 24.3 GB | — | ~2.6 | −0.8% ppl | 13.7 GB over |
| Q4_K_M | 18.4 GB | 21.0 GB | — | ~3.3 | −1.9% ppl | 10.4 GB over |
| Q3_K_M | 14.9 GB | 17.5 GB | — | ~4.9 | −5.4% ppl | 6.9 GB over |
| Q2_K | 12.8 GB | 15.4 GB | — | ~6.8 | −15% ppl | 4.8 GB over |
Quality is the published perplexity delta against f16 weights. Max context assumes an f16 KV cache; q8_0 roughly doubles it.
How to run it
$ llama-server \
-hf Qwen/Qwen2.5-Coder-32B-Instruct:Q4_K_M \
-c 8192 -ngl 27
The engine underneath most of the others. Every knob is exposed. More on llama.cpp.
Related models on a GeForce RTX 3080 Ti
| Model | Total | Tok/s est. | |
|---|---|---|---|
| Qwen2.5-Coder 14B 14.8B · same family |
Q4_K_M · 10.4 GB | 66 | Tight fit |
| Qwen2.5-Coder 7B 7.62B · same family |
Q4_K_M · 5.3 GB | 129 | Runs great |
| Qwen3 32B 32.8B · similar size |
Q4_K_M · 21.0 GB | ~3.3 | CPU offload |
| DeepSeek-R1-Distill-Qwen 32B 32.8B · similar size |
Q4_K_M · 21.0 GB | ~3.3 | CPU offload |
| LLM-jp 4 33B Thinking 33.2B · similar size |
Q4_K_M · 21.3 GB | ~3.2 | CPU offload |
| Olmo 3.1 32B Instruct 32.2B · similar size |
Q4_K_M · 20.0 GB | ~3.7 | CPU offload |
| Nemotron 3.5 Lightning 30B-A3B 31.6B MoE · similar size |
Q4_K_M · 18.4 GB | ~16 | CPU offload |
Qwen2.5-Coder 32B on other hardware
| Device | Total | Tok/s est. | |
|---|---|---|---|
| GeForce RTX 4060 Laptop 7.0 GB usable · 256 GB/s |
21.0 GB | — | Won't fit |
| GeForce RTX 3060 12 GB 10.6 GB usable · 360 GB/s |
21.0 GB | ~3.1 | CPU offload |
| M4 · 16 GB 10.7 GB usable · 120 GB/s |
21.0 GB | — | Won't fit |
| GeForce RTX 4090 22.4 GB usable · 1008 GB/s |
21.0 GB | 33 | Tight fit |
| Radeon RX 7900 XTX 22.4 GB usable · 960 GB/s |
21.0 GB | 32 | Tight fit |
| GeForce RTX 5090 30.4 GB usable · 1792 GB/s |
21.0 GB | 59 | Runs great |
| M4 Max · 36 GB 27.0 GB usable · 410 GB/s |
21.0 GB | 12 | Runs great |
| M3 Max · 64 GB 48.0 GB usable · 400 GB/s |
21.0 GB | 12 | Runs great |