Hardware requirements
What hardware do I need to run Qwen3.5 0.8B?
0.87B
256K context
Apache 2.0
At Q4_K_M the weights are 0.5 GB; with an 8K KV cache and runtime overhead that is 1.2 GB, so it wants a 8 GB card to run with headroom — in practice 8 GB VRAM: GeForce RTX 3070 Ti and 7 others. Every figure below is computed from the model’s published shape against each device’s usable memory; tokens per second are estimates. The method.
Minimum practical
8 GB VRAM
GeForce RTX 3070 Ti — Q8_0, 1.6 GB of 7.0 GB, ~428 tok/s, up to 256K context.
the smallest card that holds it at all, at Q3_K_M — the lowest quantisation worth running
Recommended
8 GB VRAM
GeForce RTX 3070 Ti — Q4_K_M, 1.2 GB of 7.0 GB, ~752 tok/s, up to 256K context.
runs the recommended quantisation with headroom at 8K context
High quality
8 GB VRAM
GeForce RTX 3070 Ti — Q8_0, 1.6 GB of 7.0 GB, ~428 tok/s, up to 256K context.
near-lossless Q8_0 with headroom
Long context
8 GB VRAM
GeForce RTX 3070 Ti — Q4_K_M, 2.6 GB of 7.0 GB, ~752 tok/s at 128K.
128K tokens (or the model’s maximum) at the recommended quantisation
Apple Silicon
16 GB unified
M1 · 16 GB — Q4_K_M, 1.2 GB of 10.7 GB, ~78 tok/s, up to 256K context.
smallest Mac that runs the recommended quantisation with headroom
CPU only
32 GB RAM
CPU only · DDR4 dual-channel — Q8_0, 1.6 GB of 25.6 GB, ~25 tok/s, up to 256K context.
no GPU — weights stream from system RAM
How much VRAM at each quantisation
| Quant | Weights | +KV 8K | +KV 32K | +KV 128K | Total @ 8K | Needs | Quality |
|---|---|---|---|---|---|---|---|
| F16 | 1.6 GB | 0.09 GB | 0.38 GB | 1.5 GB | 2.3 GB | 8 GB card | Reference |
| Q8_0 | 0.9 GB | 0.09 GB | 0.38 GB | 1.5 GB | 1.6 GB | 8 GB card | −0.1% ppl |
| Q6_K | 0.7 GB | 0.09 GB | 0.38 GB | 1.5 GB | 1.4 GB | 8 GB card | −0.4% ppl |
| Q5_K_M | 0.6 GB | 0.09 GB | 0.38 GB | 1.5 GB | 1.3 GB | 8 GB card | −0.8% ppl |
| Q4_K_M | 0.5 GB | 0.09 GB | 0.38 GB | 1.5 GB | 1.2 GB | 8 GB card | −1.9% ppl |
| Q3_K_M | 0.4 GB | 0.09 GB | 0.38 GB | 1.5 GB | 1.1 GB | 8 GB card | −5.4% ppl |
| Q2_K | 0.3 GB | 0.09 GB | 0.38 GB | 1.5 GB | 1.0 GB | 8 GB card | −15% ppl |
Totals add 0.6 GB runtime overhead and an f16 KV cache; "needs" is the smallest discrete card class whose usable memory (after display and driver reserve) leaves 15% headroom. A q8_0 cache roughly halves the KV columns.
NVIDIA · GeForce
| Device | Memory | Best quant with headroom | Fits at all | Max context @ Q4_K_M | Tok/s est. | 128K | |
|---|---|---|---|---|---|---|---|
| GeForce RTX 5090 | 32 GB | Q8_0 | Q8_0 | 256K | 2218 | yes | Check |
| GeForce RTX 5080 | 16 GB | Q8_0 | Q8_0 | 256K | 1188 | yes | Check |
| GeForce RTX 5070 Ti | 16 GB | Q8_0 | Q8_0 | 256K | 1109 | yes | Check |
| GeForce RTX 5070 | 12 GB | Q8_0 | Q8_0 | 256K | 832 | yes | Check |
| GeForce RTX 5060 Ti 16 GB | 16 GB | Q8_0 | Q8_0 | 256K | 554 | yes | Check |
| GeForce RTX 5060 | 8 GB | Q8_0 | Q8_0 | 256K | 554 | yes | Check |
| GeForce RTX 4090 | 24 GB | Q8_0 | Q8_0 | 256K | 1247 | yes | Check |
| GeForce RTX 4080 Super | 16 GB | Q8_0 | Q8_0 | 256K | 911 | yes | Check |
| GeForce RTX 4080 | 16 GB | Q8_0 | Q8_0 | 256K | 887 | yes | Check |
| GeForce RTX 4070 Ti Super | 16 GB | Q8_0 | Q8_0 | 256K | 832 | yes | Check |
| GeForce RTX 4070 Ti | 12 GB | Q8_0 | Q8_0 | 256K | 624 | yes | Check |
| GeForce RTX 4070 Super | 12 GB | Q8_0 | Q8_0 | 256K | 624 | yes | Check |
| GeForce RTX 4070 | 12 GB | Q8_0 | Q8_0 | 256K | 624 | yes | Check |
| GeForce RTX 4060 Ti 16 GB | 16 GB | Q8_0 | Q8_0 | 256K | 356 | yes | Check |
| GeForce RTX 4060 Ti | 8 GB | Q8_0 | Q8_0 | 256K | 356 | yes | Check |
| GeForce RTX 4060 | 8 GB | Q8_0 | Q8_0 | 256K | 337 | yes | Check |
| GeForce RTX 3090 Ti | 24 GB | Q8_0 | Q8_0 | 256K | 1247 | yes | Check |
| GeForce RTX 3090 | 24 GB | Q8_0 | Q8_0 | 256K | 1158 | yes | Check |
| GeForce RTX 3080 Ti | 12 GB | Q8_0 | Q8_0 | 256K | 1129 | yes | Check |
| GeForce RTX 3080 12 GB | 12 GB | Q8_0 | Q8_0 | 256K | 1129 | yes | Check |
| GeForce RTX 3080 | 10 GB | Q8_0 | Q8_0 | 256K | 940 | yes | Check |
| GeForce RTX 3070 Ti | 8 GB | Q8_0 | Q8_0 | 256K | 752 | yes | Check |
| GeForce RTX 3070 | 8 GB | Q8_0 | Q8_0 | 256K | 554 | yes | Check |
| GeForce RTX 3060 Ti | 8 GB | Q8_0 | Q8_0 | 256K | 554 | yes | Check |
| GeForce RTX 3060 12 GB | 12 GB | Q8_0 | Q8_0 | 256K | 445 | yes | Check |
| GeForce RTX 2080 Ti | 11 GB | Q8_0 | Q8_0 | 256K | 762 | yes | Check |
| GeForce RTX 2060 12 GB | 12 GB | Q8_0 | Q8_0 | 256K | 416 | yes | Check |
| GeForce GTX 1080 Ti | 11 GB | Q8_0 | Q8_0 | 256K | 599 | yes | Check |
NVIDIA · GeForce laptop
| Device | Memory | Best quant with headroom | Fits at all | Max context @ Q4_K_M | Tok/s est. | 128K | |
|---|---|---|---|---|---|---|---|
| GeForce RTX 5090 Laptop | 24 GB | Q8_0 | Q8_0 | 256K | 1109 | yes | Check |
| GeForce RTX 4090 Laptop | 16 GB | Q8_0 | Q8_0 | 256K | 713 | yes | Check |
| GeForce RTX 4080 Laptop | 12 GB | Q8_0 | Q8_0 | 256K | 535 | yes | Check |
| GeForce RTX 4070 Laptop | 8 GB | Q8_0 | Q8_0 | 256K | 317 | yes | Check |
NVIDIA · Workstation
| Device | Memory | Best quant with headroom | Fits at all | Max context @ Q4_K_M | Tok/s est. | 128K | |
|---|---|---|---|---|---|---|---|
| RTX 6000 Ada | 48 GB | Q8_0 | Q8_0 | 256K | 1188 | yes | Check |
| RTX 5000 Ada | 32 GB | Q8_0 | Q8_0 | 256K | 713 | yes | Check |
| RTX 4000 Ada | 20 GB | Q8_0 | Q8_0 | 256K | 445 | yes | Check |
| RTX 2000 Ada | 16 GB | Q8_0 | Q8_0 | 256K | 277 | yes | Check |
| RTX A6000 | 48 GB | Q8_0 | Q8_0 | 256K | 950 | yes | Check |
| RTX A5000 | 24 GB | Q8_0 | Q8_0 | 256K | 950 | yes | Check |
| RTX A4000 | 16 GB | Q8_0 | Q8_0 | 256K | 554 | yes | Check |
NVIDIA · Data centre
| Device | Memory | Best quant with headroom | Fits at all | Max context @ Q4_K_M | Tok/s est. | 128K | |
|---|---|---|---|---|---|---|---|
| L40S | 48 GB | Q8_0 | Q8_0 | 256K | 1069 | yes | Check |
| L4 | 24 GB | Q8_0 | Q8_0 | 256K | 371 | yes | Check |
| A10 | 24 GB | Q8_0 | Q8_0 | 256K | 742 | yes | Check |
| A100 40 GB | 40 GB | Q8_0 | Q8_0 | 256K | 1924 | yes | Check |
| A100 80 GB | 80 GB | Q8_0 | Q8_0 | 256K | 2523 | yes | Check |
| H100 PCIe | 80 GB | Q8_0 | Q8_0 | 256K | 2475 | yes | Check |
| H100 SXM | 80 GB | Q8_0 | Q8_0 | 256K | 4146 | yes | Check |
| H200 SXM | 141 GB | Q8_0 | Q8_0 | 256K | 5940 | yes | Check |
| Tesla P40 | 24 GB | Q8_0 | Q8_0 | 256K | 428 | yes | Check |
AMD · Radeon
| Device | Memory | Best quant with headroom | Fits at all | Max context @ Q4_K_M | Tok/s est. | 128K | |
|---|---|---|---|---|---|---|---|
| Radeon RX 9070 XT | 16 GB | Q8_0 | Q8_0 | 256K | 798 | yes | Check |
| Radeon RX 9070 | 16 GB | Q8_0 | Q8_0 | 256K | 798 | yes | Check |
| Radeon RX 7900 XTX | 24 GB | Q8_0 | Q8_0 | 256K | 1188 | yes | Check |
| Radeon RX 7900 XT | 20 GB | Q8_0 | Q8_0 | 256K | 990 | yes | Check |
| Radeon RX 7900 GRE | 16 GB | Q8_0 | Q8_0 | 256K | 713 | yes | Check |
| Radeon RX 7800 XT | 16 GB | Q8_0 | Q8_0 | 256K | 772 | yes | Check |
| Radeon RX 7700 XT | 12 GB | Q8_0 | Q8_0 | 256K | 535 | yes | Check |
| Radeon RX 7600 XT | 16 GB | Q8_0 | Q8_0 | 256K | 356 | yes | Check |
| Radeon RX 6900 XT | 16 GB | Q8_0 | Q8_0 | 256K | 634 | yes | Check |
| Radeon RX 6800 XT | 16 GB | Q8_0 | Q8_0 | 256K | 634 | yes | Check |
| Radeon RX 6700 XT | 12 GB | Q8_0 | Q8_0 | 256K | 475 | yes | Check |
AMD · Radeon Pro
| Device | Memory | Best quant with headroom | Fits at all | Max context @ Q4_K_M | Tok/s est. | 128K | |
|---|---|---|---|---|---|---|---|
| Radeon Pro W7900 | 48 GB | Q8_0 | Q8_0 | 256K | 1069 | yes | Check |
| Radeon Pro W7800 | 32 GB | Q8_0 | Q8_0 | 256K | 713 | yes | Check |
AMD · Instinct
| Device | Memory | Best quant with headroom | Fits at all | Max context @ Q4_K_M | Tok/s est. | 128K | |
|---|---|---|---|---|---|---|---|
| Instinct MI210 | 64 GB | Q8_0 | Q8_0 | 256K | 2027 | yes | Check |
| Instinct MI300X | 192 GB | Q8_0 | Q8_0 | 256K | 6559 | yes | Check |
AMD · Ryzen AI Max
| Device | Memory | Best quant with headroom | Fits at all | Max context @ Q4_K_M | Tok/s est. | 128K | |
|---|---|---|---|---|---|---|---|
| Ryzen AI Max+ 395 · 128 GB | 128 GB | Q8_0 | Q8_0 | 256K | 292 | yes | Check |
| Ryzen AI Max+ 395 · 64 GB | 64 GB | Q8_0 | Q8_0 | 256K | 292 | yes | Check |
Apple · Apple Silicon
| Device | Memory | Best quant with headroom | Fits at all | Max context @ Q4_K_M | Tok/s est. | 128K | |
|---|---|---|---|---|---|---|---|
| M1 · 16 GB | 16 GB | Q8_0 | Q8_0 | 256K | 78 | yes | Check |
| M1 Pro · 32 GB | 32 GB | Q8_0 | Q8_0 | 256K | 228 | yes | Check |
| M1 Max · 64 GB | 64 GB | Q8_0 | Q8_0 | 256K | 457 | yes | Check |
| M1 Ultra · 128 GB | 128 GB | Q8_0 | Q8_0 | 256K | 914 | yes | Check |
| M2 · 24 GB | 24 GB | Q8_0 | Q8_0 | 256K | 114 | yes | Check |
| M2 Pro · 32 GB | 32 GB | Q8_0 | Q8_0 | 256K | 228 | yes | Check |
| M2 Max · 96 GB | 96 GB | Q8_0 | Q8_0 | 256K | 457 | yes | Check |
| M2 Ultra · 192 GB | 192 GB | Q8_0 | Q8_0 | 256K | 914 | yes | Check |
| M3 · 24 GB | 24 GB | Q8_0 | Q8_0 | 256K | 114 | yes | Check |
| M3 Pro · 36 GB | 36 GB | Q8_0 | Q8_0 | 256K | 171 | yes | Check |
| M3 Max · 64 GB | 64 GB | Q8_0 | Q8_0 | 256K | 457 | yes | Check |
| M3 Max · 128 GB | 128 GB | Q8_0 | Q8_0 | 256K | 457 | yes | Check |
| M3 Ultra · 256 GB | 256 GB | Q8_0 | Q8_0 | 256K | 914 | yes | Check |
| M3 Ultra · 512 GB | 512 GB | Q8_0 | Q8_0 | 256K | 914 | yes | Check |
| M4 · 32 GB | 32 GB | Q8_0 | Q8_0 | 256K | 137 | yes | Check |
| M4 Pro · 48 GB | 48 GB | Q8_0 | Q8_0 | 256K | 312 | yes | Check |
| M4 Pro · 64 GB | 64 GB | Q8_0 | Q8_0 | 256K | 312 | yes | Check |
| M4 Max · 64 GB | 64 GB | Q8_0 | Q8_0 | 256K | 468 | yes | Check |
| M4 Max · 128 GB | 128 GB | Q8_0 | Q8_0 | 256K | 624 | yes | Check |
Intel · Arc
| Device | Memory | Best quant with headroom | Fits at all | Max context @ Q4_K_M | Tok/s est. | 128K | |
|---|---|---|---|---|---|---|---|
| Arc B580 | 12 GB | Q8_0 | Q8_0 | 256K | 564 | yes | Check |
| Arc B570 | 10 GB | Q8_0 | Q8_0 | 256K | 470 | yes | Check |
| Arc A770 16 GB | 16 GB | Q8_0 | Q8_0 | 256K | 693 | yes | Check |
| Arc A750 | 8 GB | Q8_0 | Q8_0 | 256K | 634 | yes | Check |
CPU · CPU + system RAM
| Device | Memory | Best quant with headroom | Fits at all | Max context @ Q4_K_M | Tok/s est. | 128K | |
|---|---|---|---|---|---|---|---|
| CPU only · DDR4 dual-channel | 32 GB | Q8_0 | Q8_0 | 256K | 44 | yes | Check |
| CPU only · DDR5 dual-channel | 64 GB | Q8_0 | Q8_0 | 256K | 77 | yes | Check |
| CPU only · DDR5 8-channel server | 256 GB | Q8_0 | Q8_0 | 256K | 263 | yes | Check |