Yes — with 5.5 GB to spare
Ling 3.0 Tiny 7.9B-A1.3B at Q4_K_M fits your GeForce RTX 3060 12 GB entirely on the GPU at 8K context, at an estimated 115 tokens per second. There is room for its full 128K window.
An 8B MoE with 1.3B active and a latent KV cache on only 6 of 24 layers — reasoning and tool use sized for Apple Silicon and edge boxes.
The VRAM budget
Quantisation ladder
| Quant | Weights | Total @ 8K | Max context | Tok/s | Quality | Fit |
|---|---|---|---|---|---|---|
| F16 | 14.7 GB | 15.4 GB | — | ~13 | Reference | 4.8 GB over |
| Q8_0 | 7.8 GB | 8.5 GB | 128K | 65 | −0.1% ppl | Long context |
| Q6_K | 6.0 GB | 6.7 GB | 128K | 84 | −0.4% ppl | Long context |
| Q5_K_M | 5.2 GB | 5.9 GB | 128K | 98 | −0.8% ppl | Long context |
| Q4_K_M | 4.4 GB | 5.1 GB | 128K | 115 | −1.9% ppl | Recommended |
| Q3_K_M | 3.6 GB | 4.2 GB | 128K | 142 | −5.4% ppl | Long context |
| Q2_K | 3.1 GB | 3.7 GB | 128K | 165 | −15% ppl | Long context |
Quality is the published perplexity delta against f16 weights. Max context assumes an f16 KV cache; q8_0 roughly doubles it. Only 6 of its 24 blocks keep a per-token KV cache; the rest are linear-attention, Mamba or convolution blocks with a fixed-size state. This model uses multi-head latent attention, so its cache is a compressed latent rather than full K and V.
How to run it
$ llama-server \
-hf inclusionAI/Ling-3.0-tiny:Q4_K_M \
-c 8192 -ngl 99
The engine underneath most of the others. Every knob is exposed. More on llama.cpp.