Yes — with 138.9 GB to spare
Ling 3.0 Tiny 7.9B-A1.3B at Q4_K_M fits your M2 Ultra · 192 GB entirely in unified memory at 8K context, at an estimated 285 tokens per second. There is room for its full 128K window.
An 8B MoE with 1.3B active and a latent KV cache on only 6 of 24 layers — reasoning and tool use sized for Apple Silicon and edge boxes.
The VRAM budget
Quantisation ladder
| Quant | Weights | Total @ 8K | Max context | Tok/s | Quality | Fit |
|---|---|---|---|---|---|---|
| F16 | 14.7 GB | 15.4 GB | 128K | 86 | Reference | Long context |
| Q8_0 | 7.8 GB | 8.5 GB | 128K | 162 | −0.1% ppl | Long context |
| Q6_K | 6.0 GB | 6.7 GB | 128K | 210 | −0.4% ppl | Long context |
| Q5_K_M | 5.2 GB | 5.9 GB | 128K | 243 | −0.8% ppl | Long context |
| Q4_K_M | 4.4 GB | 5.1 GB | 128K | 285 | −1.9% ppl | Recommended |
| Q3_K_M | 3.6 GB | 4.2 GB | 128K | 353 | −5.4% ppl | Long context |
| Q2_K | 3.1 GB | 3.7 GB | 128K | 411 | −15% ppl | Long context |
Quality is the published perplexity delta against f16 weights. Max context assumes an f16 KV cache; q8_0 roughly doubles it. Only 6 of its 24 blocks keep a per-token KV cache; the rest are linear-attention, Mamba or convolution blocks with a fixed-size state. This model uses multi-head latent attention, so its cache is a compressed latent rather than full K and V.
How to run it
$ pip install mlx-lm $ mlx_lm.generate --model mlx-community/Ling-3.0-tiny-4bit \ --max-tokens 512 --prompt "Hello"
Apple's own array framework. The fastest path on Apple Silicon. More on MLX.