Hardware requirements

What hardware do I need to run Ornith 1.5 397B-A17B?

397B MoE · 17B active 256K context MIT New this week

At Q4_K_M the weights are 223.2 GB; with an 8K KV cache and runtime overhead that is 224.1 GB, so it wants more memory than any single card we list to run with headroom. Every figure below is computed from the model’s published shape against each device’s usable memory; tokens per second are estimates. The method.

Minimum practical 192 GB VRAM

Instinct MI300X — Q3_K_M, 181.5 GB of 190.4 GB, ~159 tok/s, up to 256K context.

the smallest card that holds it at all, at Q3_K_M — the lowest quantisation worth running
Recommended Nothing in the catalogue

No single device we list clears this bar — this is a multi-GPU or server-class model.

High quality Nothing in the catalogue

No single device we list clears this bar — this is a multi-GPU or server-class model.

Long context Nothing in the catalogue

No single device we list clears this bar — this is a multi-GPU or server-class model.

Apple Silicon 512 GB unified

M3 Ultra · 512 GB — Q4_K_M, 224.1 GB of 384.0 GB, ~22 tok/s, up to 256K context.

smallest Mac that runs the recommended quantisation with headroom
CPU only 256 GB RAM

CPU only · DDR5 8-channel server — Q3_K_M, 181.5 GB of 204.8 GB, ~7.4 tok/s, up to 256K context.

no GPU — weights stream from system RAM

How much VRAM at each quantisation

QuantWeights+KV 8K+KV 32K+KV 128KTotal @ 8KNeedsQuality
Q8_0392.8 GB0.23 GB0.94 GB3.8 GB 393.7 GBno single card−0.1% ppl
Q6_K303.2 GB0.23 GB0.94 GB3.8 GB 304.0 GBno single card−0.4% ppl
Q5_K_M262.0 GB0.23 GB0.94 GB3.8 GB 262.9 GBno single card−0.8% ppl
Q4_K_M223.2 GB0.23 GB0.94 GB3.8 GB 224.1 GBno single card−1.9% ppl
Q3_K_M180.7 GB0.23 GB0.94 GB3.8 GB 181.5 GBno single card−5.4% ppl
Q2_K154.8 GB0.23 GB0.94 GB3.8 GB 155.7 GB192 GB card−15% ppl

Totals add 0.6 GB runtime overhead and an f16 KV cache; "needs" is the smallest discrete card class whose usable memory (after display and driver reserve) leaves 15% headroom. A q8_0 cache roughly halves the KV columns.

Every device, checked

System RAM for offload:
163264128256

NVIDIA · GeForce

DeviceMemoryBest quant with headroomFits at allMax context @ Q4_K_MTok/s est.128K
GeForce RTX 5090 32 GB — offload — ~2.1 no Check
GeForce RTX 5080 16 GB — offload — ~1.9 no Check
GeForce RTX 5070 Ti 16 GB — offload — ~1.9 no Check
GeForce RTX 5070 12 GB — offload — ~1.9 no Check
GeForce RTX 5060 Ti 16 GB 16 GB — offload — ~1.9 no Check
GeForce RTX 5060 8 GB — offload — ~1.9 no Check
GeForce RTX 4090 24 GB — offload — ~2.0 no Check
GeForce RTX 4080 Super 16 GB — offload — ~1.9 no Check
GeForce RTX 4080 16 GB — offload — ~1.9 no Check
GeForce RTX 4070 Ti Super 16 GB — offload — ~1.9 no Check
GeForce RTX 4070 Ti 12 GB — offload — ~1.9 no Check
GeForce RTX 4070 Super 12 GB — offload — ~1.9 no Check
GeForce RTX 4070 12 GB — offload — ~1.9 no Check
GeForce RTX 4060 Ti 16 GB 16 GB — offload — ~1.9 no Check
GeForce RTX 4060 Ti 8 GB — offload — ~1.9 no Check
GeForce RTX 4060 8 GB — offload — ~1.9 no Check
GeForce RTX 3090 Ti 24 GB — offload — ~2.0 no Check
GeForce RTX 3090 24 GB — offload — ~2.0 no Check
GeForce RTX 3080 Ti 12 GB — offload — ~1.9 no Check
GeForce RTX 3080 12 GB 12 GB — offload — ~1.9 no Check
GeForce RTX 3080 10 GB — offload — ~1.9 no Check
GeForce RTX 3070 Ti 8 GB — offload — ~1.9 no Check
GeForce RTX 3070 8 GB — offload — ~1.9 no Check
GeForce RTX 3060 Ti 8 GB — offload — ~1.9 no Check
GeForce RTX 3060 12 GB 12 GB — offload — ~1.9 no Check
GeForce RTX 2080 Ti 11 GB — offload — ~1.9 no Check
GeForce RTX 2060 12 GB 12 GB — offload — ~1.9 no Check
GeForce GTX 1080 Ti 11 GB — offload — ~1.9 no Check

NVIDIA · GeForce laptop

DeviceMemoryBest quant with headroomFits at allMax context @ Q4_K_MTok/s est.128K
GeForce RTX 5090 Laptop 24 GB — offload — ~2.0 no Check
GeForce RTX 4090 Laptop 16 GB — offload — ~1.9 no Check
GeForce RTX 4080 Laptop 12 GB — offload — ~1.9 no Check
GeForce RTX 4070 Laptop 8 GB — offload — ~1.9 no Check

NVIDIA · Workstation

DeviceMemoryBest quant with headroomFits at allMax context @ Q4_K_MTok/s est.128K
RTX 6000 Ada 48 GB — offload — ~2.4 no Check
RTX 5000 Ada 32 GB — offload — ~2.1 no Check
RTX 4000 Ada 20 GB — offload — ~2.0 no Check
RTX 2000 Ada 16 GB — offload — ~1.9 no Check
RTX A6000 48 GB — offload — ~2.4 no Check
RTX A5000 24 GB — offload — ~2.0 no Check
RTX A4000 16 GB — offload — ~1.9 no Check

NVIDIA · Data centre

DeviceMemoryBest quant with headroomFits at allMax context @ Q4_K_MTok/s est.128K
L40S 48 GB — offload — ~2.4 no Check
L4 24 GB — offload — ~2.0 no Check
A10 24 GB — offload — ~2.0 no Check
A100 40 GB 40 GB — offload — ~2.3 no Check
A100 80 GB 80 GB — offload — ~3.1 no Check
H100 PCIe 80 GB — offload — ~3.1 no Check
H100 SXM 80 GB — offload — ~3.1 no Check
H200 SXM 141 GB — offload — ~7.4 no Check
Tesla P40 24 GB — offload — ~2.0 no Check

AMD · Radeon

DeviceMemoryBest quant with headroomFits at allMax context @ Q4_K_MTok/s est.128K
Radeon RX 9070 XT 16 GB — offload — ~1.9 no Check
Radeon RX 9070 16 GB — offload — ~1.9 no Check
Radeon RX 7900 XTX 24 GB — offload — ~2.0 no Check
Radeon RX 7900 XT 20 GB — offload — ~2.0 no Check
Radeon RX 7900 GRE 16 GB — offload — ~1.9 no Check
Radeon RX 7800 XT 16 GB — offload — ~1.9 no Check
Radeon RX 7700 XT 12 GB — offload — ~1.9 no Check
Radeon RX 7600 XT 16 GB — offload — ~1.9 no Check
Radeon RX 6900 XT 16 GB — offload — ~1.9 no Check
Radeon RX 6800 XT 16 GB — offload — ~1.9 no Check
Radeon RX 6700 XT 12 GB — offload — ~1.9 no Check

AMD · Radeon Pro

DeviceMemoryBest quant with headroomFits at allMax context @ Q4_K_MTok/s est.128K
Radeon Pro W7900 48 GB — offload — ~2.4 no Check
Radeon Pro W7800 32 GB — offload — ~2.1 no Check

AMD · Instinct

DeviceMemoryBest quant with headroomFits at allMax context @ Q4_K_MTok/s est.128K
Instinct MI210 64 GB — offload — ~2.7 no Check
Instinct MI300X 192 GB — Q3_K_M — — no Check

AMD · Ryzen AI Max

DeviceMemoryBest quant with headroomFits at allMax context @ Q4_K_MTok/s est.128K
Ryzen AI Max+ 395 · 128 GB 128 GB — no — — no Check
Ryzen AI Max+ 395 · 64 GB 64 GB — no — — no Check

Apple · Apple Silicon

DeviceMemoryBest quant with headroomFits at allMax context @ Q4_K_MTok/s est.128K
M1 · 16 GB 16 GB — no — — no Check
M1 Pro · 32 GB 32 GB — no — — no Check
M1 Max · 64 GB 64 GB — no — — no Check
M1 Ultra · 128 GB 128 GB — no — — no Check
M2 · 24 GB 24 GB — no — — no Check
M2 Pro · 32 GB 32 GB — no — — no Check
M2 Max · 96 GB 96 GB — no — — no Check
M2 Ultra · 192 GB 192 GB — no — — no Check
M3 · 24 GB 24 GB — no — — no Check
M3 Pro · 36 GB 36 GB — no — — no Check
M3 Max · 64 GB 64 GB — no — — no Check
M3 Max · 128 GB 128 GB — no — — no Check
M3 Ultra · 256 GB 256 GB — Q3_K_M — — no Check
M3 Ultra · 512 GB 512 GB Q6_K Q6_K 256K 22 yes Check
M4 · 32 GB 32 GB — no — — no Check
M4 Pro · 48 GB 48 GB — no — — no Check
M4 Pro · 64 GB 64 GB — no — — no Check
M4 Max · 64 GB 64 GB — no — — no Check
M4 Max · 128 GB 128 GB — no — — no Check

Intel · Arc

DeviceMemoryBest quant with headroomFits at allMax context @ Q4_K_MTok/s est.128K
Arc B580 12 GB — offload — ~1.9 no Check
Arc B570 10 GB — offload — ~1.9 no Check
Arc A770 16 GB 16 GB — offload — ~1.9 no Check
Arc A750 8 GB — offload — ~1.9 no Check

CPU · CPU + system RAM

DeviceMemoryBest quant with headroomFits at allMax context @ Q4_K_MTok/s est.128K
CPU only · DDR4 dual-channel 32 GB — no — — no Check
CPU only · DDR5 dual-channel 64 GB — no — — no Check
CPU only · DDR5 8-channel server 256 GB — Q3_K_M — — no Check