How to run an LLM locally: hardware, model, runtime

Self-hosting a language model is three decisions, in this order: what memory you have, which model fits it, and which program loads it. Everything else — the flags, the file names, the quantisation letters — follows from those three. This guide walks them in order, with the numbers computed from the same catalogue the tool uses, and ends with the exact command.

1 · Hardware: memory is the only number that matters

A model either fits in your GPU's memory or it does not. When it fits, generation runs at the speed of that memory — hundreds of gigabytes a second. When it does not, the overflow streams from system RAM at 60 GB/s and you lose an order of magnitude. So the first question is not "which GPU" but "how many gigabytes", and the answer by class is:

Three kinds of machine run these models, and they differ in one way each:

• A discrete GPU (NVIDIA, AMD, Intel) has its own fast memory — 8 to 48 GB on consumer and workstation cards. It is the fastest option when the model fits, and the slowest when it does not.
• A Mac or a Strix Halo PC has one memory pool shared with the OS, of which the GPU may use about three quarters. Capacity is cheap (64 and 128 GB are ordinary), bandwidth is a third to a half of a discrete flagship. More below.
• A CPU alone streams weights from system RAM at 50–90 GB/s. It runs anything that fits in RAM, slowly; usable for models under about 10B, or MoE models with a few billion active parameters. More below.

System RAM on a GPU machine matters only for overflow: 32 GB lets a card borrow room for a model a step too large, at a few tokens per second. It does not make a small card fast.

2 · Model: pick the job, then the largest that fits

Open-weight models are released at a handful of sizes — roughly 2B, 4B, 9B, 12B, 27B, 70B, and mixture-of-experts models from 20B to several hundred billion where only a fraction is active per token. Larger is better at the same generation, and newer beats older at the same size: a 2026 9B outperforms a 2024 70B on most tasks. The practical rule:

1. Decide the job. The shortlists differ — the best coding model is not the best writer, and the fastest is neither.
2. Take the strongest model on that job's list that fits your memory at Q4_K_M, the quantisation that costs least quality per gigabyte saved. What that means.
3. Check the context you need fits too. The KV cache grows with every token you keep in the window, and a 27B at 128K context can want more cache than weights. Price it.

The pages that do this for you: best local LLM by VRAM class, best for coding, or your own device's page, which picks one model per job for your exact card. If a model is not in the catalogue, paste its Hugging Face id and it is sized from its config.

3 · Runtime: the program that loads it

Five programs cover nearly everyone. They wrap the same two engines (llama.cpp on everything, MLX on Macs); what differs is how much they decide for you.

Start with Ollama. It is one install, it downloads the right file, and it keeps the model loaded between questions. Two things to know: its default context is 4K tokens regardless of the model, and its library tags are Q4_K_M unless you say otherwise. A 9B assistant on a 12 GB card, with the context set to the size that fits:

When the fit is tight — the model page says so — switch to llama.cpp, which exposes the two flags Ollama hides: how many layers go on the GPU, and what the KV cache is quantised to. The same model and card, with a q8_0 cache that halves the cache cost:

LM Studio is the same engines behind a window, and shows the fit estimate before you download. vLLM is for serving: several users at once, an OpenAI-compatible API, unquantised or AWQ weights, NVIDIA or AMD only — the right tool for a team box and the wrong one for a laptop. The five compared.

Running an LLM on a Mac

Apple Silicon changes the first decision. The GPU shares the whole memory pool, so a 64 GB MacBook Pro holds a 70B model at Q4 that no 24 GB card can, and a 128 GB machine holds the 100B-class MoEs. macOS caps what the GPU may wire down at roughly 75% of total (67% at 16 GB and below), and that cap is what every verdict on this site uses. The trade is bandwidth: 410 GB/s on an M4 Max against 1008 on an RTX 4090, so the same 27B runs at a third of the speed.

Use MLX, Apple's own framework: noticeably faster than GGUF on the same Mac, with community conversions of most models under mlx-community. Ollama and LM Studio run on Macs too and are simpler; LM Studio can use either engine.

The 16 GB and 24 GB base chips are laptop class: a 4B or 9B model with modest context. Pro and Max chips are where local models become comfortable.

Running with no GPU

llama.cpp and Ollama both run on a bare CPU. Weights stream from system RAM, so speed is RAM bandwidth divided by model size: a dual-channel DDR5 desktop at ~90 GB/s runs a 4B model at a usable pace and a 9B at reading speed. Mixture-of-experts models are the CPU's friend — Qwen3.6 35B-A3B touches only 3.3B parameters per token and is estimated at 9.0 tok/s here, despite 20.2 GB of weights. Use one thread per physical core, and put the file on an SSD.

What runs at 5 tokens per second or better on a CPU only · DDR5 dual-channel: Qwen3.6 35B-A3B (9.0), Ornith 1.5 35B-A3B (9.0), Nemotron 3.5 Lightning 30B-A3B (9.3), GLM-4.7-Flash 30B-A3B (9.9), Qwen3 30B-A3B (9.0). The full list.

Self-hosting for more than one person

A local model becomes a self-hosted service the moment a second person or program talks to it. Three things change:

• Use a serving engine. vLLM batches concurrent requests and scales throughput with users; llama.cpp's server handles a few at a time. Ollama exposes the same OpenAI-compatible API on port 11434 and is fine for a household.
• Budget the KV cache per user. Every open conversation holds its own context in memory. Eight users at 8K is 64K of cache — size for it with the calculator before choosing the model.
• Do not expose the port. None of these runtimes authenticate by default. Put them behind a reverse proxy with a key, or on a VPN. The OpenAI-compatible API means any client that takes a base URL will work once it is reachable.

Questions people arrive with

Can I run this LLM?

Pick your device from the catalogue and every model is sorted into runs-great, tight, needs-offload and won't-fit, at the context and quantisation you choose. Or open any model page and the what hardware do I need? link runs it the other way.

How much VRAM do I need for an LLM?

Parameters × bits per weight ÷ 8, plus the KV cache for your context, plus 0.6 GB overhead. At Q4_K_M that is roughly 0.6 GB per billion parameters: Qwen3.5 9B 5.4 GB, Qwen3.8 27B 15.6 GB, Llama 3.3 70B Instruct 39.7 GB. The VRAM calculator does it per model and device.

What are the hardware requirements for a local LLM?

There is no minimum, only a trade: 8 GB of VRAM runs a 9B model tightly, 24 GB runs the current 27B generation, 48 GB or a 64 GB Mac runs a 70B. The CPU, motherboard and storage barely matter; a fast SSD helps the first load. What each class unlocks.

Is running locally cheaper than an API?

It depends entirely on volume: a card is paid once, an API per token. Price the per-token side with the AI article cost calculator and hold it against the card you were about to buy.