Method and data sources
Memory
Required memory = model weights + KV cache + runtime overhead + safety margin. Weights use the real GGUF file size when it has been imported; otherwise parameters × effective bits per weight (below), shown as an estimate with a ±6% range. The KV cache is computed from the architecture (layers × KV heads × head dimension × context × bytes), with MLA and sliding-window attention handled explicitly.
| F16 | 16 |
|---|---|
| BF16 | 16 |
| Q8_0 | 8.5 |
| Q6_K | 6.57 |
| Q5_K_M | 5.7 |
| Q4_K_M | 4.9 |
| Q4_0 | 4.64 |
| IQ4_XS | 4.46 |
| Q3_K_M | 4 |
| Q2_K | 3.17 |
| MXFP4 | 4.9 |
Compatibility levels
Fits in usable VRAM with 15% headroom and at least 10 tok/s → Excellent. Fits → Compatible. Fits only at the low end of the estimate → Compatible with limits. Needs system RAM with at least 50% on the GPU → CPU/GPU offload; less → Possible but slow. Below 5 tok/s → Possible but slow; below 2 tok/s → Not recommended. Does not fit at all → Incompatible.
Apple Silicon: the GPU may use about 67% of unified memory up to 36 GB and 75% above, by default. There is no offload.
Speed
Real benchmarks on the same hardware, model and quantization come first (labelled measured). Then benchmarks of the same family on the same hardware, or of the same model on other hardware, scaled. Otherwise a bandwidth model: time per token ≈ bytes read per token ÷ (efficiency × bandwidth) + layers × fixed overhead per layer. The overhead (kernel launches, synchronisation) is why a very fast GPU does not run small models proportionally faster. Both ranges, per backend and per model type (below), are fitted on the approved llama-bench results: dense models from the llama.cpp scoreboards (Llama 2 7B), mixture-of-experts models from gpt-oss and Qwen3 Coder, which read scattered experts and pay about twice the overhead per layer. Backends without benchmarks keep a zero overhead. Formula-only estimates are always low confidence.
| Backend | Model type | Efficiency (share of peak bandwidth) | Overhead per layer and token |
|---|---|---|---|
| CUDA | Dense | 72–92 % | 0.02–0.06 ms |
| CUDA | Mixture of experts | 60–78 % | 0.04–0.09 ms |
| ROCm | Dense | 66–88 % | 0.04–0.1 ms |
| ROCm | Mixture of experts | 50–70 % | 0.07–0.16 ms |
| Vulkan | Dense | 60–88 % | 0.01–0.06 ms |
| Vulkan | Mixture of experts | 50–74 % | 0.02–0.12 ms |
| SYCL | Dense | 35–55 % | 0–0 ms |
| SYCL | Mixture of experts | 35–55 % | 0–0 ms |
| Metal | Dense | 62–86 % | 0–0.1 ms |
| Metal | Mixture of experts | 55–75 % | 0.11–0.24 ms |
| CPU | Dense | 45–70 % | 0–0 ms |
| CPU | Mixture of experts | 45–70 % | 0–0 ms |
How accurate are the speed estimates?
Every approved benchmark is re-estimated with the formula alone, as if the measurement did not exist, then compared with what was measured. 157 of 172 measured speeds (91%) fall inside the estimated range, on 40 different GPUs, Macs and unified-memory machines. At the median, the middle of the range is 1.03× the measured speed, and 95% of midpoints are within 0.75× to 1.35× of the measurement. Rows outside the range are kept: they are real measurements. When a benchmark exists for your exact hardware, model and quantization, the site shows it instead, labelled measured.
| Backend | Measurements | Inside the estimated range |
|---|---|---|
| CUDA | 79 | 67 |
| Metal | 10 | 10 |
| ROCm | 39 | 39 |
| Vulkan | 44 | 41 |
Where the benchmarks come from
Every measured speed is a llama-bench result published by its author in the llama.cpp repository or its GitHub discussions. A row enters as pending and is used only after a manual check against its source; rejected rows stay recorded so they are not imported again.
Thanks to the 72 people whose published results are used here: @1337hero, @65a, @aajll, @AdamNiederer, @andj1210, @ariya, @bloodsweatncode, @brunodantas, @CarvalhoLeonardo, @cebeach, @chavdarb, @ckane, @daMustermann, @daniandtheweb, @davispuh, @ddpasa, @Diablo-D3, @easyfab, @elgris, @eugr, @fassn, @firefox42, @fkdosilovic, @ggerganov, @grgsim, @gyhor, @Hadrianneue, @Hassan-A-K, @Hedede, @hspak, @hypengw, @ilcommm, @ivanfioravanti, @Johnreidsilver, @JonRickSanchez, @KawaiiHachimi, @Kraust, @kresle, @l8bloom, @lhl, @marcingomulkiewicz, @max-krasnyansky, @MihaiBojescu, @NO-ob, @odbguru, @panasalik, @pontostroy, @prototypicall, @pwilkin, @QuantiusBenignus, @ranma, @Ristovski, @RodriMora, @rohan-sircar, @roykim98, @savvadesogle, @slaren, @spamshaker, @StarGuardian, @suneet-kumar-yunextraffic, @tabletuser-blogspot, @TinyServal, @Tom94, @totaldev, @ubergarm, @vgrebinski, @volyrique, @wbruna, @xog64, @xxxajk, @ygafarov, @zacharyarnaise.
Report an error or cite a figure
If a figure looks wrong for your hardware, write to contact@runllmlocally.com with the page, your setup and, if you have it, your llama-bench output. It is checked against the source and corrected.
To cite a figure, link to the page it comes from and give the data version shown at the bottom of the page, for example: “Run LLM Locally, Method and data sources, data of 25 September 2026.”
Data sources
Model architectures from each model's config.json on Hugging Face; hardware specifications from NVIDIA, AMD, Intel and Apple. Every record carries its source, URL and retrieval date. Missing data is shown as not available, never guessed.
Official specData updated 24 days agoSource: Hugging Face, NVIDIA, AMD, Apple