Q4 vs Q5 vs Q8: which quantization for a local LLM?
What GGUF quantizations change in memory and quality, and how to choose between Q4_K_M, Q5_K_M, Q6_K and Q8_0.
Quantization lowers weight precision to save memory. Q8_0 is near-lossless at half the original memory; Q4_K_M nearly halves it again with a small loss, which makes it the default choice. Below Q4, quality drops clearly: a smaller model at Q4 often beats a bigger one at Q2.
| Model | Q8_0 | Q6_K | Q5_K_M | Q4_K_M | Q3_K_M | Q2_K |
|---|---|---|---|---|---|---|
| Qwen3 8B | ~10.5 GB | ~8.6 GB | ~7.7 GB | ~6.9 GB | ~6 GB | ~5.3 GB |
| Qwen3 14B | ~17.5 GB | ~14.1 GB | ~12.5 GB | ~11 GB | ~9.4 GB | ~7.9 GB |
| Qwen3 32B | ~37 GB | ~29.3 GB | ~25.8 GB | ~22.6 GB | ~19 GB | ~15.7 GB |
| Llama 3.3 70B Instruct | ~76.8 GB | ~60.2 GB | ~52.7 GB | ~45.8 GB | ~38 GB | ~30.8 GB |
File sizes: not available until imported from the model repository. Memory = weights (estimated from bits per weight) + KV cache + runtime overhead + safety margin.
| Quantization | Quality |
|---|---|
| Q8_0 | Near-lossless |
| Q6_K | Near-lossless |
| Q5_K_M | Very good |
| Q4_K_M | Good — the usual choice |
| Q3_K_M | Noticeable loss |
| Q2_K | Significant loss |
EstimateData updated 24 days agoSource: Run LLM Locally engine