How much VRAM do you need for a local LLM?
The formula behind LLM memory — weights, KV cache, overhead — with the memory popular models need at each quantization.
A local LLM needs memory for four things: its weights (the parameters, compressed by quantization), the KV cache (the conversation's memory, which grows with context), the runtime overhead (CUDA/Metal context, compute buffers) and a safety margin.
Weights dominate: an 8-billion-parameter model at Q4_K_M takes about 4.6 GB, a 32B about 19 GB. Note that Q4 does not mean exactly 4 bits per weight: some tensors stay at higher precision, for about 4.9 bits on average.
The KV cache depends on the architecture: number of layers, KV heads (GQA) and context length. Qwen3 32B needs about 256 KiB per token in f16 — 8 GB at 32K tokens, almost half the weights.
Memory needed by model and quantization
| Model | Q8_0 | Q5_K_M | Q4_K_M | Q3_K_M |
|---|---|---|---|---|
| Llama 3.1 8B Instruct | ~10.2 GB | ~7.4 GB | ~6.7 GB | ~5.8 GB |
| Qwen3 14B | ~17.5 GB | ~12.5 GB | ~11 GB | ~9.4 GB |
| Gemma 3 27B IT | ~30.6 GB | ~21.3 GB | ~18.6 GB | ~15.6 GB |
| Qwen3 32B | ~37 GB | ~25.8 GB | ~22.6 GB | ~19 GB |
| Qwen3 30B-A3B | ~33.3 GB | ~22.8 GB | ~19.9 GB | ~16.5 GB |
| Llama 3.3 70B Instruct | ~76.8 GB | ~52.7 GB | ~45.8 GB | ~38 GB |
File sizes: not available until imported from the model repository. Memory = weights (estimated from bits per weight) + KV cache + runtime overhead + safety margin.
The rule of thumb
To run entirely on the GPU, total memory must fit in VRAM with a little headroom. Beyond that, llama.cpp can offload layers to system RAM, but speed drops because RAM is 5 to 20 times slower than VRAM. Check your exact case with the configurator.
EstimateData updated 24 days agoSource: Run LLM Locally engine