GeForce RTX 4060
NVIDIA
CompatibleRuns Llama 2 7B Q4_0 entirely in memory
Model · Meta
Llama 2 7B: VRAM and hardware requirements
At Q4_0 with an 8K context, Llama 2 7B needs about 8.6 GB: a GPU with 8 GB or more runs it entirely in VRAM.
- Parameters6.74B
- Max context4K
- LicenseLlama 2 Community License
Specifications
- Organization
- Meta
- Architecture
- llama
- Released
- July 2023
- Layers
- 32
- Hidden size
- 4096
- Attention heads
- 32
- KV heads
- 32
- Head dimension
- 128
- Vocabulary
- 32,000
- KV cache per 1K tokens (f16)
- 0.5 GB
- Repository
- meta-llama/Llama-2-7b-hf
Official specData updated 24 days agoSource: GGUF header of TheBloke/Llama-2-7B-GGUF (llama-2-7b.Q4_0.gguf) + Hugging Face API (parameter count)
Quantizations and memory
| Quantization | File size | Memory needed | Minimum GPU memory | Quality |
|---|---|---|---|---|
| F16 | Not available | ~18 GB | 20 GB | Reference (unquantized) |
| Q8_0 | Not available | ~11.8 GB | 11 GB | Near-lossless |
| Q4_0Most used | Not available | ~8.6 GB | 8 GB | Good — the usual choice |
File sizes: not available until imported from the model repository. Memory = weights (estimated from bits per weight) + KV cache + runtime overhead + safety margin. (F16 ≈ 16 bpw, Q8_0 ≈ 8.5 bpw, Q4_0 ≈ 4.64 bpw)
Q4, Q5 or Q8: how to choose a quantization
GPUs that run Llama 2 7B
| RTX 509032 GB · 64 GB RAM~6.5 GB · 191–249 tok/s estimate | Excellent |
|---|---|
| RTX 409024 GB · 64 GB RAM~6.5 GB · 135–150 tok/s estimate | Excellent |
| RTX 309024 GB · 64 GB RAM~6.5 GB · 109–135 tok/s estimate | Excellent |
| RX 7900 XTX24 GB · 64 GB RAM~6.6 GB · 90–133 tok/s estimate | Excellent |
| RTX 5060 Ti 16 GB16 GB · 64 GB RAM~6.5 GB · 59–72 tok/s estimate | Excellent |
| RTX 5070 Ti16 GB · 64 GB RAM~6.5 GB · 108–141 tok/s estimate | Excellent |
| RTX 508016 GB · 64 GB RAM~6.5 GB · 139–154 tok/s estimate | Excellent |
| RTX 4060 Ti 16 GB16 GB · 64 GB RAM~6.5 GB · 20–68 tok/s estimate | Excellent |
| RTX 4070 Ti SUPER16 GB · 64 GB RAM~6.5 GB · 95–116 tok/s estimate | Excellent |
| RX 9070 XT16 GB · 64 GB RAM~6.6 GB · 67–110 tok/s estimate | Excellent |
| RTX 3060 12 GB12 GB · 64 GB RAM~6.5 GB · 56–62 tok/s estimate | Excellent |
| RTX 407012 GB · 64 GB RAM~6.5 GB · 35–120 tok/s estimate | Excellent |
| RTX 507012 GB · 64 GB RAM~6.5 GB · 47–160 tok/s estimate | Excellent |
| RTX 40608 GB · 64 GB RAM~6.5 GB · 19–65 tok/s estimate | Compatible |
Macs that run Llama 2 7B
| M3 Ultra (80-core GPU) · 512 GBMac Studio~6.3 GB · 81–152 tok/s estimate | Excellent |
|---|---|
| M4 Max (40-core GPU) · 128 GBMacBook Pro 14, MacBook Pro 16~6.3 GB · 59–101 tok/s estimate | Excellent |
| M3 Max (40-core GPU) · 128 GBMacBook Pro 14, MacBook Pro 16~6.3 GB · 46–74 tok/s estimate | Excellent |
| M4 Pro (20-core GPU) · 64 GBMac mini, MacBook Pro 14~6.3 GB · 33–51 tok/s estimate | Excellent |
| M4 Max (32-core GPU) · 36 GBMacBook Pro 14, MacBook Pro 16~6.3 GB · 60–78 tok/s estimate | Excellent |
| M4 · 32 GBMacBook Air, MacBook Pro 14~6.3 GB · 15–22 tok/s estimate | Excellent |
| M5 · 32 GBMacBook Pro 14~6.3 GB · 20–29 tok/s estimate | Excellent |
| M3 · 24 GBMacBook Air, MacBook Pro 14~6.3 GB · 13–19 tok/s estimate | Excellent |
EstimateData updated 24 days agoSource: Run LLM Locally engine
Limits in plain words
The KV cache grows with context: at 4K it adds about 2 GB on top of the weights.
With under 8 GB of VRAM, use a smaller quantization or a smaller model of the same family.
Where to buy
Prices and links follow the market you choose.
Radeon RX 7600 XT
AMD
ExcellentRuns Llama 2 7B Q4_0 entirely in memory
AMD Ryzen AI Max+ 395
Unified memory PC
ExcellentRuns Llama 2 7B Q4_0 entirely in memory
GeForce RTX 5090
Highest estimated speed
ExcellentRuns Llama 2 7B Q4_0 entirely in memory
No current offer imported for this market: the link opens a marketplace search.
Frequently asked questions
How much VRAM does Llama 2 7B need?
About 8.6 GB at Q4_0 with an 8K context, including the KV cache and runtime overhead. At Q8_0 it needs about 11.8 GB. A longer context adds more: about 2 GB of KV cache at 32K.
Can Llama 2 7B run on a Mac?
Yes on a Mac whose GPU can use at least 6.3 GB, e.g. M3 24 GB. macOS lets the GPU use about two thirds to three quarters of unified memory by default.
How fast is Llama 2 7B?
On a RTX 4090, we estimate 135–150 tok/s at Q4_0. This is an estimate from memory bandwidth (confidence medium) until benchmarks are imported.