GeForce RTX 5090
NVIDIA
CompatibleRuns Mixtral 8x7B Instruct Q4_K_M entirely in memory
Model · Mistral AI
Mixtral 8x7B Instruct: VRAM and hardware requirements
At Q4_K_M with an 8K context, Mixtral 8x7B Instruct needs about 29.6 GB: a GPU with 32 GB or more runs it entirely in VRAM.
- Parameters46.7B
- Active12.9B
- Max context32K
- LicenseApache-2.0
Specifications
- Organization
- Mistral AI
- Architecture
- mixtral
- Released
- December 2023
- Layers
- 32
- Hidden size
- 4096
- Attention heads
- 32
- KV heads
- 8
- Head dimension
- 128
- Vocabulary
- 32,000
- KV cache per 1K tokens (f16)
- 0.125 GB
- Repository
- mistralai/Mixtral-8x7B-Instruct-v0.1
Official specData updated 24 days agoSource: Hugging Face config.json + model card
Quantizations and memory
| Quantization | File size | Memory needed | Minimum GPU memory | Quality |
|---|---|---|---|---|
| BF16 | Not available | ~93 GB | 96 GB | Reference (unquantized) |
| Q8_0 | 46.2 GB | ~50.2 GB | 80 GB | Near-lossless |
| Q6_K | 35.7 GB | ~39.2 GB | 48 GB | Near-lossless |
| Q5_K_M | 30 GB | ~34.2 GB | 48 GB | Very good |
| Q4_K_MMost used | 24.6 GB | ~29.6 GB | 32 GB | Good — the usual choice |
| IQ4_XS | Not available | ~27.1 GB | 32 GB | Good — the usual choice |
| Q3_K_M | 19 GB | ~24.5 GB | 24 GB | Noticeable loss |
| Q2_K | 14.6 GB | ~19.7 GB | 20 GB | Significant loss |
File sizes: not available until imported from the model repository. Memory = weights (estimated from bits per weight) + KV cache + runtime overhead + safety margin. (BF16 ≈ 16 bpw, Q8_0 ≈ 8.5 bpw, Q6_K ≈ 6.57 bpw, Q5_K_M ≈ 5.7 bpw, Q4_K_M ≈ 4.9 bpw, IQ4_XS ≈ 4.46 bpw, Q3_K_M ≈ 4 bpw, Q2_K ≈ 3.17 bpw)
Q4, Q5 or Q8: how to choose a quantization
GPUs that run Mixtral 8x7B Instruct
| RTX 509032 GB · 64 GB RAM~27.5 GB · 103–154 tok/s estimate | Compatible |
|---|---|
| RTX 409024 GB · 64 GB RAM~27.5 GB · 13–25 tok/s estimate | CPU/GPU offload |
| RTX 309024 GB · 64 GB RAM~27.5 GB · 13–24 tok/s estimate | CPU/GPU offload |
| RX 7900 XTX24 GB · 64 GB RAM~27.6 GB · 12–23 tok/s estimate | CPU/GPU offload |
| RTX 5060 Ti 16 GB16 GB · 64 GB RAM~27.5 GB · 5–10 tok/s estimate | CPU/GPU offload |
| RTX 5070 Ti16 GB · 64 GB RAM~27.5 GB · 5.5–11 tok/s estimate | CPU/GPU offload |
| RTX 508016 GB · 64 GB RAM~27.5 GB · 5.5–11 tok/s estimate | CPU/GPU offload |
| RTX 4060 Ti 16 GB16 GB · 64 GB RAM~27.5 GB · 5–9 tok/s estimate | CPU/GPU offload |
| RTX 4070 Ti SUPER16 GB · 64 GB RAM~27.5 GB · 5.5–10 tok/s estimate | CPU/GPU offload |
| RX 9070 XT16 GB · 64 GB RAM~27.6 GB · 5–10 tok/s estimate | CPU/GPU offload |
| RTX 3060 12 GB12 GB · 64 GB RAM~27.5 GB · 4–7.5 tok/s estimate | Possible but slow |
| RTX 407012 GB · 64 GB RAM~27.5 GB · 4–8 tok/s estimate | Possible but slow |
| RTX 507012 GB · 64 GB RAM~27.5 GB · 4–8 tok/s estimate | Possible but slow |
| RTX 40608 GB · 64 GB RAM~27.5 GB · 3.5–6.5 tok/s estimate | Possible but slow |
Macs that run Mixtral 8x7B Instruct
| M3 Ultra (80-core GPU) · 512 GBMac Studio~27.3 GB · 42–65 tok/s estimate | Excellent |
|---|---|
| M4 Max (40-core GPU) · 128 GBMacBook Pro 14, MacBook Pro 16~27.3 GB · 31–47 tok/s estimate | Excellent |
| M3 Max (40-core GPU) · 128 GBMacBook Pro 14, MacBook Pro 16~27.3 GB · 24–36 tok/s estimate | Excellent |
| M4 Pro (20-core GPU) · 64 GBMac mini, MacBook Pro 14~27.3 GB · 18–26 tok/s estimate | Excellent |
| M4 Max (32-core GPU) · 36 GBMacBook Pro 14, MacBook Pro 16~27.3 GB · No data | Compatible with limits |
| M4 · 32 GBMacBook Air, MacBook Pro 14~27.3 GB · No data | Incompatible |
| M5 · 32 GBMacBook Pro 14~27.3 GB · No data | Incompatible |
| M3 · 24 GBMacBook Air, MacBook Pro 14~27.3 GB · No data | Incompatible |
EstimateData updated 24 days agoSource: Run LLM Locally engine
Limits in plain words
Mixtral 8x7B Instruct is a mixture-of-experts model: all 46.7B parameters must be in memory, but only 12.9B are read per token, so it generates much faster than a dense model of the same size.
The KV cache grows with context: at 32K it adds about 4 GB on top of the weights.
With under 8 GB of VRAM, use a smaller quantization or a smaller model of the same family.
Where to buy
Prices and links follow the market you choose.
Radeon PRO W7900
AMD
ExcellentRuns Mixtral 8x7B Instruct Q4_K_M entirely in memory
MacBook Pro 14 M3 Max
Apple Silicon
ExcellentRuns Mixtral 8x7B Instruct Q4_K_M entirely in memory
AMD Ryzen AI Max+ 395
Unified memory PC
ExcellentRuns Mixtral 8x7B Instruct Q4_K_M entirely in memory
No current offer imported for this market: the link opens a marketplace search.
Frequently asked questions
How much VRAM does Mixtral 8x7B Instruct need?
About 29.6 GB at Q4_K_M with an 8K context, including the KV cache and runtime overhead. At Q8_0 it needs about 50.2 GB. A longer context adds more: about 4 GB of KV cache at 32K.
Can Mixtral 8x7B Instruct run on a Mac?
Yes on a Mac whose GPU can use at least 27.3 GB, e.g. M4 Max (32-core GPU) 36 GB. macOS lets the GPU use about two thirds to three quarters of unified memory by default.
How fast is Mixtral 8x7B Instruct?
On a RTX 4090, we estimate 13–25 tok/s at Q4_K_M. This is an estimate from memory bandwidth (confidence low) until benchmarks are imported.