CPU vs GPU for local LLMs: why memory bandwidth decides
Token generation reads every weight once per token. Here is why GPUs are 10× faster than CPUs, and when a CPU is enough.
To generate each token, the model must read all its active weights from memory. Generation speed is therefore limited by memory bandwidth far more than by compute. A dual-channel DDR5 desktop gives about 90 GB/s; an RTX 3090, 936 GB/s. Hence a 10× difference in speed.
| Machine | Bandwidth | Generation speed |
|---|---|---|
| Ryzen 7 9700X (DDR5-5600, 2 ch.) | 89.6 GB/s | 8–13 tok/s |
| Apple M4 Pro (20-core GPU) | 273 GB/s | 30–47 tok/s |
| RTX 3060 12 GB | 360 GB/s | 47–64 tok/s |
| RTX 4070 | 504 GB/s | 63–88 tok/s |
| RTX 3090 | 936 GB/s | 105–154 tok/s |
| RTX 5090 | 1,792 GB/s | 167–268 tok/s |
Qwen3 8B Q4_K_M · estimate · memory bandwidth (… GB/s) and model size
A CPU remains viable for small models and mixture-of-experts models, which read few weights per token: Qwen3 30B-A3B activates only 3.3 billion parameters per token. These speeds are theoretical estimates (low confidence); real benchmarks will replace them once imported.
EstimateData updated 24 days agoSource: Run LLM Locally engine