With 80 GB of VRAM, the H100 PCIe runs Qwen3 0.6B at Q4_K_M entirely on the GPU with an 8K context (about 2.4 GB needed). At Q8_0 it fits entirely.
Q4_K_M · 8K context
The model needs about 2.4 GB and fits in the 77.6 GB available, with 75.2 GB to spare.
~2.4 GB (2.1 GB–2.7 GB) needed · estimate
Fits entirely in VRAM · 75.2 GB left
EstimateData updated 24 days agoSource: Run LLM Locally engine, Hugging Face config.json + model card
What makes this estimate less reliable
| Qwen3 0.6B Q8_08K~2.5 GB · 417–892 tok/s estimate | Excellent |
|---|---|
| Qwen3 0.6B Q6_K8K~2.5 GB · 419–898 tok/s estimate | Excellent |
| Qwen3 0.6B Q5_K_M8K~2.4 GB · 427–929 tok/s estimate | Excellent |
| Qwen3 0.6B Q4_K_M8K~2.4 GB · 435–959 tok/s estimate | Excellent |
| Qwen3 0.6B IQ4_XS8K~2.3 GB · 440–975 tok/s estimate | Excellent |
| Qwen3 0.6B Q3_K_M8K~2.3 GB · 444–993 tok/s estimate | Excellent |
Prices and links follow the market you choose.
NVIDIA
ExcellentRuns Qwen3 0.6B Q4_K_M entirely in memory
AMD
ExcellentRuns Qwen3 0.6B Q4_K_M entirely in memory
Unified memory PC
ExcellentRuns Qwen3 0.6B Q4_K_M entirely in memory
No current offer imported for this market: the link opens a marketplace search.
