Which GPU to buy for local LLMs: VRAM first

How to choose a graphics card for local AI: VRAM sets the model size, bandwidth sets the speed. The largest model each popular GPU runs.

For a local LLM, a graphics card is judged on two numbers, in this order: the amount of VRAM, which decides the size of the model that fits entirely on the GPU, then the memory bandwidth, which decides how fast it generates. Raw compute (the gaming "TFLOPS") matters much less.

The largest model each card runs

The table gives, for common cards, the largest model in our catalogue that runs entirely in VRAM at its default quantization, with an 8K context. Each row links to the card's page, with its compatible models and offers.

How to choose

Tight budget: 12 GB (RTX 3060 12 GB) is enough for 8–14B models, which handle everyday uses well. Avoid 8 GB cards: you hit the limit quickly, see 12, 16 or 24 GB of VRAM.

The 24 GB tier (used RTX 3090, RTX 4090, RX 7900 XTX) is where you move to 27–32B models, a clear jump in quality. The RTX 5090's 32 GB adds context headroom but does not make 70B models fit, which need far more: the configurator computes each case.

NVIDIA, AMD or Intel: NVIDIA (CUDA) is the most widely supported ecosystem for local tools; AMD and Intel mostly run through Vulkan or dedicated backends, with fewer user reports. Speeds shown on the site are bandwidth estimates, or real measurements where they exist.

Need more than 32 GB? Compare with a Mac with a large unified memory: Mac or PC for a local LLM.

EstimateData updated 38 days agoSource: Run LLM Locally engine