Local LLMs for beginners: everything explained simply

Run an AI model on your own computer, without the jargon: memory, model size, quantization, speed, first steps and a glossary.

You may already use ChatGPT or another online assistant. Those assistants run on a company's servers. A local LLM is the same kind of assistant, running directly on your own computer.

This guide starts from zero. No technical background needed: every word is explained, and a glossary waits for you at the end.

What is an LLM?

LLM stands for Large Language Model. It is a program that learned from a huge amount of text and does one thing: predict the most likely continuation of a text. You ask a question, it writes the answer piece by piece. Those pieces are called tokens: a short word, part of a long word or a punctuation mark.

In practice, a model is a big file you download. It holds billions of numbers, the parameters, which are what the model learned. To use it, you need an app that loads that file and lets you chat with it.

Why run it at home?

• Privacy: your questions and documents never leave your machine.
• Offline: once the model is downloaded, no internet needed.
• No subscription: many models and apps are free to download.
• Freedom: you pick the model and keep it; nobody changes it behind your back.

The limits, honestly

Models that fit on a personal computer are smaller than those behind the big online services. They are very useful for summarising, rewording, writing, translating or helping with code, but less reliable on very specialised questions. And everything depends on your hardware: that is exactly what this site helps you check.

The one question that matters: memory

To answer quickly, the model must be loaded entirely into fast memory. Think of a workbench: if every piece of the puzzle fits on it, you work fast; if you have to fetch pieces from the attic each time, everything slows down.

Depending on your machine, that workbench has a different name:

• PC with a graphics card: VRAM, the graphics card's own memory (8, 12, 16, 24 GB…). It is the fastest.
• Apple Silicon Mac (M1 to M5 chips): unified memory, shared by the processor and the graphics chip. macOS keeps part of it: on a 16 GB Mac, the model gets about 10.7 GB by default.
• PC without a graphics card: regular RAM. It works, but much more slowly.

To give you an idea, according to our engine and with a normal-sized conversation (8K context): a small model such as Llama 3.1 8B Instruct needs about 6.7 GB, a big one such as Llama 3.3 70B Instruct about 45.8 GB. The full calculation is in How much VRAM do you need for a local LLM?

Model size: the "B"

Model names often contain a number followed by a B: Llama 3.1 8B, Qwen3 32B… B stands for billion: 8B means 8 billion parameters.

The rule is simple: the bigger the number, the more capable the model… and the more memory it needs. To start, a model of about 4 to 14B is a very good choice.

You will also see names like 30B-A3B. These are "mixture of experts" (MoE) models: the model is big, but it only uses a small part of its parameters for each token (here about 3B), which makes it much faster than its size suggests.

Compression: quantization

A raw model is too heavy for most machines, so it is compressed, a bit like a JPEG photo: that is quantization. You read it in the file name: Q8, Q6, Q5, Q4, Q3…

The smaller the number, the lighter the file, with slightly less quality. Example with Llama 3.1 8B Instruct: about 10.2 GB at Q8_0, about 6.7 GB at Q4_K_M.

The safe choice: Q4_K_M. It is the most common trade-off, with a small loss in quality. And a medium model at Q4 beats a huge model squeezed down to Q2. To go further: Q4 vs Q5 vs Q8: which quantization?

Speed: tokens per second

Speed is measured in tokens per second (tok/s): how many pieces of text the model writes each second. These are the thresholds our engine uses:

• 10 tok/s or more: comfortable for chatting.
• Between 5 and 10 tok/s: usable, with a little patience.
• Under 5 tok/s: slow; under 2 tok/s, we advise against it.

Speed depends less on computing power than on memory bandwidth: how fast the machine can read the whole model again for every token. That is why a graphics card is much faster than a processor alone. More in CPU vs GPU: why memory bandwidth decides.

What can my machine run?

A few examples computed by our engine: for each machine, the largest model in our catalogue that fits entirely in fast memory.

Your machine is not listed? The configurator checks any graphics card or Mac in seconds.

How to read our results

For each model, the site gives a verdict. Here is what it means in plain words:

Excellent

Fits in fast memory with room to spare, and runs fast. Go for it.

Compatible

Fits in fast memory. Works well.

Compatible with limits

Fits, but only just. If loading fails, shorten the conversation or pick a lighter quantization.

CPU/GPU offload

Does not fit entirely on the graphics card: part of it goes to RAM. Works, but more slowly.

Possible but slow

It starts, but the answer comes out slowly.

Not recommended

Possible in theory, but too slow to be pleasant. Pick a smaller model.

Incompatible

Not enough memory: the model cannot load.

Unknown

We are missing data to decide. We would rather say so than guess.

Our speeds are estimates, given as a range with a confidence level. When a real test exists on the same hardware, it is labelled "measured". Everything is detailed in our method.

Get started in 4 steps

1. Check your machine. Enter your graphics card or Mac in the configurator: it lists the models that run, with the memory they need and an estimated speed.
2. Install an app. The easiest way to start is LM Studio, an app with a chat window. Ollama is another popular option that many apps can connect to. The curious can look at llama.cpp, the engine many tools are built on.
3. Download a model. From the app, pick a model the configurator rates "Excellent" or "Compatible" for your machine, ideally at Q4_K_M. The file weighs several gigabytes: make room on your disk.
4. Chat. Ask your questions as with any assistant. Everything happens on your machine.

Beginner traps

• Picking the biggest model possible. A model that spills out of fast memory becomes very slow. A slightly smaller model that fits entirely is better.
• Forgetting the conversation length. The longer the conversation (the context), the more memory it takes. A model that fits for a short chat may not fit with a long document.
• Mixing up the Mac's memory and the model's memory. On a 16 GB Mac, the model only gets about 10.7 GB by default. See unified memory explained.
• Mixing up RAM and VRAM. 32 GB of RAM does not replace a graphics card: the model will run, but slowly.

Local LLM glossary

LLM

Large Language Model. The assistant's brain: a big file that predicts how a text continues.

Token

A piece of text: short word, part of a word or punctuation. Models read and write token by token.

Parameters (B)

The numbers the model learned. 8B = 8 billion parameters. The more there are, the more capable and memory-hungry the model.

VRAM

The graphics card's memory. Very fast: on a PC, this is where the model needs to fit. Learn more

Unified memory

On Apple Silicon Macs, the memory shared by the processor and the graphics chip. macOS keeps part of it for itself. Learn more

RAM

The computer's main memory. Much slower than VRAM for running a model.

Quantization

Compressing a model (Q8, Q5, Q4…). Less memory, slightly less quality. Q4_K_M is the most common choice. Learn more

GGUF

The most widespread file format for local models, used by llama.cpp, LM Studio and Ollama among others.

Context

How much text the model keeps in mind: your conversation and the documents you give it. Counted in tokens (8K ≈ 8,000 tokens).

KV cache

The memory taken by the current conversation. It grows with the context.

Offload

When the model does not fit on the graphics card, part of it is placed in RAM. Works, but more slowly.

MoE (mixture of experts)

A model that activates only part of its parameters for each token, which makes it faster than its size suggests.

Memory bandwidth

How fast the machine reads its memory, in GB/s. It sets how fast the model writes. Learn more

Tokens per second (tok/s)

How fast the model writes.

Inference

Running a model to get an answer, as opposed to training it.

Backend (CUDA, ROCm, Metal…)

The technology that puts the graphics chip to work: CUDA for NVIDIA, ROCm for AMD, Metal for Apple.

What next?

EstimateData updated 24 days agoSource: Run LLM Locally engine