Local LLM glossary
Every term explained in a sentence or two, with links to the guides that go deeper.
- LLM
- Large Language Model. The assistant's brain: a big file that predicts how a text continues.
- Token
- A piece of text: short word, part of a word or punctuation. Models read and write token by token.
- Parameters (B)
- The numbers the model learned. 8B = 8 billion parameters. The more there are, the more capable and memory-hungry the model.
- VRAM
- The graphics card's memory. Very fast: on a PC, this is where the model needs to fit. Learn more
- Unified memory
- On Apple Silicon Macs, the memory shared by the processor and the graphics chip. macOS keeps part of it for itself. Learn more
- RAM
- The computer's main memory. Much slower than VRAM for running a model.
- Quantization
- Compressing a model (Q8, Q5, Q4…). Less memory, slightly less quality. Q4_K_M is the most common choice. Learn more
- GGUF
- The most widespread file format for local models, used by llama.cpp, LM Studio and Ollama among others.
- Context
- How much text the model keeps in mind: your conversation and the documents you give it. Counted in tokens (8K ≈ 8,000 tokens).
- KV cache
- The memory taken by the current conversation. It grows with the context.
- Offload
- When the model does not fit on the graphics card, part of it is placed in RAM. Works, but more slowly.
- MoE (mixture of experts)
- A model that activates only part of its parameters for each token, which makes it faster than its size suggests.
- Memory bandwidth
- How fast the machine reads its memory, in GB/s. It sets how fast the model writes. Learn more
- Tokens per second (tok/s)
- How fast the model writes.
- Inference
- Running a model to get an answer, as opposed to training it.
- Backend (CUDA, ROCm, Metal…)
- The technology that puts the graphics chip to work: CUDA for NVIDIA, ROCm for AMD, Metal for Apple.