Pit stop · 2026-10-01 · Checked 2026-10-10

Out of memory: what to try, in order

A model needs room for three things: its weights, the context cache, and some working buffers. When the total is bigger than your VRAM, something has to give. Try these in order.

1. Shorten the context

The context cache grows with every token of context. Going from 32K to 8K can free several GB.

llama-server -m model.gguf -c 8192

2. Quantize the context cache

llama-server -m model.gguf -fa on --cache-type-k q8_0 --cache-type-v q8_0

The cache takes about half the room, with very little quality loss. Quantizing the V cache needs FlashAttention (-fa on).

3. Close what else uses the card

nvidia-smi

A desktop session and a browser can hold 0.5 to 1.5 GB. On a headless machine that memory is yours.

4. Drop one quant level

Q4_K_M to IQ4_XS saves about 12 % of the weights. Below Q3_K_M, quality drops quickly. See Which quant should I pick.

5. Mixture-of-experts: move experts to RAM

llama-server -m model.gguf -ngl 99 --n-cpu-moe 8

6. Offload fewer layers

llama-server -m model.gguf -ngl 30

The remaining layers run on the CPU. It works, but speed falls sharply.

7. With Ollama

OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 OLLAMA_CONTEXT_LENGTH=8192 ollama serve

Not sure where you stand? Will it fit shows the split between weights, cache and buffers for your cards.

When it is not memory at all

A server that stops at startup having allocated nothing is not an out-of-memory case. If the message says the card is too old, or names a minimum compute capability or architecture, the engine was simply not compiled for that card: the model may fit perfectly and never be loaded. Check your card’s generation against the engine’s minimum on The architecture ladder.

Questions

Why does a model that fits still fail on a long prompt?
VRAM holds three things at once: the weights, the context cache and some working buffers. The cache grows with every token of context, so a model that loads easily at 8K can fail at 32K.
What is the first fix to try when a model will not load?
Shorten the context. Going from 32K to 8K can free several GB and it is the cheapest change there is: llama-server -m model.gguf -c 8192.
Can the context cache be compressed without hurting quality?
Yes. --cache-type-k q8_0 --cache-type-v q8_0 with -fa on roughly halves the cache for very little quality loss. Quantising the V cache needs FlashAttention turned on.
How do I run a model that is bigger than my VRAM?
Offload fewer layers (-ngl 30), or for a mixture-of-experts model keep the experts in RAM (--n-cpu-moe 8) and let the GPU hold the rest. It runs, but generation speed falls sharply.