GPU VRAM Calculator
The weights are only part of it. The context cache grows with how much you feed the model, and it is what usually pushes a model off a card.
How the math works
Weights = parameters x bits / 8, in bytes. KV cache = 2 x layers x hidden size x context length x 2 bytes, the last 2 being FP16 storage for the cache. Add about 10 percent for activations and runtime overhead. GB here are binary, 1024^3 bytes, which is how GPUs report them.
Common questions
- How much VRAM do I need to run a 13B model?
- At 4 bit with a short context, around 9 to 10 GB, so it fits a 12 GB card. At FP16 it needs over 30 GB. Quantization is what makes local models practical on consumer hardware.
- What is the KV cache and why does it matter?
- It stores the attention keys and values for every token in the context so they are not recomputed each step. It grows linearly with context length, and at long contexts it can exceed the size of the model itself.
- How much quality is lost to quantization?
- Going from FP16 to 8 bit is close to free. 4 bit costs a small, usually acceptable amount. At 3 bit and below the degradation becomes obvious, and a smaller model at 4 bit is generally better than a larger one at 3.