Your VRAM Is Gone and the Model Weights Didn’t Do It
You load a 13B model. It fits fine — 8 GB on an RTX 3090, comfortable. Then you start a long conversation, feed it a 30K-token document, ask it to summarize and reason across the whole thing. Suddenly the GPU is gasping, inference slows to a crawl, and you’re either getting OOM errors or watching tokens/sec crater into single digits.
The model didn’t grow. The weights are the same. What happened?
The KV cache happened. And if you’re not quantizing it, you’re leaving a massive amount of VRAM on the table every single time you run inference at anything beyond a few thousand tokens.
Here’s the thing: most guides focus obsessively on model quantization — Q4_K_M, Q5_K_S, GGUF formats, all that. And yeah, quantizing model weights matters for fitting models in VRAM. But once the model is loaded, the dominant VRAM consumer at long context lengths isn’t the weights anymore. It’s the cache. And you can quantize that too.
What the KV Cache Actually Is
When a transformer generates tokens, it runs attention over every previous token in the sequence. That means for token N, it needs to “look at” tokens 1 through N-1. Recomputing those attention keys and values from scratch every single step would be wildly expensive — O(n²) work per token.
So instead, transformers cache the key (K) and value (V) tensors from every previous layer and every previous token. That’s the KV cache. It trades memory for compute: you pay VRAM up front, and attention becomes a lookup instead of a full recompute.
Great trade. Until the context gets long.
The math that ruins your day
The KV cache size formula is:
KV cache bytes = 2 × layers × num_heads × head_dim × seq_len × bytes_per_elementThe 2 is for K and V — both tensors get cached. Let’s plug in a real model. Llama 3.1 8B:
- 32 layers
- 32 KV heads (after GQA it’s actually 8, but let’s be precise: 8 KV heads)
- Head dimension: 128
- Sequence length: 32,768 tokens (32K context)
- Float16 (2 bytes per element)
2 × 32 × 8 × 128 × 32768 × 2 bytes = 4,294,967,296 bytes = ~4 GBFour gigabytes. For a single request. On a model whose weights fit in ~5 GB quantized. At 32K context you’ve nearly doubled your VRAM requirement just from the cache.
Scale that to a 70B model with 64 layers and you’re well past the point where a 24 GB card starts crying. If you want to understand the full VRAM picture, the GPU memory math article goes deeper on all the pieces.
Now here’s the good news: those K and V tensors don’t need to be stored in full float16 precision. Not even close.
What KV Cache Quantization Does
Standard KV cache stores everything in float16 or bfloat16 — 2 bytes per element. Quantizing to int8 (Q8) cuts that to 1 byte. Quantizing to float8 (FP8) also hits 1 byte. Going to Q4 gets you 0.5 bytes per element.
Think of it like vacuum-packing leftovers in the fridge. The food is still there, still food, still gets the job done — you’ve just removed the air and compressed the container. The KV cache is the same: you lose a small amount of precision but the attention mechanism still works because the errors introduced by quantization are small relative to the signal.
Results by quantization level on that same Llama 3.1 8B at 32K context:
| Cache dtype | Size | VRAM vs FP16 |
|---|---|---|
| float16 | ~4 GB | baseline |
| Q8_0 | ~2 GB | -50% |
| Q4_0 | ~1 GB | -75% |
That’s not a rounding error. That’s a free extra 2-3 GB of headroom, which at a 32K context is the difference between fitting the request and not.
llama.cpp: --cache-type-k and --cache-type-v
llama.cpp has had KV cache quantization for a while and it’s dead simple to enable. The flags are --cache-type-k and --cache-type-v. Valid types: f16, f32, q8_0, q4_0, q4_1, q5_0, q5_1.
# Q8 for both K and V — good default, very safellama-server \ --model /models/llama-3.1-8b-instruct.Q4_K_M.gguf \ --ctx-size 32768 \ --cache-type-k q8_0 \ --cache-type-v q8_0 \ --n-gpu-layers 99# Q4 for both — more aggressive, more VRAM savedllama-server \ --model /models/llama-3.1-8b-instruct.Q4_K_M.gguf \ --ctx-size 65536 \ --cache-type-k q4_0 \ --cache-type-v q4_0 \ --n-gpu-layers 99The --n-gpu-layers 99 just means “offload everything to GPU.” Adjust to your hardware.
For llama-cli (interactive mode), same flags:
llama-cli \ --model /models/llama-3.1-8b-instruct.Q4_K_M.gguf \ --ctx-size 32768 \ --cache-type-k q8_0 \ --cache-type-v q8_0 \ -iYou actually want flash attention on
One gotcha that trips people up: KV cache quantization basically wants flash attention enabled (-fa, or --flash-attn). Without it, llama.cpp has to dequantize the cache back to float for every attention computation — which can end up slower than just not quantizing at all. So the flags travel together. Recent llama.cpp builds will turn flash attention on automatically when you quantize the cache, but it doesn’t hurt to be explicit:
llama-server \ --model /models/llama-3.1-8b-instruct.Q4_K_M.gguf \ --ctx-size 32768 \ --flash-attn \ --cache-type-k q8_0 \ --cache-type-v q8_0 \ --n-gpu-layers 99The short version: if you’re quantizing the KV cache, leave flash attention on. They’re a package deal.
vLLM: --kv-cache-dtype fp8
If you’re running vLLM for serving (OpenAI-compatible API, batching, etc.), the flag is different but the concept is the same:
python -m vllm.entrypoints.openai.api_server \ --model meta-llama/Llama-3.1-8B-Instruct \ --kv-cache-dtype fp8 \ --max-model-len 32768 \ --gpu-memory-utilization 0.90vLLM uses FP8 rather than integer quantization formats. FP8 is 1 byte per element, same as int8, but preserves more of the floating-point dynamic range. On Hopper (H100) and Ada (4090, 4080) GPUs, FP8 also gets hardware acceleration. On older cards it emulates in software, but the memory savings are still real.
You can also set it via the Python API if you’re embedding vLLM:
from vllm import LLM, SamplingParams
llm = LLM( model="meta-llama/Llama-3.1-8B-Instruct", kv_cache_dtype="fp8", max_model_len=32768,)Real Numbers: 24 GB Card, 32K Context
Here’s what this looks like in practice on an RTX 3090 (24 GB) running Llama 3.1 8B Instruct (Q4_K_M weights, ~5 GB):
| Config | Context | VRAM used | tokens/s (gen) |
|---|---|---|---|
| No KV quant (f16) | 8K | ~9 GB | ~42 tok/s |
| No KV quant (f16) | 32K | ~17 GB | ~28 tok/s |
| Q8_0 KV cache | 32K | ~13 GB | ~40 tok/s |
| Q4_0 KV cache | 32K | ~11 GB | ~41 tok/s |
| Q4_0 KV cache | 65K | ~16 GB | ~32 tok/s |
A couple of things to notice. First: Q8 KV cache at 32K context uses about the same VRAM as no-quant at 8K context. You’ve effectively quadrupled your usable context window for the same VRAM budget. Second: tokens/s actually goes up slightly with KV quantization because you’re doing less memory bandwidth work per attention step. The cache fits better in GPU L2, attention memory reads are smaller, and the overall bottleneck moves from memory bandwidth back toward compute.
The quality hit? Negligible at Q8. Barely measurable at Q4 on most tasks. You’d need a very long context, very precision-sensitive task (think: complex multi-hop reasoning chains over 50K tokens) to see meaningful degradation. For summarization, chat, code gen, and document Q&A — you won’t notice.
What Not to Quantize
This comes up: can you just crank everything to Q4 everywhere and call it a day? Mostly yes, but there are cases to be careful about.
Small models. On a 3B or 7B model, the KV cache is already smaller and the model has less “buffer” to absorb quantization noise. Q8 is a safer floor. Q4 on a 3B model at very long contexts can produce noticeably worse coherence.
Very aggressive Q on K specifically. The Key tensors are used to compute attention scores — they determine what the model pays attention to. The Value tensors are just what gets retrieved after attention is computed. In practice this means K quantization errors can be slightly more damaging than V quantization errors at the same level. Some people run --cache-type-k q8_0 --cache-type-v q4_0 — keep K at Q8, let V be more aggressive. It’s a reasonable middle ground if you’re VRAM-constrained but nervous about quality.
If you’re benchmarking or evaluating a model. Run with full-precision KV cache. You don’t want quantization artifacts polluting your eval results. Production is different from measurement.
The One-Liner Summary
If you’re running llama.cpp and you’re not passing --cache-type-k q8_0 --cache-type-v q8_0, you’re leaving 50% of your KV cache VRAM on the floor every single inference run. It costs you almost nothing in quality. It costs you nothing in setup. It’s four words on a command line.
The KV cache is the silent VRAM hog that everyone ignores until they’re OOM at 32K context wondering what went wrong. Now you know what went wrong, and more importantly, you know how to fix it.
Add the flags. Run longer context. Stop buying more GPUs to solve a software problem.
Running into VRAM math confusion in general? The GPU memory math breakdown covers weights, activations, and optimizer states in full. For the model quantization side of the equation, LoRA and QLoRA fine-tuning explains how weight quantization works and where it fits in the picture.
