Context Window Math: Paying to Remember

Your 50-turn chat did not cost 50 messages

You ask a model a question. It answers. You ask a follow-up. It answers again. Fifty turns later, the bill says you sent a million input tokens, and you typed a few pages of text.

The model has no memory between calls. Every request carries the whole conversation: system prompt, tool definitions, every earlier message, every tool result. Turn 40 pays for turns 1 through 39 again, and so did turn 39. Add up the input tokens across the session and the total grows roughly with the square of the turn count. The context you “keep around” is billed again on every call.

Full example: Clone the working files at github.com/KingPin/sumguy-examples/tree/main/llm/context-window-math.

You pay for that in one of two currencies. On a hosted API you pay per token, and prompt caching can cut the repeat cost by 90% or more if you respect its rules. On a local box you pay in VRAM, because the KV cache grows linearly with context length and servers reserve it up front. The scripts below run both calculations. If you want the definitions first, context window vs output limit is the short version.

The quadratic you did not sign up for

Say a session starts with S tokens of system prompt and tool definitions. Each turn adds p tokens of new material (your message plus the reply), and the reply is part of that. On turn t, the request carries S, plus everything from the t minus 1 earlier turns, plus your new message.

Summed over N turns, the input tokens come to:

total_input = N * (S + new_user_part) + p * N * (N - 1) / 2

The first term is linear. The second term is the one that hurts. Double the turns and you roughly quadruple the second term.

The companion script runs this as a plain loop and checks it against the closed form in a self-test. Here is a 50-turn session with a 3,000-token system prompt and 700 tokens added per turn (300 of them model output), priced at Claude Sonnet 5.5 rates as of October 2026:

python3 context_cost.py --turns 50 --system 3000 --per-turn 700 --out 300 \
  --in-price 2 --out-price 10 --read-mult 0.05 --write-mult 1.25

turns=50  first input=3,400  last input=37,700 tokens
cumulative input tokens: 1,027,500
cost, no caching:   $2.2050
cost, with caching: $0.3432  (16% of uncached)
  turn   10: sends     9,700 input tokens
  turn   20: sends    16,700 input tokens
  turn   30: sends    23,700 input tokens
  turn   40: sends    30,700 input tokens
  turn   50: sends    37,700 input tokens

Fifty turns, and you moved over a million input tokens through the API. The last request alone is 11 times the size of the first. Nothing is wrong. That is how a stateless API works.

Now stretch it to 200 turns, which is a normal afternoon for a coding agent:

python3 context_cost.py --turns 200

turns=200  first input=3,400  last input=142,700 tokens
cumulative input tokens: 14,610,000
cost, no caching:   $29.8200
cost, with caching: $2.4035  (8% of uncached)

Four times the turns, 14 times the input tokens. The output side grew only linearly: 200 turns times 300 tokens is 60,000 output tokens, about $0.60 at $10 per MTok. Input is where the money goes, and the history is the input.

Agents make this worse than chat does, because tool results are fat. A file read or a command’s log output lands in the history at thousands of tokens each, and stays there. With 4,000 tokens added per turn instead of 700 (500 of them model output), a 50-turn run sends 5,225,000 input tokens and costs $10.70 uncached at the same rates, against $1.26 with caching:

python3 context_cost.py --turns 50 --per-turn 4000 --out 500

The API bill, and what caching actually does

Prompt caching is a discount on the part of the request that matches a previous request. The server remembers the processed prefix, so you pay a reduced rate to reuse it. The shape of the deal differs by vendor, so read the order form, not the blog posts. These are the numbers from each vendor’s pricing and caching docs as of October 2026, per million tokens:

Anthropic’s default cache lifetime is 5 minutes, refreshed free each time the entry is read. A 1-hour lifetime exists at 2 times the base input price for the write. Cache writes cost 1.25 times base for the 5-minute tier. Reads cost 0.1 times base on most models, but Opus 5.5 and Sonnet 5.5 sit at 0.05 times, and Fable 5.1 at 0.025 times. The minimum cacheable prompt is 512 tokens on those models; shorter prompts are processed normally with no error.

OpenAI’s page describes the same idea with different knobs. For GPT-5.6 and later, cached reads cost 0.1 times the input rate (0.05 times on GPT-6.1 Sol and GPT-6.1 Terra). Writes cost 1.25 times for a 30-minute entry, which is the default, and 2.5 times for a 12-hour entry. The minimum is 1,024 tokens. Older OpenAI models cache implicitly with no write premium. The current GPT-6.1 Sol lists at $2 input, $0.10 cached, $10 output, which lines up with Sonnet 5.5 almost exactly.

DeepSeek is the cheap end. On its pricing page, deepseek-flash bills cache hits at $0.003 per MTok off-peak ($0.006 peak) against $0.15 ($0.30 peak) for a miss, with output at $0.60 off-peak ($1.20 peak). Peak hours run 01:00 to 04:00 and 06:00 to 10:00 UTC on weekdays. The page does not document a write premium or a TTL, so I will not quote one. The deepseek-flash name now serves DeepSeek-V4.1-Flash-1017, and the page lists the older deepseek-v4-flash models as retired, so launch-day prices for V4 Flash 0731 no longer apply.

Reading the savings

In the 200-turn run, caching took the session from $29.82 to $2.40. The same session on Claude Opus 5.5 rates goes from $59.64 to $4.81, which is the same 8% ratio at double the price:

python3 context_cost.py --turns 200 --in-price 4 --out-price 20 \
  --read-mult 0.05 --write-mult 1.25

cost, no caching:   $59.6400
cost, with caching: $4.8070  (8% of uncached)

The model assumes the best case: every call lands inside the TTL and every call only appends. Break that and the picture flips. If every call is a cache write because something early in the prompt changed each time, you pay the 1.25 times premium on the entire history and get no reads. The script models that when you set the read multiplier equal to the write multiplier:

python3 context_cost.py --turns 200 --read-mult 1.25

cost, no caching:   $29.8200
cost, with caching: $37.1250  (124% of uncached)

On Anthropic and on OpenAI GPT-5.6 and later, a broken cache is not a neutral state. It costs more than having no cache at all, because you pay the write premium on every call and never collect a read.

The prefix rules

Caching matches a prefix, byte for byte. Anthropic renders the request in the order tools, then system, then messages, and a change at one level invalidates that level and everything after it. Editing a tool definition busts the whole cache. Toggling web search or citations keeps the tool cache but busts the system and message caches. Changing thinking parameters invalidates the message blocks.

OpenAI says the same thing in different words: reuse needs the entire rendered prefix to match, and a change before a breakpoint prevents reuse of anything after it. Settings like reasoning.effort, text.verbosity, and tool lists can change the prefix.

That gives you the ordering rule for your own prompts:

1. Static system prompt and tool definitions first, and never reorder them.
2. Long-lived reference material next (a repo map, a style guide, a document you are asking questions about).
3. Conversation history after that, append-only.
4. The volatile stuff last: today’s date, a timestamp, a per-request ID, retrieved snippets that change every turn.

The classic mistake is a timestamp at the top of the system prompt. “Current time: 14:03:22” changes on every call, so the prefix never matches. Move it to the end of the latest user message and the cache survives.

Anthropic also has an automatic mode: one top-level cache_control field and the system moves the breakpoint to the last cacheable block as the conversation grows. For a plain chat loop that is the lazy, correct choice.

import anthropic

client = anthropic.Anthropic()

resp = client.messages.create(
    model="claude-sonnet-5-5",
    max_tokens=1024,
    cache_control={"type": "ephemeral"},  # automatic breakpoint, 5 min TTL
    system="You are a code reviewer. Follow the style guide below.\n" + STYLE_GUIDE,
    messages=history + [{"role": "user", "content": new_message}],
)

u = resp.usage
print(u.cache_creation_input_tokens, u.cache_read_input_tokens, u.input_tokens)

Always print the usage fields. If cache_read_input_tokens is zero on turn two, your prefix is changing and you are paying the premium for nothing. Also remember the lookback is limited: the system checks at most 20 positions back from each breakpoint, so an agent loop that appends more than 20 blocks between calls needs a second explicit breakpoint on the last block of the previous request.

The local bill is VRAM, and you pay it up front

Run the model yourself and nobody sends you an invoice per token. You pay in memory instead. Every token in the context stores a key and a value vector at every layer, and that KV cache lives in VRAM for as long as the sequence does. The formula:

bytes per token = 2 (K and V) x layers x kv_heads x head_dim x bytes_per_element

Take a model with plain grouped-query attention and no sliding window. I used Qwen3-8B, whose config.json on Hugging Face lists 36 hidden layers, 8 KV heads, a head dimension of 128, and bfloat16 weights. Models with sliding-window or hybrid layers need a per-layer calculation, so I picked one where the simple formula holds.

python3 kv_cache.py --layers 36 --kv-heads 8 --head-dim 128 --ctx 32768
python3 kv_cache.py --layers 36 --kv-heads 8 --head-dim 128 --ctx 131072

per token: 144.0 KiB
ctx=32,768 x slots=1: 4.50 GiB
per token: 144.0 KiB
ctx=131,072 x slots=1: 18.00 GiB

144 KiB per token, so a 32K context eats 4.5 GiB at f16 and 128K eats 18 GiB. That is before the weights. An 8B model in 4-bit is a few GB; the cache at 128K is bigger than the model. (Qwen3-8B’s native window is 40,960 tokens, so 128K means extending it with RoPE scaling. Use the number for sizing only.) Larger models cost more per token: Qwen3-32B has 64 layers with the same 8 KV heads and head dimension, which works out to 256 KiB per token and 8.00 GiB at 32K.

Linear growth sounds gentle next to the API’s quadratic bill, but it is a hard wall. The API charges you for tokens you actually send. A local server reserves memory up front, whether you use it or not.

Slots multiply it

Servers that handle several users reserve cache per parallel sequence. Four slots of 32K context is four times the memory:

python3 kv_cache.py --layers 36 --kv-heads 8 --head-dim 128 --ctx 32768 --slots 4

per token: 144.0 KiB
ctx=32,768 x slots=4: 18.00 GiB

In llama.cpp the context is set with -c / --ctx-size and the number of server slots with -np / --parallel. When you leave -np on auto, recent builds use a unified KV buffer shared across slots (--kv-unified), and --kv-unified-per-slot N without -c sizes that pool to slots times N. Read your build’s --help before you assume how --ctx-size divides among slots, because it changed across versions. In vLLM the knobs are --max-model-len for the longest sequence, --max-num-seqs for concurrency, and --gpu-memory-utilization for how much VRAM the engine claims. vLLM pre-allocates whatever is left after weights as KV blocks, so raising the memory fraction buys more concurrent context, not faster single requests.

The practical rule: set the context to what you need, not the model’s maximum. A 128K window “just in case” costs 18 GiB on this model, and most chats never pass 8K.

Quantize the cache, not your expectations

Storing K and V in 8-bit halves the cache, give or take the block scales. In llama.cpp, --cache-type-k q8_0 --cache-type-v q8_0 does it, with flash attention on. At roughly 1.0625 bytes per element for q8_0:

python3 kv_cache.py --layers 36 --kv-heads 8 --head-dim 128 --ctx 32768 --dtype-bytes 1.0625

per token: 76.5 KiB
ctx=32,768 x slots=1: 2.39 GiB

From 4.50 GiB to 2.39 GiB. The details, the quality tradeoffs, and the vLLM equivalent are in KV cache quantization, so I will not repeat them here.

Local servers also have a prefix cache. llama.cpp reuses the KV state of a matching prompt prefix across requests, and vLLM has automatic prefix caching, on by default in current releases (turn it off with --no-enable-prefix-caching). The same prefix-stability rules apply. You do not pay a dollar premium, but a busted prefix means recomputing the whole prompt, and on a long history that is seconds of prefill before the first token appears.

Fixes that work on both sides

Whichever currency you pay in, the same habits cut the bill.

Keep the prefix stable. Static content first, volatile content last, no edits to earlier history. If you rewrite turn 3 on turn 30, you throw away the cache from turn 3 onward and pay full price for 27 turns of text. The DeepSeek V4 Flash post makes the same point about append-only loops.

Clear or restart when the task changes. A fresh session with a three-line summary of what you decided beats dragging 90,000 tokens of finished debugging into an unrelated question. In Claude Code that is /clear. In any API loop it is starting a new messages list.

Compact on purpose. Summarizing old turns shrinks every future request, but it rewrites history, so it busts the cache once. Do it at a task boundary, once, rather than a little every turn. A compaction every 40 turns costs one cache write on the new prefix. A “helpful” rolling summary that rewrites turn 2 on every call costs a cache write on every call.

Trim tool output before it enters the history. A 6,000-token test log that you only needed the last 20 lines of will ride along for the rest of the session. Filter at the source: tail, grep, a --quiet flag, a wrapper that truncates. Tools built for this are covered in RTK vs snip vs lean-ctx.

Match the TTL to your rhythm. The 5-minute tier stays alive while you keep calling. If you step away for 10 minutes, the next call pays the write premium on the whole history again. For slow human-in-the-loop sessions the 1-hour tier at 2 times base can be cheaper than repeated 1.25 times rewrites. For tight agent loops, the 5-minute default is fine.

Size local context to the job. Pick --ctx-size or --max-model-len from the longest conversation you actually run, check the KV math, and leave headroom for the weights and activations. Quantize the cache if you are tight.

The SumGuy take

Treat context like a lease. Every token you keep is rent due on every call: dollars on the API, gigabytes on your GPU. The cheapest token is the one you never send, and the second cheapest is the one the cache already holds.

Before you shop for a cheaper model, print the usage fields and check your cache-read count. A cache hit rate near zero on a long session is often a bigger lever than a model swap: on the 200-turn run, fixing it is the difference between $37.13 and $2.40. The fix is often small. Check it once, pin your system prompt, move the timestamp to the end, and your 2 AM self will appreciate it.

Common Questions

Does a bigger context window cost more if I don’t fill it?

On a hosted API, no. You pay only for the tokens you send, so an unused 1M window costs nothing. On a local server, yes. llama.cpp allocates KV cache for the full --ctx-size at startup, used or not. vLLM claims its whole --gpu-memory-utilization share up front, so a larger --max-model-len leaves room for fewer concurrent long sequences.

How long does a prompt cache last?

Five minutes on Anthropic and 30 minutes on OpenAI, by default. Anthropic refreshes the 5-minute clock free on each read and sells a 1-hour option. OpenAI GPT-5.6 and later default to 30 minutes, with a 12-hour option. After the TTL passes with no reads, the next request pays the write price again.

Can I cache a short system prompt?

Only above the minimum length. Anthropic requires 512 tokens on Claude Fable 5.1, Opus 5.5, Sonnet 5.5, and Haiku 5.5, and 1,024 to 4,096 on older models (4,096 on Claude Haiku 4.5). OpenAI requires 1,024 for GPT-5.6 and later. Shorter prompts run normally with no error and no discount, so a 300-token prompt never caches.

Why is my cache read count zero on the second turn?

Something earlier in the prompt changed. Common causes are a timestamp or request ID in the system prompt, a reordered or edited tool definition, a changed reasoning.effort or thinking setting, or a call that arrived after the TTL. Compare the two rendered requests byte for byte and move the changing part to the end.

Related Reading

• Context Window vs Token Limit: Not the Same Thing
• KV Cache Quantization: Free LLM Context, Almost
• Local Coding Agents Need Less Context
• RTK vs snip vs lean-ctx: Token Killers
• DeepSeek V4 Flash: 28 Cents a Million