Your Support Bot Answered “How Do I Reset My Password” 400 Times This Week
Open your LLM request log and sort by prompt. A support bot, an internal docs Q&A bot, or a Home Assistant voice assistant gets the same dozen questions in slightly different clothes all day. Each one goes to the model, and each one costs tokens and a few seconds of latency.
An exact-match cache catches the lines that are byte-for-byte identical, which is fewer than you think. A semantic cache catches the paraphrases too, by comparing embeddings instead of strings. It also has a failure mode that exact-match caches do not: it can hand someone the wrong answer with total confidence. This post builds one by hand, shows the off-the-shelf options, and puts real numbers on the danger.
Full example: Clone the working files at github.com/KingPin/sumguy-examples/llm/llm-semantic-caching
Exact-Match vs Semantic: Two Different Bets
An exact-match cache hashes the whole request (model, messages, parameters) and looks the hash up in a key-value store. A hit means the request was identical, so replaying the answer is safe. The cost is a hash and a lookup. The miss rate on human-typed questions is high, because “How do I restart nginx?” and “how to restart nginx” are different strings.
A semantic cache embeds the query into a vector and compares it to the vectors of past queries. If the cosine similarity clears a threshold, it returns the stored answer. The hit rate is much higher. The safety guarantee is gone, because “close in embedding space” is not the same as “same question.”
Use exact-match for anything where the input is machine-generated or templated (cron jobs, batch classification, repeated tool prompts). Use semantic caching only for single-shot, human-typed questions where a slightly stale answer is acceptable.
One more distinction, because the names collide. Provider prompt caching (Anthropic and OpenAI prefix caching) gives you a discount on the repeated front of a long prompt. You still get a fresh generation every time. A semantic cache skips the model call entirely. They stack, and the math for the provider kind is in Context Window Math: Paying to Remember. This post is only about the skip-the-call kind.
The Danger: Near-Misses Look Like Paraphrases
Embedding models are trained to put related text close together. “Restart” and “stop” are related. So are “enable” and “disable”, “Postgres 16” and “Postgres 17”, and “web01” and “db01”. That is a feature for search and a trap for caching.
Here is a real run. The kit’s score_pairs.py embeds each pair with two small models through fastembed and prints the cosine similarity. These are the numbers from results.txt, not recalled ones:
== BAAI/bge-small-en-v1.5 ==0.971 PARAPHRASE 'How do I restart nginx?' vs "What's the command to restart nginx?"0.965 PARAPHRASE 'How do I reset my router password?' vs 'I forgot my router password, how can I reset it?'0.953 PARAPHRASE "How do I renew a Let's Encrypt certificate?" vs "Renewing a Let's Encrypt cert, what do I run?"0.958 PARAPHRASE 'Turn on the living room lights' vs 'Switch on the lights in the living room'0.890 NEAR-MISS 'How do I restart nginx?' vs 'How do I stop nginx?'0.907 NEAR-MISS 'How do I enable the firewall?' vs 'How do I disable the firewall?'0.936 NEAR-MISS 'How do I upgrade to PostgreSQL 16?' vs 'How do I upgrade to PostgreSQL 17?'0.892 NEAR-MISS 'How do I SSH into web01?' vs 'How do I SSH into db01?'0.882 NEAR-MISS 'Turn on the living room lights' vs 'Turn off the living room lights'0.900 NEAR-MISS 'Delete the backup from Monday' vs 'Restore the backup from Monday'0.387 UNRELATED 'How do I restart nginx?' vs 'What is the capital of France?'The second model, all-MiniLM-L6-v2, scored the paraphrases between 0.903 and 0.964 and the near-misses between 0.685 and 0.899. The worst paraphrase (0.903) and the best near-miss (0.899) are 0.004 apart.
Three things follow from the table.
- The LiteLLM docs example uses
similarity_threshold: 0.8. All six near-misses clear 0.8 on bge-small, and four of six clear it on MiniLM. At 0.8, “turn off the lights” returns the cached “turn on the lights” answer. - The gap between “same question” and “opposite question” is a few hundredths. Threshold tuning works on a margin of a few hundredths, and you re-check it every time the embedding model or the traffic changes.
- Version numbers are the worst case. “Postgres 16” vs “Postgres 17” scored 0.936 on bge, only 0.017 under the lowest paraphrase (0.953). A single digit barely moves the vector.
This is a sample of 11 pairs. It tells you the shape of the problem. It does not tell you your threshold. Tune on your own traffic.
A 45-Line Semantic Cache You Can Read in One Sitting
The mechanism is small enough to read. Embed the query, compare against stored query embeddings in the same namespace, return the stored answer if the best score clears the threshold, and drop entries past their TTL.
import hashlibimport mathimport time
def cosine(a, b): dot = sum(x * y for x, y in zip(a, b)) na = math.sqrt(sum(x * x for x in a)) nb = math.sqrt(sum(y * y for y in b)) return dot / (na * nb) if na and nb else 0.0
class SemanticCache: def __init__(self, embed, threshold=0.92, ttl=3600, clock=time.time): self.embed = embed # callable: str -> list[float] self.threshold = threshold # cosine SIMILARITY, higher = stricter self.ttl = ttl # seconds self.clock = clock self.entries = {} # namespace -> list of (vector, query, answer, expires_at)
@staticmethod def namespace(model, system_prompt): return hashlib.sha256(f"{model}\x00{system_prompt}".encode()).hexdigest()
def get(self, model, system_prompt, query): ns = self.namespace(model, system_prompt) now = self.clock() live = [e for e in self.entries.get(ns, []) if e[3] > now] self.entries[ns] = live q = self.embed(query) best, best_sim = None, 0.0 for vec, _, answer, _ in live: sim = cosine(q, vec) if sim > best_sim: best, best_sim = answer, sim if best is not None and best_sim >= self.threshold: return best, best_sim return None, best_sim
def put(self, model, system_prompt, query, answer): ns = self.namespace(model, system_prompt) self.entries.setdefault(ns, []).append( (self.embed(query), query, answer, self.clock() + self.ttl) )Two design choices matter more than the cosine math.
The namespace. The key is a hash of the model name plus the system prompt. Without it, a cached answer from your “terse sysadmin” bot gets served by your “friendly pirate” bot, or an answer from last month’s model gets served after you upgrade. Change the model or the prompt and you get a clean, empty cache. That is what you want.
The scan. get loops over every live entry in the namespace. That is O(n) and fine for a few thousand entries. Past that, use a vector index (Redis, Qdrant) so the nearest-neighbor search stops being your latency.
Wire it to a real model like this:
from fastembed import TextEmbeddingfrom semantic_cache import SemanticCache
model = TextEmbedding("BAAI/bge-small-en-v1.5")embed = lambda t: [float(x) for x in next(iter(model.embed([t])))]cache = SemanticCache(embed, threshold=0.95, ttl=6 * 3600)
SYSTEM = "You are the internal docs bot."answer, sim = cache.get("gemma-4", SYSTEM, user_query)if answer is None: answer = call_llm(user_query) # your real call cache.put("gemma-4", SYSTEM, user_query, answer)The kit’s test_semantic_cache.py checks the logic with a fake bag-of-words embedder, so it runs with no model download. It asserts a hit on a reordered question, a miss on “stop” vs “restart”, isolation between two system prompts and two models, and expiry after the TTL. It passes.
Picking the Threshold Without Guessing
Look at the table again. On bge-small the paraphrases bottom out at 0.953 and the near-misses top out at 0.936. A threshold of 0.95 separates this sample, with 0.003 of margin on the paraphrase side. On MiniLM the whole gap is 0.004, so the line has to sit at 0.90 with almost no room on either side.
Two rules, in order.
- Set the threshold from your own near-misses, not from paraphrases. Collect 30 to 50 real queries from your logs. Write the dangerous sibling for each (flip the verb, change the number, swap the hostname). Score both sets with your embedding model. Put the threshold just above your highest near-miss.
- Accept a low hit rate. A threshold that cleanly rejects near-misses will also reject some honest paraphrases. That is the right trade. A missed hit costs you one model call. A false hit tells a user to stop the service they asked to restart.
If no threshold separates your two sets, the embedding model is too weak for the job, or the job is wrong for semantic caching. A bigger embedding model helps (Embedding Model Choice covers the options). Domains where one word flips the meaning (commands, medical, legal, anything with numbers) are the worst fit. For those, stay exact-match.
What Never Goes in the Cache
Skip semantic caching, or exclude the request, when any of these is true:
- User-specific context. “What’s my balance?” and “What’s my order status?” embed similarly across users and have different answers per user. If the prompt carries personal data, the cache key needs the user ID, or the request bypasses the cache.
- Tool calls. A response that is a tool call (“call
restart_servicewithnginx”) is an action, not an answer. Replaying it from a cache runs it again. - Anything with time or date. “What’s on my calendar today?” is correct for 24 hours at most. “Is the backup done?” is correct for seconds.
- High-temperature creative output. If you want variety, a cache defeats it.
- Agent loops. The next section covers why.
A Home Assistant voice assistant is a good example of both sides. “What’s the capital of Australia?” caches fine. “Turn off the porch light” must never come from a cache, because the right behavior is to run the action, and the cached text is not the action.
When Semantic Caching Is a Waste
Semantic caching pays only when the repeat rate is high. Measure first. Log queries for a week, embed them, and count how many have a neighbor above your threshold. If it is under 15 to 20 percent, the embedding call and the infrastructure cost more than they save.
Agents are the clearest misfit. A tool-calling loop resends the entire conversation each turn, plus a small delta. Consecutive turns are nearly identical text. The LiteLLM docs say this directly: consecutive turns embed about 0.99 similar, every turn matches the previous turn’s cache entry, and the agent replays a stale response, typically the same tool call over and over. Raising the threshold does not reliably fix it, per the same page. Their recommendation is to exclude agentic traffic, either by setting "cache": {"no-cache": true} in the metadata of the virtual keys your agents use, or by switching to an exact-match cache for them.
Long unique contexts fail for a different reason. A 30,000-token document plus a question never repeats, so the hit rate is zero. That is the case for provider prompt caching instead.
The Off-the-Shelf Route: LiteLLM
If you already run LiteLLM as a proxy, caching is a config block. As of November 2026 the latest release on PyPI is 1.108.0, and the docs list redis-semantic, valkey-semantic, and qdrant-semantic as the semantic cache types. These are the keys from the docs page, with the embedding model defined in model_list like any other:
model_list: - model_name: docs-bot litellm_params: model: openai/stub api_key: stub mock_response: "systemctl restart nginx" - model_name: cache-embedder litellm_params: model: openai/stub-embed api_base: http://embed-stub:8081/v1 api_key: stub
litellm_settings: cache: True cache_params: type: "redis-semantic" redis_url: os.environ/REDIS_URL similarity_threshold: 0.95 redis_semantic_cache_embedding_model: cache-embedder ttl: 3600Swap the stub models for your real chat and embedding models. For Qdrant the keys are qdrant_semantic_cache_embedding_model, qdrant_collection_name, qdrant_semantic_cache_vector_size (it must match your embedding dimensions), and similarity_threshold.
Redis needs vector search. Redis 8 ships the query engine in the main image, so plain redis:8 works. I started redis:8 and MODULE LIST showed the search module loaded (server 8.10.3), so no special image is required.
The kit’s compose.yaml runs LiteLLM, Redis 8, and a stub embedding server. I started it and sent the same request twice. Here is what happened, and it is the limit of what I tested:
== How do I restart nginx?x-litellm-semantic-similarity: 0.0== How do I restart nginx?x-litellm-cache-key: bb84cee2db6b3ce854e3ad44728fadb164a9531c9a157f015ddd9ba27dbf4db8x-litellm-semantic-similarity: 0.9999999403953552The second response carried the same completion id as the first and a cache key header. That is a hit. The chat model was a mock_response and the embedder was a toy hashed bag of words, so this proves the wiring, not the threshold behavior. I did not test it against a real chat model or a real embedder, and I did not test the Qdrant or Valkey backends.
Two gotchas from the run. The current image refuses to start without a master key (LITELLM_MASTER_KEY). And redis-semantic died at boot with Missing required Redis configuration: REDIS_PASSWORD until I set redis_url under cache_params.
One more detail from the docs: the semantic cache embeds the text of the entire messages array, system prompt included. A long system prompt dominates the vector, and everything starts to look similar. Keep the system prompt short, or consider caching at the application layer where you control what gets embedded.
The Other Off-the-Shelf Route: RedisVL
RedisVL (version 0.30.0 on PyPI) has a SemanticCache class if you want the cache inside your own code. Its constructor takes name, distance_threshold (default 0.1), ttl, vectorizer, filterable_fields, and redis_url. The methods are check(prompt=..., num_results=1) and store(prompt=..., response=...).
The inversion is real. LiteLLM takes a similarity threshold, where higher is stricter. RedisVL takes a distance threshold in Redis cosine units (0 to 2), where lower is stricter. For normalized vectors, distance is 1 - similarity. A LiteLLM threshold of 0.90 is a RedisVL distance_threshold of 0.10. Copy a number between them without converting and you will either cache nothing or cache everything.
from fastembed import TextEmbeddingfrom redisvl.extensions.cache.llm import SemanticCachefrom redisvl.utils.vectorize import CustomTextVectorizer
model = TextEmbedding("sentence-transformers/all-MiniLM-L6-v2")vec = CustomTextVectorizer( embed=lambda t, **kw: [float(x) for x in next(iter(model.embed([t])))], embed_many=lambda ts, **kw: [[float(x) for x in v] for v in model.embed(ts)],)cache = SemanticCache(name="demo", vectorizer=vec, distance_threshold=0.10, ttl=3600, redis_url="redis://localhost:6379", overwrite=True)cache.store(prompt="How do I restart nginx?", response="sudo systemctl restart nginx")for q in ["What's the command to restart nginx?", "How do I stop nginx?"]: hit = cache.check(prompt=q, num_results=1) print(q, "->", (hit[0]["response"], round(float(hit[0]["vector_distance"]), 3)) if hit else "MISS")I ran this against redis:8. Output:
What's the command to restart nginx? -> ('sudo systemctl restart nginx', 0.051)How do I stop nginx? -> MISSA distance of 0.051 matches the MiniLM similarity of 0.949 from the earlier table. The “stop nginx” near-miss (similarity 0.756 on MiniLM) missed, which is correct. RedisVL does not namespace by model or system prompt for you. Use filterable_fields and a filter on a hash of both, or give each model and prompt its own cache name.
A Word About GPTCache
GPTCache from Zilliz is the project that popularized this idea. Its last push on GitHub was 2026-09-22 and its latest tagged release is 0.1.44 from 2024-08-01, so it has had no release in over two years. I did not evaluate it for this post. If you adopt it, pin the version and expect to read the source when something breaks.
TTL and Invalidation
Every cached answer is a claim about the world at one moment. Treat it that way.
- Set a TTL on everything. For docs Q&A, hours to a day. For anything touching live state, seconds or skip caching.
- Flush on source change. If the docs the bot answers from get re-indexed, clear the namespace. A model swap or system prompt edit already clears it in the hand-rolled version because the namespace hash changes.
- Keep a delete path. When someone reports a wrong answer, you need to remove that entry by hand without nuking the cache.
- Log hits with their similarity score. Review hits in the 0.90 to 0.95 band weekly. Those are your future incidents.
Your 2 AM self will appreciate a log line that says which cached query produced which answer.
Common Questions
What similarity threshold should I use for semantic caching?
Start at 0.95 cosine similarity with a small embedding model like bge-small, then tune it on your own queries. In the sample here, paraphrases scored 0.953 and up and near-misses topped out at 0.936. Set the threshold just above your highest dangerous near-miss, not at a generic value.
Do I need a vector database for semantic caching?
No. A few thousand cached queries fit in a Python list with a linear cosine scan, as the hand-rolled cache above shows. Move to Redis 8, Qdrant, or Valkey with the search module once the scan gets slow or you need the cache shared across several workers or processes.
Is semantic caching the same as Anthropic or OpenAI prompt caching?
No. Provider prompt caching discounts the repeated prefix of a long prompt, and the model still generates a fresh answer. Semantic caching skips the model call and returns a stored answer for a similar query. The two work together, and prompt caching carries no risk of false hits.
Why does RedisVL use distance while LiteLLM uses similarity?
RedisVL reports Redis cosine distance, where 0 means identical and lower is stricter. LiteLLM uses cosine similarity, where higher is stricter. For normalized vectors, distance equals one minus similarity, so a LiteLLM threshold of 0.90 equals a RedisVL distance_threshold of 0.10.
Can I use semantic caching with an agent or tool-calling loop?
No. Agent loops resend the whole conversation each turn, so consecutive turns embed about 0.99 similar and a semantic cache typically replays stale responses. The LiteLLM docs recommend excluding agent traffic from semantic caching and using an exact-match cache for it.
