Semantic Caching for AI Requests, Explained Simply

Four people ask your AI assistant the same thing:

"How do I reset my password?"
"how can i change my password"
"I forgot my password, what now?"
"password reset help please"

You and I can see those are one question. Your code sees four different strings, so it calls the model four times, waits four times, and pays four times.

A semantic cache fixes that. It stores answers by meaning instead of by exact text, so the last three people get the first person's answer, instantly and for free.

If you have called an AI model from code before, you know enough to follow this. Everything else gets explained as it comes up.

Why a normal cache does not help

The first idea everyone has is a normal cache, with the question as the key:

const cache = new Map<string, string>();

async function ask(question: string) {
  const hit = cache.get(question);
  if (hit) return hit;

  const answer = await callModel(question);
  cache.set(question, answer);
  return answer;
}

This code is correct. It also almost never helps, because the key has to match letter for letter:

cache.get("How do I reset my password?"); // hit
cache.get("how do I reset my password");  // miss, two characters different

You can tidy the text first, lowercase it and strip the punctuation, and you will catch a few more. You will still miss "I forgot my password", because that is entirely different words for the same thing.

Comparing text cannot see meaning. That is the whole problem, and the rest of this article is one way around it.

The idea: turn meaning into numbers

An embedding is a list of numbers that stands for the meaning of a piece of text. You send text to an embedding model, and it hands back an array of a few hundred to a few thousand numbers.

await embed("How do I reset my password?");
// [0.021, -0.114, 0.088, ... ]  a long list of numbers

await embed("I forgot my password");
// [0.019, -0.109, 0.091, ... ]  a very similar list

You never need to know what any single number means. One thing matters and nothing else does:

Similar meaning gives you similar numbers. Different meaning gives you different numbers.

Picture a map. Every sentence gets a pin dropped on it. Sentences about resetting passwords all land in the same neighbourhood. Sentences about refunds land on the other side of town.

So a semantic cache asks a different question than a normal cache. Not "have I seen this exact text before" but "have I seen anything that means this before".

Where embeddings come from

embed() is just an HTTP call to an embedding model, the same way callModel() is an HTTP call to a chat model. It is a different endpoint and usually a different, much cheaper model.

Note that Anthropic's API does not have an embeddings endpoint, so if you are building on Claude you get your embeddings from somewhere else. Any provider with an embeddings API will do, and you can also run a small open-source embedding model yourself. Whichever you pick, the shape is the same:

async function embed(text: string): Promise<number[]> {
  // call your embedding provider, return the array it gives you
}

Two rules to save you a confusing afternoon. Use the same model everywhere, because numbers from two different models are not comparable, and lists of different lengths cannot be compared at all. And embed the raw question, not a prompt you have wrapped around it, or every question will look similar because they share all that wrapper text.

Measuring how close two meanings are

You compare two embeddings with cosine similarity. It takes two lists of numbers and gives you one number back:

• 1.0 means the same direction, so the same meaning
• 0.0 means unrelated
• below 0 means opposite

Here it is in full. It is shorter than its name suggests:

function cosineSimilarity(a: number[], b: number[]): number {
  let dot = 0;
  let lenA = 0;
  let lenB = 0;

  for (let i = 0; i < a.length; i++) {
    dot += a[i] * b[i];      // multiply each pair, add them up
    lenA += a[i] * a[i];     // how long list a is
    lenB += b[i] * b[i];     // how long list b is
  }

  return dot / (Math.sqrt(lenA) * Math.sqrt(lenB));
}

Real embeddings have hundreds of numbers, which is impossible to picture, so try it with two:

cosineSimilarity([3, 4], [6, 8]);   // 1.0
cosineSimilarity([3, 4], [-4, 3]);  // 0.0

[6, 8] is [3, 4] doubled. It points the exact same way, just further, and cosine similarity gives it 1.0. The word "cosine" is doing one useful job here: it looks at direction only and ignores length. That is what you want, because you care whether two questions mean the same thing, not whether one of them is longer.

[-4, 3] points at a right angle to [3, 4], and the multiplied pairs cancel out to zero. Unrelated.

Then you pick a threshold. Above it you call it a match and return the cached answer. Below it you call the model.

The whole thing in five steps

Embed, search, compare, answer or call, store. Everything below is that same loop with the sharp edges filed off.

Building one

Start with the smallest thing that works. An array, a loop, and the two functions from above:

type Entry = {
  question: string;
  embedding: number[];
  answer: string;
};

const entries: Entry[] = [];
const THRESHOLD = 0.9;

async function findCached(question: string): Promise<string | null> {
  const embedding = await embed(question);

  let best: Entry | null = null;
  let bestScore = 0;

  for (const entry of entries) {
    const score = cosineSimilarity(embedding, entry.embedding);
    if (score > bestScore) {
      bestScore = score;
      best = entry;
    }
  }

  return best && bestScore >= THRESHOLD ? best.answer : null;
}

async function store(question: string, answer: string): Promise<void> {
  entries.push({ question, embedding: await embed(question), answer });
}

And the function your app actually calls:

async function ask(question: string): Promise<string> {
  const cached = await findCached(question);
  if (cached) return cached;

  const answer = await callModel(question);
  await store(question, answer);
  return answer;
}

That is a real semantic cache, and it will work. Now add one thing: cached answers should not live forever, so give each entry a timestamp and skip the old ones.

type Entry = {
  question: string;
  embedding: number[];
  answer: string;
  createdAt: number;   // Date.now() when we stored it
};

const TTL_MS = 60 * 60 * 1000; // one hour

// inside the loop in findCached:
if (Date.now() - entry.createdAt > TTL_MS) continue; // too old, ignore it

TTL means "time to live", the age at which an entry stops counting. An hour is a fine starting point for support-style answers.

Everything from here on is about not getting hurt by the cache you just built.

Picking the threshold

The threshold is the one number you will spend real time on, because it is a trade with a nasty side.

Set it too low and unrelated questions match. Someone asks about refunds and gets told how to reset their password. This is the failure that hurts, because nothing looks broken. No error, no exception, no alert. The user just gets a confident wrong answer.

Set it too high and almost nothing matches. You now pay for an extra embedding call on every request and get nearly no hits in return.

Do not take a number from a blog post, this one included. Start at 0.9, then log the similarity score of every lookup for a week, hits and misses both, with the two questions next to each other. Read that log. Your own data will show you very quickly where the real matches stop and the accidents start, because your users ask questions in a way that belongs to your product.

Two rules that will save you pain:

• A wrong hit costs far more than a miss. When you are unsure, go higher.
• Different kinds of question want different thresholds. "What is your refund policy" is safe to match loosely. "What is the balance on invoice 4471" should never match anything.

The dangerous part: what goes into the lookup

This is where people get hurt, so read this section twice.

Two questions can use identical words and still need different answers, because something outside the question decides the answer. Match on the question text alone and you will hand one customer another customer's answer.

// Don't
const cached = await findCached(question);

// Do
const cached = await findCached(question, {
  userId: user.id,
  tenantId: user.tenantId,
  locale: user.locale,
  model: "claude-opus-5",
  promptVersion: 7,
});

Anything that changes the answer belongs in the lookup, not just the meaning of the words.

The easy way to do this is buckets. Build one exact-match string out of the context, keep a separate list of entries per bucket, and only ever compare meaning inside a single bucket:

const buckets = new Map<string, Entry[]>();

function bucketKey(ctx: Context): string {
  return [ctx.tenantId, ctx.locale, ctx.model, ctx.promptVersion].join("|");
}

// then, in findCached:
const entries = buckets.get(bucketKey(ctx)) ?? [];

Now a question from tenant A can never reach an answer given to tenant B, however it is worded. Meaning gets compared only where comparing it is safe.

When not to cache at all

Some answers must never come from a cache, no matter how close the questions look:

• Anything personal. "How much do I owe?" is the same sentence for every customer and a different answer for each one.
• Anything about right now. Stock levels, prices, order status, live data. A correct answer from ten minutes ago is a wrong answer now.
• Anything the user is steering. "Make it shorter", "try again", "now in French". These depend on the conversation, not on the sentence.
• Anything creative. Handing someone a poem you already gave someone else defeats the point of asking for a poem.

Check for these before you touch the cache:

function isCacheable(question: string, ctx: Context): boolean {
  if (ctx.hasConversationHistory) return false;
  if (/\b(my|mine|our)\b/i.test(question)) return false; // probably personal
  return true;
}

That word check is crude and it will refuse to cache some things it could have cached. That is the right way round. Start strict and loosen it as you learn what your users actually ask.

Keeping answers fresh

A cached answer is a promise that nothing has changed since you stored it. Three ways to keep that promise, from least to most work:

Time. The TTL you already added. Every entry expires on its own.

Versions. When you change your system prompt, switch models, or update the documents your answers come from, bump a version number that is part of the bucket key. Old entries simply stop being found and expire quietly. This is much safer than working out which entries to delete, because getting that wrong leaves stale answers in circulation.

Events. When a source document changes, drop the entries built from it. This means storing which documents each answer came from, so it is the most work. It is also the most precise.

This is not the same as prompt caching

Two different things share the word "cache", and the mix-up is common enough to be worth its own section.

Prompt caching is a feature of the model provider. You mark part of your prompt, and the provider keeps its processed form so later requests skip re-processing it. On the Claude API this is cache_control. Cached reads cost roughly a tenth of normal input, and an entry lives five minutes by default, with a one hour option. The catch is that it is an exact prefix match: the cached part must be byte for byte identical, and it must sit at the front of the prompt. Put a timestamp near the top of your system prompt and you have quietly switched off caching for everything after it.

Semantic caching is something you build. It compares meaning, it lives in your own code, and on a hit it makes no model call at all.

They are not alternatives, so use both. The semantic cache removes repeat questions entirely, and prompt caching makes the calls that remain cheaper. A request that misses your semantic cache should still hit the provider's prompt cache for its long, unchanging system prompt.

Knowing whether it works

Log four things from day one:

• Hit rate. Hits divided by total requests. If it is a couple of percent, something is off: either your threshold is too strict, or your users genuinely never repeat each other.
• The score of every lookup, with both questions written out. This is the log that tells you where the threshold belongs.
• Money saved. Count the model calls you did not make, times what a call costs you.
• Wrong hits. The one that matters most and the hardest to measure, because nothing throws. Read a sample of your hits by hand, or put a thumbs-down button on answers and watch which side it lands on.

Embedding calls are not free either, so the cache only pays for itself if it hits often enough. Embeddings are far cheaper and faster than generating an answer, so the bar is low. It is not zero.

Growing past the in-memory version

The version above compares against every stored entry, one at a time. That is genuinely fine for a few thousand entries on a single server, and it will hold longer than you expect.

Past that you want a vector database. That is a store built for exactly this job: hand it an embedding, and it hands back the closest matches without checking every entry. It can do that because it indexes the entries by position ahead of time, the same way a normal database index saves you from scanning every row.

You have options at every size. Postgres with the pgvector extension if you already run Postgres, Redis with its vector search if you already run Redis, or a dedicated service. Your code barely changes. findCached and store keep the same shape, and only the search inside them moves.

Start with the array. Move when the search shows up in your latency, not before.

The five things worth remembering

1. An embedding turns text into numbers, and similar meaning gives similar numbers.
2. Cosine similarity compares two embeddings and gives you one score between -1 and 1.
3. Above your threshold is a hit. Start at 0.9 and tune it against your own logged scores.
4. Put everything that changes the answer into the lookup, not just the question. Tenant, user, language, model, prompt version.
5. Never cache personal data, live data, follow-up messages, or creative output.

I write about system design and the senior-to-staff transition every week in Monday BY Gazar on Substack, and I break down architecture and engineering decisions on Gazar Breakpoint on YouTube.

If you are an engineer targeting staff or principal and want this kind of thinking applied to your actual situation, I do 1:1 mentorship.