Semantic Caching for AI Requests, Explained Simply

Ten people ask your AI assistant the same thing:

"How do I reset my password?"
"how can i change my password"
"I forgot my password, what now?"
"password reset help please"

Those are the same question. To your code they are four different strings, so you call the model four times, wait four times, and pay four times.

A semantic cache fixes this. It stores answers by meaning rather than by exact text, so the last three requests get the first one's answer, instantly and for free.

This article assumes you have called an AI model from code before. Nothing else.

First try: the normal cache

The obvious idea is a normal cache. Use the question as the key:

const cache = new Map<string, string>();

async function ask(question: string) {
  const hit = cache.get(question);
  if (hit) return hit;

  const answer = await callModel(question);
  cache.set(question, answer);
  return answer;
}

This works, and it almost never helps. The key has to match letter for letter. A capital letter, an extra space, a question mark, and you miss.

cache.get("How do I reset my password?"); // hit
cache.get("how do I reset my password");  // miss, one character different

You can clean the text up a bit (lowercase it, strip punctuation) and you will catch a few more. You will still miss "I forgot my password", because those are completely different words for the same thing.

Text matching cannot see meaning. That is the whole problem.

The idea: turn meaning into numbers

An embedding is a list of numbers that represents the meaning of a piece of text. You send text to an embedding model, and it gives you back an array of a few hundred to a few thousand numbers.

await embed("How do I reset my password?");
// [0.021, -0.114, 0.088, ... ]  a long list of numbers

await embed("I forgot my password");
// [0.019, -0.109, 0.091, ... ]  a very similar list

You do not need to understand what any single number means. Only one thing matters:

Text with similar meaning produces similar numbers. Text with different meaning produces different numbers.

Think of it like a map. Every sentence gets a pin. Sentences about resetting passwords all land in the same neighbourhood. Sentences about refunds land somewhere else entirely.

So a semantic cache asks a different question. Not "have I seen this exact text before" but "have I seen anything that means this before".

Measuring how close two meanings are

To compare two embeddings you use cosine similarity. It gives you one number:

• 1.0 means the same direction, so the same meaning
• 0.0 means unrelated
• below 0 means opposite

The maths is short enough to write out:

function cosineSimilarity(a: number[], b: number[]): number {
  let dot = 0;
  let lenA = 0;
  let lenB = 0;

  for (let i = 0; i < a.length; i++) {
    dot += a[i] * b[i];
    lenA += a[i] * a[i];
    lenB += b[i] * b[i];
  }

  return dot / (Math.sqrt(lenA) * Math.sqrt(lenB));
}

That is the entire comparison. Multiply the pairs, add them up, divide by the lengths. No library needed.

Then you pick a threshold. If the similarity is above it, you call it a match and return the cached answer. If not, you call the model.

The full flow

Five steps. Embed, search, compare, answer or call, store.

Building one

Here is a working cache in about forty lines. It keeps everything in memory, which is fine for learning and for a single server.

type Entry = {
  question: string;
  embedding: number[];
  answer: string;
  createdAt: number;
};

class SemanticCache {
  private entries: Entry[] = [];

  constructor(
    private threshold = 0.9,
    private ttlMs = 60 * 60 * 1000, // one hour
  ) {}

  async get(question: string): Promise<string | null> {
    const embedding = await embed(question);
    const now = Date.now();

    let best: Entry | null = null;
    let bestScore = 0;

    for (const entry of this.entries) {
      if (now - entry.createdAt > this.ttlMs) continue; // too old
      const score = cosineSimilarity(embedding, entry.embedding);
      if (score > bestScore) {
        bestScore = score;
        best = entry;
      }
    }

    if (best && bestScore >= this.threshold) {
      console.log(`cache hit (${bestScore.toFixed(3)}): "${best.question}"`);
      return best.answer;
    }

    console.log(`cache miss (best was ${bestScore.toFixed(3)})`);
    return null;
  }

  async set(question: string, answer: string): Promise<void> {
    this.entries.push({
      question,
      embedding: await embed(question),
      answer,
      createdAt: Date.now(),
    });
  }
}

And using it:

const cache = new SemanticCache();

async function ask(question: string): Promise<string> {
  const cached = await cache.get(question);
  if (cached) return cached;

  const answer = await callModel(question);
  await cache.set(question, answer);
  return answer;
}

That is a real semantic cache. Everything after this point is about not getting hurt by it.

Picking the threshold

The threshold is the one number you will spend real time on. It is a trade.

Set it too low and unrelated questions match. Someone asks about refunds and gets an answer about passwords. This is the bad failure, because the user cannot tell it went wrong. They just get a confidently wrong answer.

Set it too high and almost nothing matches. The cache costs you an extra embedding call per request and returns almost nothing.

Do not guess this from a blog post, including this one. Start around 0.9, then log the similarity score of every request for a week, hit or miss, next to the two questions being compared. Read the log. You will see very quickly where your own data separates the real matches from the accidents, because your users ask questions in a way that is specific to your product.

Two rules that save pain:

• The cost of a wrong hit is much higher than the cost of a miss. When you are unsure, go higher.
• Different question types want different thresholds. "What is your refund policy" is safe to match loosely. "What is the balance on invoice 4471" should never match anything.

The dangerous part: what goes in the key

This is where people get hurt, so read this section twice.

Two questions can be worded identically and still need different answers, because something outside the question changes the answer. If you only match on the question text, you will serve one user's answer to another user.

// Don't
const cached = await cache.get(question);

// Do
const cached = await cache.get(question, {
  userId: user.id,
  tenantId: user.tenantId,
  locale: user.locale,
  model: "claude-opus-5",
  promptVersion: 7,
});

Everything that changes the answer has to be part of the lookup, not just the meaning of the words.

The simple way to do this is to keep a separate bucket per exact-match part, and only compare meaning inside a bucket:

private buckets = new Map<string, Entry[]>();

private bucketKey(ctx: Context): string {
  return [ctx.tenantId, ctx.locale, ctx.model, ctx.promptVersion].join("|");
}

Now a question from tenant A can never match an answer given to tenant B, whatever the wording. Meaning is only compared where it is safe to compare it.

When not to cache at all

Some answers must never come from a cache, no matter how similar the question:

• Anything personal. "How much do I owe?" is the same sentence for every customer and a different answer for each one.
• Anything about right now. Stock levels, prices, order status, live data. A right answer from ten minutes ago is a wrong answer today.
• Anything the user is meant to steer. "Make it shorter", "try again", "now in French". These depend on the conversation, not on the sentence.
• Anything creative. If someone asks for a poem, giving them a poem you already gave someone else defeats the point.

The clean way to handle this is a skip list checked before the cache:

function isCacheable(question: string, ctx: Context): boolean {
  if (ctx.hasConversationHistory) return false;
  if (/\b(my|mine|our)\b/i.test(question)) return false; // likely personal
  return true;
}

Crude, and much better than nothing. Start strict and open it up as you learn.

Keeping answers fresh

A cached answer is a promise that nothing has changed. Three ways to keep that promise:

Time. Give every entry a TTL, as in the code above. An hour is a reasonable starting point for support-style answers.

Versions. When you change your system prompt, change your model, or update the documents behind your answers, bump a version number that is part of the bucket key. Old entries stop being found and age out on their own. This is much safer than trying to delete the right entries.

Events. When the underlying source changes, drop the entries built from it. This needs you to store which documents an answer came from, so it is the most work and the most precise.

This is not the same as prompt caching

Two different things share the word "cache", and mixing them up is common.

Prompt caching is a feature of the model provider. You mark part of your prompt, and the provider keeps its processed form so repeat requests skip re-processing it. On the Claude API this is cache_control, cached reads cost roughly a tenth of normal input, and the entry lives for five minutes by default (one hour is available). It is an exact prefix match: the cached part must be byte for byte identical, and it must come at the front of the prompt, so a timestamp near the top of your system prompt quietly disables everything after it.

Semantic caching is something you build. It compares meaning, it lives in your code, and on a hit it makes no model call at all.

They are not alternatives. Use both: the semantic cache removes repeat questions, and prompt caching makes the calls that remain cheaper. A request that misses your semantic cache should still hit the provider's prompt cache for its long, unchanging system prompt.

Knowing whether it works

Log four things from day one:

• Hit rate. Hits divided by total requests. Below a few percent, something is wrong: your threshold is too strict, or your users genuinely never repeat themselves.
• The score of every lookup, with both questions. This is the log that tells you where the threshold belongs.
• Money saved. Count the model calls you did not make, times what a call costs you.
• Wrong hits. The one that matters most and the hardest to measure. Sample your hits and read them by hand, or offer a thumbs-down and watch which side it lands on.

An embedding call is not free either, so the cache only pays for itself if it hits often enough. Embeddings are far cheaper and faster than a full generation, so the bar is low, but it is not zero.

Growing past the in-memory version

The version above compares against every entry, one at a time. That is fine for a few thousand entries and one server. Past that you want a vector database, which is a store built for exactly this: give it an embedding, and it gives you the closest matches without checking all of them.

You have options at every size. Postgres with the pgvector extension if you already run Postgres, Redis with its vector search if you already run Redis, or a dedicated service. The logic in your code does not change at all. get and set keep the same shape, and only the search inside them moves.

Start with the array. Move when the search shows up in your latency, not before.

I write about system design and the senior-to-staff transition every week in Monday BY Gazar on Substack, and I break down architecture and engineering decisions on Gazar Breakpoint on YouTube.

If you are an engineer targeting staff or principal and want this kind of thinking applied to your actual situation, I do 1:1 mentorship.