Token Economics: How to Cut LLM Cost Without Making Your Product Worse

Words you need

• Token -- A small piece of text the AI reads or writes (roughly a word or part of a word). You pay per token. Long answers and long context cost more.
• LLM -- Large Language Model. The AI that generates text. When we say "the model", we mean this.
• RAG -- Retrieval-Augmented Generation. The system fetches the right pieces of content first, then the AI answers using those pieces. More pieces in the prompt means more cost.
• Chunk -- A piece of a document you feed into the AI as context. "Retrieve 12 chunks" means you're putting 12 pieces in the prompt; each one costs tokens.
• Embedding -- A list of numbers that represents the meaning of a piece of text. Used to search for similar content. Computing and storing embeddings costs money; caching them saves.
• Cache -- A stored result you reuse instead of computing again. Cache key = the exact conditions (e.g. which prompt version, which model) that must match for the stored result to be valid.
• Model routing -- Sending an easy request to a smaller, cheaper model and only using the big, expensive model when needed.
• Streaming -- Sending the answer to the user piece by piece as it's generated, instead of waiting for the full answer. Improves perceived speed; can reduce retries and thus cost.

I've watched teams ship an AI feature, celebrate adoption, then panic when the GPU bill lands.

The pattern is always the same: nobody measured unit cost per request. Nobody set budgets. The feature worked, usage grew, and so did the invoice.

Token economics isn't a finance problem. It's an engineering problem. If you can't control token usage, you can't scale.

For the full map from concept to production, see the LLM Handbook series map. For how retrieval and chunk count affect cost, chunking and RAG at inference are the right levers.

The fastest win: cap output length

Long answers cost more. They're also often unnecessary.

Start here:

• Default to short, direct answers
• Add an "expand" button for users who want more detail
• Set strict max_tokens per mode

You'll be surprised how often users prefer shorter, clearer responses. The verbose answer isn't just expensive -- it's often worse UX.

Retrieval is a cost lever

RAG increases prompt length. Every chunk you retrieve adds tokens to the bill.

If you retrieve 12 chunks every time, you're paying for 12 chunks every time. Most of them probably aren't helping.

Make retrieval smarter:

• Filter by metadata before vector search
• Retrieve fewer candidates
• Rerank and keep only the best matches
• Deduplicate near-identical chunks

Sloppy retrieval is expensive and produces worse answers. Tightening it saves money and improves quality. That's rare -- most optimizations involve trade-offs.

Cache the right things

You can cache at three levels:

1. Embeddings for documents that don't change often
2. Retrieval results for repeated or similar queries
3. Final answers for identical questions

But caching without versioning is dangerous. If you change the prompt template or retrieval logic and don't update the cache key, users get stale answers from the old system. Confidently wrong.

Cache keys must include:

• Prompt version
• Retrieval policy version
• Model ID

If you can't version your cache properly, don't cache. "Ghost bugs" -- bugs that only appear when the cache serves an answer that doesn't match the current system -- are brutal to debug.

Model routing: cheap first, expensive when needed

This is a production pattern that pays for itself quickly.

Route easy requests to a smaller, cheaper model. Escalate to the expensive model only when confidence is low.

Confidence signals:

• Retrieval score below threshold
• Output validation failures
• Self-check prompts (careful -- these add tokens too)

It's not magic. It's tiered services. The same pattern we've used in backend engineering for years: try the fast path first, fall back to the expensive path when necessary.

Streaming reduces cost indirectly

Streaming doesn't change token pricing. But it reduces perceived latency. Users who see text appearing immediately are less likely to:

• Close the tab and retry
• Spam the submit button
• Assume the system is broken

Retries are a direct cost multiplier. Better UX means fewer retries means lower spend.

Log token usage as a first-class metric

Every response should log:

• Prompt tokens
• Output tokens
• Retrieval chunk count
• Latency
• Model ID
• Feature/mode

Then you can answer real questions: Which feature is burning cost? Which user cohort is expensive? Which prompt version increased token usage?

This is how you move from "it feels expensive" to "this endpoint is 62% of spend."

Token budgets per mode

I like budgets that are explicit and enforced:

export interface ModeBudget {
  mode: "chat" | "grounded_answer" | "rewrite";
  maxPromptTokens: number;
  maxOutputTokens: number;
  maxChunks: number;
}

Each mode gets hard limits. Your system enforces them before the request hits the model. Budgets aren't constraints on creativity. They're guardrails that prevent runaway costs.

Where to start

Pick one lever. Measure your current baseline. Apply the change. Measure again.

The three biggest levers, in order:

1. Cap output tokens
2. Reduce retrieved chunks
3. Add model routing

If you had to cut your LLM bill by 40% this month, which one would you reach for first?