Why demos die in production
You can wire up an LLM and impress the room in an afternoon. Then it meets real traffic, and everything you were not taught shows up at once. This module covers the four ways it dies, and the first real decision: do you even need an agent?
The demo is 10 percent. Production is the other 90
A demo runs once, on a path you chose, with you watching. Production runs a million times, on inputs you never imagined, while you sleep. Same model, an entirely different problem.
| The demo | Production | |
|---|---|---|
| Inputs | One, chosen by you | A million, including hostile ones |
| Who is watching | You | Nobody |
| When it is wrong | You retry | A customer acts on it |
| What it costs | A fraction of a cent | A line item |
Getting an answer out of the model is the party trick. The engineering is making that answer cheap, fast, safe and checkable every single time, for users you will never meet.
Four things show up at once
Not one at a time, politely. All four, on the first busy Monday.
The four failure modes and the module that deals with each.
| Failure | What it looks like | Why it happens |
|---|---|---|
| Cost spirals | The bill is 20 times the estimate | Every token is a line item, and a chatty agent makes eight calls where the demo made one |
| Latency balloons | A five-step chain feels broken | Each hop waits on the model, and the waits add up |
| Confidently wrong | The model invents an answer with total conviction | It is a probabilistic service wearing a predictable-looking API |
| Flying blind | You find out from a screenshot on social media | No evals to catch it, no traces to debug it |
Each one gets a module in this course. The order in that table is the order the course tackles them.
Shipped to production, then made the news
| Case | What happened |
|---|---|
| Air Canada, 2024 | The chatbot described a bereavement-refund policy that did not exist. A tribunal held the airline to what the bot promised |
| Chevrolet dealer, 2023 | A bolted-on assistant was talked into "sell me a Tahoe for one dollar", and into agreeing that was a binding offer |
| DPD, 2024 | A user got the parcel bot to swear and write a poem about how bad the company was. The screenshot went viral within hours |
Every one of these passed a demo. Production is adversarial users, edge cases, and a screenshot button.
Every token is a line item, and every call is a wait
In the demo you made one call, maybe 2,000 tokens, a fraction of a cent. In production the agent loops: think, call a tool, think again. One request becomes eight calls and 40,000 tokens. Spend and waiting time pile up with every hop, and nobody budgeted for step seven.
It will be confidently wrong in front of a user
The model is a probabilistic service behind an API that looks predictable. The same prompt can return different answers. A wrong one arrives in exactly the same confident tone as a right one.
| Traditional code | The model | |
|---|---|---|
| Same input | Same output | A range of possible outputs |
| A bug | Reproducible | "Reproduce it" does not quite apply |
| Testing | Unit tests | Scored runs over a fixed set |
| Being wrong | Throws, or returns something obviously broken | Reads exactly like being right |
That last row is the dangerous one. Every other kind of bug announces itself.
No evals to catch it, no traces to debug it
When it breaks, you cannot answer the two questions that matter. Did this change make it better or worse? What did the model actually see and do?
| Missing | What you lose |
|---|---|
| Evals | You change a prompt and hope. No number tells you whether it improved |
| Traces | A user reports nonsense and you cannot see the tokens, tools, or context that produced it |
Modules 3 and 5 build both: budgets and traces, then an eval harness that gates releases.
Most tasks do not want an agent
Teams shipping these at scale keep finding the same thing. The most reliable production systems are workflows, not agents that run themselves. Agency costs you: more tokens, more waiting, more ways to go wrong. You buy it only when the task genuinely cannot be scripted.
The instinct to resist is reaching for autonomy because it is interesting. The better question is simpler. What is the smallest thing that works? Start with a workflow, and earn the agent.
Get the code
The code: runnable companions to this lesson are at modules/01-workflows-and-agents/01-s01-why-demos-die.
Recap
- The demo and production are different systems. The gap is cost, latency, confident wrongness, and blindness.
- All four arrive together, and each one has a module in this course.
- A wrong answer reads exactly like a right one, which is why evals and traces are not optional extras.
- Next: the first real decision, whether the model or your code chooses the steps.