Why demos die in production
You can wire up an LLM and impress the room in an afternoon. Then it meets real traffic, and everything you were not taught shows up at once. This module covers the four ways it dies, and the first real decision: do you even need an agent?
The demo is 10 percent. Production is the other 90
A demo runs once, on a happy path, with you watching. Production runs a million times, on inputs you never imagined, while you sleep. Same model, an entirely different problem.
| The demo | Production | |
|---|---|---|
| Inputs | One, chosen by you | A million, including hostile ones |
| Who is watching | You | Nobody |
| When it is wrong | You retry | A customer acts on it |
| What it costs | A fraction of a cent | A line item |
Getting an answer out of the model is the party trick. Making that answer cheap, fast, safe and checkable every single time, for users you will never meet, is the engineering.
Four things show up at once
Not one at a time, politely. All four, the first busy Monday.
| Failure | What it looks like | Why it happens |
|---|---|---|
| Cost spirals | The bill is 20 times the estimate | Every token is a line item, and a chatty agent makes eight calls where the demo made one |
| Latency balloons | A five-step chain feels broken | Each hop waits on the model, and the waits add up |
| Confidently wrong | The model invents an answer with total conviction | It is a probabilistic service wearing a deterministic API |
| Flying blind | You find out from a screenshot on social media | No evals to catch it, no traces to debug it |
Each one gets a module in this course, and the order they appear in that table is the order the course tackles them.
Shipped to production, then made the news
| Case | What happened |
|---|---|
| Air Canada, 2024 | The chatbot described a bereavement-refund policy that did not exist. A tribunal held the airline to what the bot promised |
| Chevrolet dealer, 2023 | A bolted-on assistant was talked into "sell me a Tahoe for one dollar", and into agreeing that was a binding offer |
| DPD, 2024 | A user got the parcel bot to swear and write a poem about how bad the company was. The screenshot went viral within hours |
Every one of these passed a demo. Production is adversarial users, edge cases, and a screenshot button.
Every token is a line item, and every call is a wait
In the demo you made one call, maybe 2,000 tokens, a fraction of a cent. In production the agent loops: think, call a tool, think again. One request becomes eight calls and 40,000 tokens. Spend and latency compound with every hop, and nobody budgeted for step seven.
It will be confidently wrong in front of a user
The model is a probabilistic service behind a deterministic-looking API. The same prompt can return different answers, and a wrong one arrives in exactly the same confident tone as a right one.
| Traditional code | The model | |
|---|---|---|
| Same input | Same output | A distribution of outputs |
| A bug | Reproducible | "Reproduce it" does not quite apply |
| Testing | Unit tests | Scored runs over a fixed set |
| Being wrong | Throws, or returns something obviously broken | Reads exactly like being right |
That last row is the dangerous one. Every other kind of bug announces itself.
No evals to catch it, no traces to debug it
When it breaks, you cannot answer the two questions that matter: did this change make it better or worse, and what did the model actually see and do?
| Missing | What you lose |
|---|---|
| Evals | You change a prompt and hope. No number tells you whether it improved |
| Traces | A user reports nonsense and you cannot see the tokens, tools, or context that produced it |
Modules 3 and 5 build both: budgets and traces, then an eval harness that gates releases.
Most tasks do not want an agent
The consistent finding from teams shipping these at scale is that the most reliable production systems are workflows, not autonomous agents. Agency is a cost: more tokens, more latency, more ways to go wrong. You buy it only when the task genuinely cannot be scripted.
The instinct to resist is reaching for autonomy because it is interesting. The better question is: what is the simplest thing that works? Start with a workflow, and earn the agent.
Get the code
The code: runnable companions to this lesson are at modules/01-workflows-and-agents/s01-why-demos-die.
Recap
- The demo and production are different systems. The gap is cost, latency, confident wrongness, and blindness.
- All four arrive together, and each one has a module in this course.
- A wrong answer reads exactly like a right one, which is why evals and traces are not optional extras.
- Next: the first real decision, whether the model or your code chooses the steps.