Inside DoorDash's AI Assistant Architecture: The LLM Is the Easy Part

When DoorDash swapped the model behind Ask DoorDash from Claude Sonnet 4.6 to Gemini 3.5 Flash, their evaluation harness said the new model was worse. Scores dropped sharply. The obvious read is that Flash wasn't good enough.

That wasn't it. Flash was passing a search query as {"dishes": [...]} where the tool signature declared a plain string, and parts of the system prompt had been tuned around conventions Sonnet happened to infer on its own. The model wasn't weaker. The system had quietly grown a dependency on one vendor's habits, and nobody knew until a different model walked into it. DoorDash's own summary of the diagnosis is the line worth keeping: they were not changing the model, they were correcting the environment around it. After the fixes, Flash hit parity and shipped with a 35% latency reduction.

That detail sits in part three of DoorDash's engineering series on the Assistant, and it's the most useful thing in the whole set. InfoQ covered the series under the headline that the assistant doesn't rely on the LLM alone, which is true but undersells it. Read the primary posts and a stronger claim emerges: the LLM is the least interesting component in the system, and almost all the engineering that made it work is the stuff we already know how to do.

How the Ask DoorDash assistant architecture is layered

Ask DoorDash lets you say "build me a $60 vegetarian list for two people this week" and get back a cart. Underneath, it's four layers, and only one of them is a model. If you've read my piece on supervisor agent architecture, the routing shape will look familiar.

Redrawn from DoorDash's Figure 2 in the engineering overview.

The decision everything else rests on is that business logic lives in the tools, not in the prompts. Cart manipulation, store lookup, deal application: all of it sits behind a Model Context Protocol layer that the same backend services power for the rest of the app. Personalization goes through the identical path, so the agent calls memory_search exactly the way it calls find_nearby_stores. Improvements to search benefit the assistant for free. So do the edge cases, which is the honest half of that trade.

DoorDash names their largest production-failure category outright, and it isn't hallucination in the abstract sense. It's grounding: stores recommended as open when they're closed, prices that don't match the catalog, items the agent claims to have added that aren't in the cart. Their fix every time was to route the agent's claim through a tool call against the system of record. The goal they state is that every consumer-visible claim comes from a tool call on the turn it's made.

That's not an AI technique. That's just refusing to let a cache lie to a customer.

What one grocery turn costs: 8 LLM calls and 30 seconds

Here's the number that should recalibrate anyone sizing an agent: a single grocery turn runs 6 to 8 LLM calls, pulls low hundreds of thousands of input tokens into context once the candidate set lands, and takes 20 to 30 seconds end to end. If you've ever tried to model what an agent actually costs per turn, that context number is where the bill comes from.

Twenty to thirty seconds. In a shopping flow.

Turn two is the part I'd steal. When the consumer taps the widget to swap a pasta brand or drop a yogurt, those edits run straight against the artifact through the Gateway while the model sits idle. Widgets are versioned objects with stable IDs; the agent reads the latest version on the next turn. No round trip, no tokens, no 20-second wait, no chance of the model garbling an edit the user already expressed perfectly by tapping a button.

Most teams reach for the model on every interaction because the model is the exciting part. DoorDash's answer is that if the user's intent is already unambiguous, inference is pure cost. The interesting design question isn't how to make the agent handle the edit. It's noticing that the edit was never an agent problem.

Agent memory is infrastructure, not a feature

I've written before about agent memory and the jump from prompts to persistence, and DoorDash's Intelligence post is the most complete production account of it I've read.

Three layers, each on its own clock.

Redrawn from DoorDash's Figure 2 in the Intelligence post.

The numbers are good: roughly 24% higher relative checkout conversion, 17% bigger baskets, 7% fewer conversational turns for grocery, and 15% higher relative conversion on open-ended restaurant queries. Those are early-rollout, seven-day figures against a no-memory baseline, and DoorDash flags them as a snapshot rather than a trend, which is more caution than most engineering blogs bother with. The metric I find more persuasive is that memory-backed sessions were about 33% less likely to misunderstand user intent.

But the lesson I keep coming back to is the one they put under "knowing what to leave out." A user with 100+ memory blocks and a finite context window needs aggressive pruning, and the reason isn't token cost. Too many competing facts injected at once caused the agent to start dropping instructions. Irrelevant context doesn't just waste budget. It degrades reasoning. Exclusion turns out to be harder than inclusion, which inverts how most people build a retrieval layer.

Two mechanisms deserve stealing outright. The Memory Bank Index is a compact directory of which tokens actually exist in a user's namespaces, injected into the system prompt once per session, with the tool description telling the agent to anchor its searches on it. Rather than asking the model to guess your vocabulary and burn tokens on searches that return nothing, you show it the vocabulary footprint first. And scan before read lets the agent do a cheap parallel check for counts and availability before deciding what to pull at full token cost.

Then there's forgetting, which they're right to call harder than remembering. Tell the agent to forget you like Nova's grilled cheese and a naive similarity threshold happily eats every adjacent memory about "Nova" or "cheese." Their fix is two-stage: semantic search gathers candidates, then a separate LLM call sees the candidates alongside the original request and is told explicitly not to remove facts that are merely related. Saves are fire-and-forget. Deletes are synchronous, because the agent has to tell you what it actually forgot. That asymmetry is a small thing that signals people thought about trust.

The durability rules are where it gets genuinely subtle. In a restaurant context, "I don't want ramen" is transient unless you say "always" or "never," because otherwise a passing mood permanently kills noodles. In grocery, casually saying "I prefer Oatly" is exactly the durable signal you want. Same sentence shape, opposite handling, and the only thing that distinguishes them is domain knowledge someone had to sit down and encode. There's no model release that does this for you.

Evals as a control plane: catching the model migration trap

The thing that makes the rest work is the harness, and it's the piece most teams skip.

Redrawn from Figures 2 to 4 across the evaluation post.

DoorDash went from an average of one employee-submitted feedback per day to 2,000 auto-graded sessions. Comprehensive regression testing dropped from six hours of a developer manually chatting with the agent to about twenty minutes. That bought an 8-point improvement in quality scores before national launch, roughly halving error rates.

The design choices that matter here are unglamorous. Fixtures freeze the upstream world so a score measures the agent rather than a catalog change. The same rubric and the same calibrated judge run offline and online, so an offline result actually carries to production. Each criterion declares what evidence it needs and the transcript builder hands the judge only that slice, because raw traces bury the signal in schema scaffolding. And they admit a real share of flagged failures are faults in the harness itself, usually a judge false positive or a tracing gap.

Which brings it back to the migration. The eval didn't just catch a regression. It was initially wrong about the cause, and the value came from having enough structure to interrogate the verdict rather than accept it. An eval that only produces a number tells you to abandon Flash. An eval wired into traces, failure clusters, and coding agents that can read the failing paths tells you your prompt has an undocumented dependency on Sonnet's parameter formatting conventions.

That's the part I'd push back on slightly, or at least sit with. Building this required an embedded platform team, GEPA prompt optimization, a ClickHouse trace store, and a simulator. It's the right architecture, and it is also a genuinely expensive one that presumes you have the org to staff it. The reusable-platform framing is real, and it's also how every internal platform gets justified before anyone knows whether the second team will adopt it. Ask DoorDash was rolling out to select US areas on iOS when they published. The architecture bet looks correct. It hasn't finished paying out yet.

What I do think transfers, regardless of your size, is the operating discipline underneath. They reversed roughly as many decisions as they kept: static embeddings became dynamic, per-agent prompt optimization became system-level, models moved in and out of the primary path. That's only survivable because every meaningful change ran through the simulator first and a dynamic value let them roll back instantly. Reversibility isn't a nice property here. It's what let them be wrong that often and still ship.

The uncomfortable conclusion in all three posts is that the hard parts of Ask DoorDash aren't AI problems wearing AI clothes. Versioned artifacts, typed tool contracts over a system of record, namespace isolation, TTLs keyed to what a fact is about, a test harness with frozen fixtures, decoupled deploy schedules per domain team. Strip out the model and you're looking at a competent distributed system. The model made the interface conversational. Everything that made it trustworthy is the work we've always known how to do, which is either reassuring or deflating depending on what you were hoping the model would do for you.

I send occasional notes on system design and the senior-to-staff transition, no fixed schedule; you can sign up here. I break down architecture and engineering decisions on Gazar Breakpoint on YouTube.

If you want this thinking applied live, my next free session is System Design for AI Agents: Senior vs Staff on Tuesday, July 21, 2026, 6:30 PM GMT+1: 45 minutes on the five things that break every LLM agent after the demo, and the design decisions that stop each one. My courses and other free lessons are on my Maven profile.