Gazar BreakpointEpisode 8June 17, 202636 min
System Design for AI Agents: What Engineers Actually Need to Know
Everyone is wiring LLMs into production, and most of the guidance out there is written for product managers, not engineers. This session is the engineer's mental model: what is ordinary distributed systems work, what genuinely breaks when an LLM sits in the path, and the few decisions that actually decide whether your agent survives real traffic, cost, and failure. We work through the patterns from first principles: RAG, tool-calling agents, routing, and orchestrator and workers. Then we get to the part that matters: an agent is just a service that calls a slow, expensive, flaky dependency, so we treat it like one.
In this episode
- When to reach for RAG, routing, or an orchestrator, and when not to
- Why an LLM in the request path is a normal service with abnormal failure modes
- State as the context window, and how to reason about it
- The four calls that decide if it works in production: boundaries, state, failure modes, evaluation
- Why reliability and cost are design inputs, not afterthoughts
Chapters
Tap a chapter to jump the video to that moment.