How I Actually Think About System Design

"How I Actually Think About System Design"

Hey, welcome back. I'm Gaz, this is Gazar Breakpoint, and today we're going to talk about something I genuinely love — system design.

But I'm not going to give you the textbook version. I'm not going to talk about CAP theorem for 40 minutes and draw boxes and arrows and call it a day. I want to tell you how I actually think about system design when I'm faced with a real problem. Because I think that mental model is more useful than memorising architectures.

Let me start with something that took me a while to really internalise: system design is not about technology. It's about trade-offs.

When I was early in my career, if someone said "design a URL shortener" or "design a notification system", I would immediately start thinking about databases and frameworks and queues. And those things matter. But I was jumping to solutions before I understood the problem.

The best system designers I've worked with — and I've worked with some really good ones — they start by asking questions. Not to show off. Not to buy time. But because the shape of the system is entirely determined by the constraints and the requirements. And if you get those wrong, the most elegant architecture in the world is useless.

So the first thing I always do when I'm approaching a system design problem is: slow down and clarify.

Here are the questions I always ask before I draw a single box.

First: what does success look like? Not in a vague hand-wavy way. Specifically. What's the read volume, the write volume? What's the acceptable latency? What happens if the system goes down for five minutes — is that a PR disaster or a minor inconvenience? These things change everything.

Second: who are the users? Internal tooling is completely different from a public consumer product. If it's internal and used by 20 engineers, you can cut a lot of corners. If it's serving millions of people, the reliability bar is totally different.

Third: what are the failure modes that actually matter? Every system fails. The question is how it fails and whether you can recover. I'd rather build a system that degrades gracefully than one that looks perfect on a diagram but collapses under any real pressure.

Fourth: what already exists? Because in the real world — and I want to stress this — you almost never design from a blank slate. There's always existing infrastructure, existing contracts, existing teams who own adjacent systems. You have to design within that reality, not pretend it doesn't exist.

And fifth: what's the timeline and what's the team? A brilliant architecture that takes 18 months to build when you need something in 6 months is a bad architecture. The constraints of who's building it and when it needs to be done are part of the design.

Okay, so you've done that upfront thinking. Now what?

This is where I like to think in layers. And I use a mental model I call breadth first, then depth.

What I mean is: I sketch the whole system at a very high level first, before I go deep on anything. Just the major moving parts. What are the big components, how do they talk to each other, what's the rough flow of data through the system. I do this deliberately because it forces me to think about the whole thing before I get lost in one part.

Because here's what happens if you don't do this. You spend an hour designing the perfect database schema, and then you realise you never thought about how you're going to handle authentication or how the data gets from the user to the database in the first place. You've optimised one part of a system that doesn't fully exist yet.

So: breadth first. Get the whole picture. Then pick the two or three areas where the real complexity lives and go deep on those.

Now, about trade-offs. This is the core skill. And I want to give you a framework for thinking about them.

Every significant design decision in a system has a cost. Usually you're trading one of these things: consistency vs availability, latency vs throughput, complexity vs simplicity, flexibility vs performance.

And the job is not to avoid trade-offs. The job is to make them consciously and to make sure the trade-off you're making is the right one given your constraints.

Let me give you a real example. I once worked on a system where we needed to show users a count — like "you have 47 notifications". And there was a real debate about how to compute that. Do we compute it on every read? Do we maintain a counter and update it on every write? Do we cache it?

Each approach has different trade-offs. Computing on read is slow but always accurate. Maintaining a counter is fast but you have to handle the case where the counter and the actual data get out of sync, which they will. Caching is fast but stale.

And the decision came down to: how important is it that the number is exactly right? If the user sees 47 instead of 48 for a few seconds, is that a problem? In this case, no. So we cached it and moved on. But in a financial system where the number represents money, that trade-off is completely different.

That's the thinking I want you to develop. Not "what's the right answer" but "what are the trade-offs and which ones can I live with given what I know about this system and its users?"

The other thing I want to talk about is the relationship between system design and code.

I've seen a lot of developers who are great at system design diagrams but struggle when it comes to actually translating that into real code. And I've seen the opposite — great coders who can't think at a system level at all.

The best engineers bridge both. And the way I think about it is: the system design tells you what to build, the code tells you how. They need to be consistent with each other.

One of the things I always push for is that the architecture should be visible in the code structure. If you've designed a layered architecture, the layers should be obvious when someone opens the repository. If you've designed a clear boundary between services, that boundary should be enforced in the code, not just in a diagram on a wiki that nobody updates.

Because diagrams lie. Code doesn't. The real architecture is what's actually in production, not what's in the slide deck.

Let me also talk about something I call architecture regret. Because I've had it many times and I want to save you some pain.

Architecture regret is when you look back at a decision you made 12 or 18 months ago and you think — why did we do it that way? And usually it's one of two things.

Either you over-engineered it. You built a sophisticated distributed system for a problem that could have been solved with a simple database and a few API endpoints. I've done this. It's embarrassing. The system was hard to operate, hard to debug, and the team hated it. And we'd built it to handle scale we never actually reached.

Or you under-engineered it. You built something quick and pragmatic, which was the right call at the time, but you didn't design any obvious extension points. So now every time you need to add something, you're working against the existing structure rather than with it.

The sweet spot — and this is hard — is building something that's simple enough to ship and operate, but with a clear enough structure that it's not painful to grow. And honestly, the way I try to avoid architecture regret now is by being explicit about assumptions. Writing down: "we're designing this for X scale, with Y team, over Z timeframe. If those things change, we should revisit this." That way, if things do change, you know why you made the decisions you made.

One last thing. System design is a skill you build by doing it and by getting feedback. Not by watching videos and doing mock interviews.

So my challenge to you — pick a real system you've built or worked on. Go back and think about the trade-offs that were made. Were they the right ones? What would you change? What constraints were you working with that you'd forgotten about?

That reflection process is how you get better at this. Because the patterns start to emerge when you've looked at enough systems and asked enough questions about them.

Alright, that's episode 3. I could honestly talk about system design for hours — it's something I genuinely love. And if you want to go deeper, I have a full System Design Mastery course on my website at gazar.dev. It covers all of this in much more detail.

See you in the next one.