• Share this post & I'll send you some rewards for the referrals.

A single agent1 has one context window2, one set of tools3, and one running loop4.

When a task outgrows any of those three, you need more than one agent. That’s what a multi-agent system is. Instead of one agent doing everything, you split the work across many agents. Each one has its own role, tools, and context.

Most multi-agent systems fail not because the model is weak but because you chose many agents before you actually need them, or chose the wrong architecture once you have many agents.

So you shouldn’t split anything yet.

Onward.

[Webinar] Stop babysitting your coding agents (Partner)

Agents can generate code. Getting it right for your system, team conventions, and past decisions is the hard part – you end up wasting time and tokens in correction loops.

More MCPs give agents access to information, but not understanding. The teams pulling ahead use a context engine to give agents exactly what they need.

Join Unblocked live on May 6 (FREE) to see:

• Where teams get stuck on the AI maturity curve

• How a context engine solves for quality, efficiency, and cost

• Live demo: the same coding task with and without a context engine

Register Now

(Thanks to Unblocked for partnering on this post.)

Here’s what’s inside:

• Why single agents break. Context overflow, slow serial work, and why one agent can’t always hold every tool, model, or permission.

• The six architectures. Orchestrator-worker, pipeline, hierarchical, swarm, mesh, and handoffs. Plus, where each one works and breaks.

• How agents coordinate. Run loops, MCP vs A2A, shared state, isolated state, memory, and stopping conditions.

• The real cost. Why more agents mean more tokens, more latency, more coordination overhead, and more ways to fail.

• Failure and security risks. Bad instructions, misalignment, weak verification, prompt injection, context contamination, and privilege creep.

• Case study. How Spotify used an orchestrator-worker system to turn ad planning from a 15-minute workflow into 5 seconds.

Limits of single-agent systems

One agent with good prompts and the right tools handles more than most people expect.

For example, Cognition’s Devin processed 5 million lines of COBOL (Common Business-Oriented Language) across 500GB of repositories with a single agent, raising its pull request merge rate from 34% to 67%.

But a single agent has three HARD limits. When your task runs into any of them, better prompts won’t help:

1. Context overflow

A context window can only hold so much.

Past that limit, the earliest information drops out, and the agent starts losing track of its own plan. When compression5 alone can’t fix the overflow, a second agent with its own context can.

2. Parallelism

Independent tasks shouldn’t wait in line.

If you have four research queries that don’t depend on each other, running them one at a time wastes time. Running them across four separate agents takes roughly as long as the slowest one.

Anthropic’s research system uses this exact pattern and reduced total query time by up to 90%.

3. Specialization

Different parts of a task often need different models6, tools, or access levels:

• A code-writing agent needs a sandbox

• A research agent needs web search

• A customer-facing agent needs user data, but shouldn’t have access to production databases

When one agent can’t hold all the tools and permissions the task needs, you give each role its own agent.

If none of those three conditions apply, stay with one agent.

Better prompts and better tools solve most problems without adding the extra work of coordinating agents. But once you know you need many agents, the next question is what shape the system takes...

Multi-agent architectures

Every multi-agent system makes a different choice about who coordinates work.

Here are 6 architectures that range from tight central control to NO coordinator at all:

1. Orchestrator-worker

One central agent breaks a task into pieces, assigns each piece to a worker agent, and then puts the results together:

• Workers don’t talk to each other; all communication goes through a central agent

• Orchestrator calls each worker as a tool call7, waits for a result, and decides what to do next

This is like an air traffic control tower that talks to every plane, while no plane talks directly to another.

Anthropic’s Claude Research system works this way:

A central agent running Opus 4 breaks a research query into parts and creates 2 to 10 worker agents on Sonnet 4 (sometimes more), all at the same time. The workers search the web, read documents, and gather evidence in parallel. When they finish, the central agent reads their results and writes a single research report.

This setup beat single-agent Opus 4 by 90.2% on Anthropic’s internal research eval8.

Tradeoffs

Central agent is both coordinator and bottleneck…

It talks to workers one at a time. If each call takes 3 seconds and 20 workers are waiting, the ceiling is about 7 tasks per second. So the central agent becomes the slowest part of the system.

Anthropic’s Claude Research system had this problem as well: workers duplicated each other’s work.

Without specific instructions, many workers run overlapping searches on the same topic, wasting both tokens9 and time. The takeaway is orchestrator-worker depends entirely on the quality of the lead agent’s instructions.

Vague task splitting turns parallelism into duplicated work.

2. Pipeline

Agents run in a fixed order, one after another.

Each agent’s output becomes the next agent’s input, and entire sequence is set in advance. While orchestrator-worker lets the central agent decide what to do as it goes, pipeline removes that choice.

An assembly line works the same way: each station does one job, passes the result forward, and never sees the finished product.

Stripe uses this pattern to check whether new businesses on its platform are legitimate.

Before agents, a human reviewer had to jump between customer databases, legal sources, and support tickets to decide whether a business was safe to approve. Now their engineering team broke that work into a fixed flow of agent stages using a directed acyclic graph (DAG). So work moves forward through stages without looping back.

Order is set at design time, and each stage has a contract: defined output format next stage expects.

Stripe calls these contracts “rails” because they keep any single agent from spending too much time on irrelevant data. This setup cut their average handling time by 26%, and reviewers rated the agent outputs 96% helpful, with a full record of every decision at every step.

Tradeoffs

Latency adds up.

A 5-stage pipeline where each stage takes 2 seconds means a 10-second wait before any output, and adding a stage to improve quality increases the response time.

Yet the upside is predictability.

When every stage has a narrow contract, you can trace any failure back to exactly one step. That’s why regulated workflows like Stripe’s continue to use pipelines.

When the cost of being wrong is a regulator flagging your process, the extra seconds are a small price to pay.

3. Hierarchical

A top-level manager agent gives work to one or more layers of manager agents below it, which then give work to individual workers.

Two levels are the minimum; big systems stack more. The result is a tree…

No single agent needs the full context.

The top-level agent holds the high-level goal and a summary from each branch, while each lower level sees only what its narrower role needs. Picture a military chain of command where orders flow down, reports flow up, and no one skips a level.

IBM watsonx Orchestrate runs on this pattern:

A top-level supervisor agent acts as a router and planner across 80+ pre-built domain agents for HR, sales, and procurement. Let’s say a user tries to “order new laptops for the design team”. The request reaches a Procure Equipment supervisor, who then hands the work to three specialized child agents:

• One requests quotes from approved vendors

• Another checks responses

• A third submits a purchase request

The supervisor decides only who gets called and in which order.

Tradeoffs

Details might get lost at each level.

A worker produces a detailed result. The mid-level manager shortens it to one sentence. By the time it reaches the top, the detail that matters might be gone. Hierarchical structures trade detail for coverage: the higher you go, the wider the scope of each agent and the less it knows about any specific piece.

These three architectures share one thing: a clear chain of command tells you exactly who’s in charge…

The next three architectures drop the chain entirely. So they're harder to debug, but they survive partial failures better…

Reminder: this is a teaser of the subscriber-only newsletter, exclusive to my golden members.

When you upgrade, you’ll get:

• High-level architecture of real-world systems.

• Deep dive into how popular real-world systems work.

• How real-world systems handle scale, reliability, and performance.

Unlock Full Access

4. Swarm

In a swarm, many agents work as equals.

They coordinate through a shared blackboard: a data store (typically Redis cache, database table, or vector store) that every agent can read from and write to. Yet there are NO direct messages between agents…

Read more

• Share this post & I'll send you some rewards for the referrals.

You ship an LLM feature.

It passes your manual tests. A day later, a user posts a screenshot of it hallucinating wildly! You tweak the prompt, run it again, and it works fine. Did you fix it, or did you just get lucky?

Welcome to the central frustration of LLM engineering: you can NO longer just run a test and call it done.

This isn’t a debugging problem…

It’s a measurement problem. And measurement has a name: evaluation.

Most articles on LLM evaluation are written for ML researchers. This one is for engineers building real applications. You know how to ship software. You’re just new to the ways LLMs fail.

We’ll cover the vocabulary, methods, and mental models. By the end, you’ll have a framework for building an eval system from scratch. Not just an understanding of why it matters.

Let’s start with why your existing testing instincts don’t work here…

Your team’s second brain. Now in Slack. (Partner)

Your engineers talk on Slack. They code in the terminal. Somewhere between those two things, context goes to die.

• A bug was debated in #incidents at 2 AM.

• An architectural call was made in a DM.

Every handoff leaks context, and every leak costs you. That’s the context tax - and your team pays it every day.

CodeRabbit Agent for Slack is built for agentic SDLC workflows. One agent for your entire Software Development Lifecycle, living in the channel where the work already happens. It’s built on four things:

• Context - your org’s operating picture, pulled from across code, tickets, docs, monitoring and cloud.

• Knowledge Base - a living memory of your team. Every run leaves a trace, so yesterday’s decisions don’t become tomorrow’s debates.

• Multi-Player - works in shared threads alongside your team. Steerable, resumable and aligned as work evolves.

• Governance - scoped access, cost attribution. Every run explainable and attributed.

Your team keeps shipping. Agent keeps the context.

From the team that pioneered AI code reviews. 2M code reviews every week. 6M repos. 15K customers. And now, one agent for your entire SDLC, right in Slack.

Try CodeRabbit's Agent Today

(Thanks to CodeRabbit for partnering on this post.)

I want to introduce Anshuman as a guest author.

He leads evals efforts at Zomato, where he built Gavel -- an internal LLM eval platform that started as a handful of scripts, got pitched to a VP, and now serves AI and ops teams across the company. Now he’s on the ground floor of making a large organization AI-native, one eval system at a time.

I highly recommend you checkout his newsletter, AI Proof -- it’ll help you stay relevant in the AI era.

Before we get into solutions, you need to understand something…

LLM evaluation feels nothing like regular software testing. It’s not that it’s harder. It’s that the rules changed.

Let’s dive in!

1. Non-determinism Problem

Write a function.

Call it with the same input twice, and you’ll get the same output twice.

But LLMs don’t work that way.

The same prompt can produce a different response on every run.

Sometimes slightly different. Sometimes completely different. This isn’t a bug. It’s called temperature.

It controls how random the outputs are. (More on this later.)

This breaks something deeper. Your entire mental model for testing assumes determinism. You run a test. It passes or fails. You know what you know.

With LLMs, a passing test is a data point, not a verdict. You’re not testing a function. Instead, you’re sampling from a probability distribution. One sample tells you almost nothing.

This is the first reason your existing instincts mislead you.

2. Fuzzy Correctness Problem

When a regex matches, it matches. Binary. Objective. Done.

But what’s the correct response to “summarize this support ticket empathetically?”

• Is the correct answer the one that’s most concise?

• The one that uses the warmest language?

• The one that captures all the key facts?

• The one a human reviewer would score highest?

LLM quality is multi-dimensional and subjective.

There usually isn’t one correct answer. There’s a range of acceptable answers and a range of bad ones. The line between them is a judgment call.

You can’t measure what you haven’t defined.

What does ‘good’ mean for your use case? You need a clear answer to that before you can evaluate anything. Most people skip this step. It comes back to haunt them.

3. Silent Regression Problem

You updated your prompt.

Ran it a few times. The outputs seemed better. You shipped it.

But did quality actually improve, or did you get lucky with the samples you checked?

This is the silent regression problem. Without a systematic evaluation process, every prompt change is a blind bet. You might be making things better. You might be fixing one failure mode while introducing another. And you have no way of knowing.

In traditional software, CI (Continuous Integration) catches regressions before they reach users. In LLM engineering, there’s NO such equivalent. So people rely on their gut feelings and manual spot checks. You find out about regressions when a user complains!

These three problems make LLM evaluation its own discipline…

Now let’s build the vocabulary to actually talk about it…

Primitives of Eval

These are six terms you’ll hear constantly.

Learn these, and the rest falls into place:

4. Criteria

Criteria are the dimensions of quality that matter for your specific use case.

A customer support bot might care about:

• Does it address the user’s actual issue?

• Is the tone empathetic?

• Does it avoid triggering escalation unnecessarily?

A code generation tool cares about something completely different:

• Is the output syntactically valid?

• Does it follow the project’s conventions?

• Does it actually solve the stated problem?

Same technology, but entirely different criteria.

This is a product decision, NOT a technical one. You can’t outsource it to your model or your eval framework. Someone on your team needs to sit down and answer: What does a good output actually look like here?

Get this wrong, and everything downstream is measuring the wrong thing…

5. Quality Dimensions

Quality dimensions are the standard industry vocabulary for LLM output quality.

Here are five of them that come up constantly:

• Relevance: Did the output address what was actually asked? A response can be accurate, well-written, and completely beside the point.

• Coherence: Does it hold together logically? No contradictions. No mid-sentence topic shifts. Flows like something a thoughtful person wrote.

• Factual Accuracy: Is what it says actually true? Distinct from relevance. A response can be relevant and wrong.

• Helpfulness: Does it give the user what they need to move forward? The difference between a technically correct answer and a useful one.

• Safety: Does it avoid harmful, biased, or inappropriate content? This dimension matters more in some domains than others -- but it always matters.

Knowing these helps you write rubrics that don’t miss important failure modes.

6. Rubric

Once you have the criteria, you need to operationalize them.

That’s what a rubric does.

Take a vague criterion like “helpfulness.” A rubric breaks it into specific, scorable questions. Does it directly answer the question? Does it avoid unnecessary hedging? Is it under 200 words? Can a non-technical user understand it?

Think of it like a code review checklist.

Instead of “Is this good code?”, a checklist asks: Are there tests? Is the function under 30 lines? Are variable names descriptive? Rubrics do the same thing for LLM outputs.

The rubric is what makes evaluation reproducible.

Two different reviewers, human or AI, should arrive at similar scores. Same output, same rubric, same conclusion.

Without a rubric, every evaluation is just someone’s opinion.

7. Test Cases

A test case is an input/output pair that forms one unit of your evaluation.

Input is a prompt: ideally, one representative of real user traffic.

Output can be one of two things. A reference answer showing what good output looks like. Or the live model output you’re about to score.

Think of test cases like unit tests…

Except that a failing test case doesn’t mean the output was wrong. It means it scored below your defined threshold on your rubric.

That distinction matters.

You need a lot of them. A handful of test cases gives you anecdotes. A few hundred gives you a signal.

8. Golden Set

Everything in your eval gets measured against one thing: your golden set.

A curated collection of high-quality test cases.

Building a good golden set is harder than it sounds:

The instinct is to write examples yourself, covering the use cases you anticipate. That’s a start. But users phrase things differently than you expect. They hit edge cases you didn’t imagine.

And they might misuse features in creative ways.

A golden set built from your imagination reflects your imagination and not your users’. So seed it with real production queries instead. Anonymized, cleaned, representative.

Your golden set is your ground truth.

Treat it like a critical system artifact. Version and update it when you discover new failure modes.

9. Pass/Fail Threshold

Eval scores are rarely binary.

A rubric usually produces a score of 1 to 5, 0 to 10, or a percentage. The pass/fail threshold is what converts that score into a decision.

If your rubric scores range from 1 to 5 and your threshold is 3, any score below 3 is a failure. Simple in theory, but HARD in practice.

Setting the right threshold is a product call, not a technical one.

It depends on a few things: how much imperfection your users can tolerate. How severe are failures in your domain? The cost of false positives.

Set your threshold too low, and you’re shipping garbage. And if you set it too high, you’re shipping nothing.

10. Eval Coverage

Eval coverage is how well your golden set reflects real user inputs.

Most teams have low coverage and don’t know it. They built their golden set from examples they had written. Happy path, a few obvious edge cases. Meanwhile, production traffic is different: weird inputs, unusual phrasings, use cases nobody anticipated.

Low coverage means your eval suite is optimistic…

i.e., you’ll pass your tests but fail your users.

The fix isn’t writing more examples yourself. Instead, sample from production regularly and review failures. Then add the inputs that exposed new weaknesses to your golden set.

Eval coverage is something you build over time.

11. Temperature, Top-p, and Reproducibility

Temperature is the dial that controls how random your model’s outputs are.

Low temperature (close to 0) makes the model nearly deterministic: same input, same output, every time. High temperature makes it more creative. It samples from a wider range of probable tokens, producing more varied responses.

Top-p (nucleus sampling) is a related setting.

Instead of a randomness dial, it sets a probability cutoff. Only the most likely tokens make the cut. Top-p = 0.9 means the model considers only the top 90% most likely next tokens.

Both settings directly affect your eval results…

Run evals at temperature = 1.0 and the same prompt might pass today and fail tomorrow. NOT because your model changed, but because randomness swung against you.

Here’s the standard practice: set temperature to 0 during eval runs. And lock in determinism. If you need creative variance in production, test at your production temperature.

Just know you’re accepting noisier results…

12. Statistical Rigor

Even at temperature = 0, a single eval run isn’t enough.

Your golden set is a sample of your input space. That sample has its own variance. One unlucky set of examples can make a good prompt look bad. And one lucky set can make a bad prompt look good.

So run many evaluations across different samples.

Then report the mean and the variance, not just the score. When comparing two prompt versions, check whether the difference is real. Or if it’s just noise.

In practice: if you change a prompt and your score goes from 4.1 to 4.3, that might be a real improvement.

It might be a random fluctuation. Without variance1 data across many runs, you can’t tell the difference. Most teams run once, report the number, and ship. That’s how confident regressions get deployed…

How Do You Score Outputs?

You have criteria, a rubric, and test cases2.

Now the question is: who does the scoring? There are three options.

Each has different tradeoffs in cost, speed, and accuracy.

13. Human Evaluation

This is the gold standard.

A human reviews the output using your rubric and gives a score. It’s slow, expensive, and can be inconsistent. But it’s the closest thing to ground truth.

You can’t use humans for every eval run; it doesn’t scale.

But you shouldn’t remove human evaluation completely either. It’s what keeps your system grounded. Everything else: metrics, LLM judges, tests, is just an approximation of human judgment.

So use human evaluation strategically:

• To build and validate your golden dataset.

• To periodically check the accuracy of your automated evals.

• To debug when something breaks and you don’t know why.

This way, you balance cost with reliability.

14. Heuristic/Code Based Evaluation

This is the fastest and cheapest type of evaluation.

It uses simple code checks to validate the output's structural properties. For example:

• Is the response valid JSON?

• Is it within the character limit?

• Does it include all required fields?

• Does it avoid banned phrases?

• Does it match a specific format (like a regex)?

Heuristic eval is good at catching structural problems.

But they don’t measure quality. They can’t tell if a response is helpful, accurate, or well-written.

Think of heuristics as your first line of defense.

They catch basic issues before you run more expensive evaluations. They’re NOT enough on their own, but they’re an essential part of a complete eval system…

15. Semantic Similarity Evaluation

Sometimes you have a reference answer, a known-good response representing ideal output.

Semantic similarity evaluation measures how close your model’s output is to the reference. i.e., in meaning, and not exact wording.

This is where embeddings come in…

Each piece of text gets converted into a vector: a list of numbers that represents its meaning. Texts with similar meanings have vectors that are close to each other. Cosine similarity measures the angle between two vectors. A score of 1.0 means identical meaning. A score near 0 means unrelated.

This matters because string matching is too strict.

Take two sentences: “API returns a 404 error” and “Endpoint responds with a not found status.” They mean the same thing, but use different words. An exact match would call the second one wrong. Semantic similarity would call them equivalent.

But here’s the limitation: it only measures closeness to your reference.

It can’t catch a response that’s fluent and factually wrong. Not if the wrong answer happens to be semantically similar to the right one. So use it as a fast, scalable layer. And don’t rely on it alone!

16. Task Specific Metrics (BLEU, ROUGE, Execution-based)

For certain tasks, there are established metrics purpose-built for automated evaluation:

• BLEU (Bilingual Evaluation Understudy): Originally for machine translation. Measures n-gram overlap between generated text and a reference. Good for tasks where exact phrasing matters.

• ROUGE (Recall-Oriented Understudy for Gisting Evaluation): Designed for summarization. Measures recall: how much of the reference answer appears in the output? Useful when you care about coverage: did the model hit all the key points?

• Execution-based evaluation: For code generation, one metric matters. Does it run? Does it produce the correct output? Execution-based eval runs the generated code against test cases and checks the results. A function that returns the wrong answer fails. Another function that passes all tests succeeds.

But all three share the same trade-off:

They measure surface-level similarity to a reference, not actual quality. So use these where they fit.

17. LLM as Judge

This is what makes evaluation scalable…

LLM-as-judge uses a more capable model to evaluate your application’s outputs. You give it three things:

• Original input,

• Output you’re evaluating,

• Your rubric.

The judge then returns a score and an explanation.

Think of it like automated load testing.

Nobody clicks through 10,000 user flows to check for regressions. You can instead automate it. LLM-as-judge does the same thing for quality. It lets you run your rubric across thousands of outputs without thousands of human hours.

GPT-5 and Claude Opus are commonly used as judges. They’re capable enough to apply nuanced rubrics reliably. You’re calling them via API, NO custom training required.

But here’s the catch: judges can be wrong.

They have their own biases and blind spots. An LLM judge is an approximation of human judgment, not a replacement for it.

More on how to handle this in a moment…

18. Pointwise vs Pairwise Evaluation

These are 2 flavors of LLM-as-judge evaluation:

• Pointwise

  • You ask, “Score this output on a scale of 1–5.”

  • It’s simple and fast.

  • And each output needs one evaluation call.

• Pairwise

  • You ask: “Here are two outputs. Which one is better?”

  • This is usually more reliable because comparing is easier than scoring.

  • It’s especially useful when you want to know if a new prompt is actually better.

But the downside of pairwise is the cost…

You’re evaluating two outputs instead of one, so it can be twice as expensive. At a small scale, this doesn’t matter, but at a production scale, it adds up quickly.

Because of this, the industry standard is to use a tiered approach:

• Online evaluation (production monitoring)

  • Use fast, cheap methods like heuristics or smaller, fine-tuned judge models.

  • These can run continuously without high cost or latency.

• Offline evaluation (pre-ship testing)

  • Use the best LLM judge you have.

  • Run pointwise and pairwise comparisons.

  • This is acceptable because you’re only running it before deployments.

Think of the powerful judge model as a gate before deployment, and NOT something you run on every request.

19. Judge Calibration

Before you rely on an LLM judge, you need to know how well it matches human judgment…

Judge calibration measures this:

You take a sample of outputs, have both humans and the judge score them, and then check how often they agree.

• If agreement is high, your judge is a good proxy for human evaluation.

• If agreement is low, the judge may be measuring the wrong thing.

A poorly calibrated judge is risky.

It can give you confidence in results that aren’t actually correct. So calibrate your judge before using it. Then recalibrate periodically, especially after changing models or prompts.

Get 20% savings by getting a group subscription right now:

Get 20% off a group subscription

RAG System Evaluation

Most LLM applications today aren’t just “prompt in, response out.”

They use Retrieval-Augmented Generation (RAG): first fetch relevant documents, inject them into the context, and then generate a grounded response.

This helps reduce hallucinations because the model can rely on real data rather than just its training data. But it doesn’t solve the problem completely…

It also introduces new failure modes that standard LLM evaluation doesn’t catch…

20. RAG Triad

The standard framework for evaluating RAG systems has three dimensions:

1 Faithfulness

Did the answer actually come from the retrieved context?

This often surprises people. You can retrieve the perfect documents, and the model can still ignore them. Faithfulness checks whether the response is grounded in the provided sources.

A failure here looks like a confident answer that isn’t supported by the retrieved documents. So it’s just a hallucination with extra steps.

2 Answer Relevance

Did the response actually answer the user’s question?

A response can be faithful to the context and still miss the point. The model may use the right documents but answer the wrong question.

Answer relevance measures how well the response matches the user’s intent. While Faithfulness checks if the model stayed within the context.

3 Context Precision

Did the retrieval step fetch the right documents?

Even a perfect generator will fail if the input context is poor. If retrieval brings in weak or loosely related documents, the model either guesses or tells the user it doesn’t know.

Context precision evaluates the retrieval stage. It checks how many of the retrieved documents were actually relevant.

Think of a RAG system as three stages: retrieval, augmentation, generation.

Each stage can fail on its own… The RAG triad is your observability layer to isolate where the problem is, so you know what to fix.

21. RAG-Specific Failure Patterns

Understanding the RAG triad is one thing. Spotting failures in practice is another…

Here are the three most common failure patterns:

1 Retrieval returns irrelevant chunks (Context precision failure)

The system retrieves the wrong or loosely related documents. And the model then either:

• Hallucinates to fill the gaps,

• Or says it doesn’t have enough information.

The fix is in your retrieval layer: embedding quality, chunking strategy, re-ranking.

2 Retrieval is correct, but the model ignores it (Faithfulness failure)

The right documents get retrieved, but the model doesn’t use them.

This is usually a prompting issue. The model isn’t being forced to rely on the provided context. The fix is to strengthen the prompt, so the model stays grounded to the sources.

3 The answer is grounded but doesn’t help the user (Answer relevance failure)

The answer is technically correct and grounded, but doesn’t solve the user’s problem. This usually means your knowledge base is missing the right information. The fix is to improve or expand your data, NOT your prompt.

Get 20% savings by getting a group subscription right now:

Get 20% off a group subscription

Offline vs Online

Scoring a single output is one problem.

Building a system that reliably catches failures (before and after deployment) is another. There are two environments where evaluation happens:

22. Offline Evaluation

Offline evaluation happens before you ship.

You make a change: a new prompt, model version, or retrieval strategy. Before it reaches users, you test it against your golden dataset. Then you score the outputs and compare them to your current system.

Think of this like a continuous integration (CI) pipeline for LLMs.

A change shouldn’t go live unless it passes your evals. If quality drops, you catch it before deployment.

Plus, this is where you use your best (most accurate) judge model. For example, GPT-5, Claude Opus. Cost matters less here because you’re only evaluating a finite set of examples, and NOT each user request.

23. Online Evaluation

Online evaluation occurs in production.

You’re sampling live outputs continuously and scoring them. The goal is to catch issues that offline evals miss: edge cases, unusual inputs, and failures that only appear at scale.

Think of online eval as your monitoring system…It helps you catch problems before users report them.

Yet the main constraint is cost. You can’t run your most expensive models on each request. So online eval relies on heuristics and smaller judge models; it’s cheap enough to run continuously.

This means you trade some accuracy for scale…

Offline and online eval should work together:

• Offline eval catches known issues before deployment

• Online eval catches unexpected issues in production

So you need both for a reliable system.

24. Prompt Versioning and Regression Testing

Your prompt is code; treat it that way.

Track every prompt change: use version control, keep history, and compare differences. When something breaks, you should know exactly what changed and when.

Without versioning, you’d NOT know what changed.

Regression testing means running your eval suite against each prompt version. If your old prompt scored 4.2 and the new one scores 3.8, you’ve introduced a regression and caught it before it reached users.

This sounds simple, but most people don’t do it. They lack a solid eval setup: no golden dataset, no clear rubric, no infrastructure.

This is what separates people who iterate confidently from people who ship & pray.

25. Benchmark-Based Evaluation

Benchmarks are standard tests used to compare different LLM models:

1 MMLU (Massive Multitask Language Understanding)

Tests knowledge across many subjects like math, science, law, and medicine. Think of it as a general knowledge exam for LLMs.

2 HellaSwag

Tests common sense reasoning. The model is given the start of a scenario and must predict what happens next. (Many earlier models struggled with this.)

3 HumanEval

Test code generation. The model gets a function signature and must write the correct implementation. It’s measured using pass@k: how often the model gets the right answer within k attempts.

Here are two useful leaderboards:

• Hugging Face Open LLM Leaderboard for open-weight models

• Chatbot Arena (LMSYS), where models get ranked based on human preferences

Remember, these benchmarks are useful for comparing models, but they don’t always reflect real-world performance for your specific use case…

26. Benchmark vs Real-World Tradeoff

Benchmarks don’t tell you if a model will work for your specific use case.

A model that scores 90% on MMLU might still struggle with your domain-specific language. A model that performs well on HumanEval might generate code that doesn’t fit your standards… Benchmarks measure general capability, but your application might need a specific capability.

So use benchmarks to narrow down your options. Then use your own eval on your own golden set to make the final decision.

27. Dataset Contamination / Data Leakage

There’s a problem with benchmarks that most engineers overlook:

Dataset contamination occurs when evaluation data overlaps with the model’s training data. The model has already seen the answers. So its high benchmark score reflects memorization, NOT capability.

This happens because training data and benchmark datasets come from the internet. And the data overlap is often unknown. Over time, widely used benchmarks become less reliable, which is why new ones keep getting created…

To avoid this, don’t rely on public examples for your evals…Plus, use real user queries and create your own datasets instead.

Failure Modes (What Not to Do)

You can have the right tools and still build a BAD eval system.

Here are the common mistakes:

28. Eval Anti-Patterns

Vibe-based evaluation.

“I tried it a few times, and it looked good.”

This is the most common mistake. Informal spot-checking creates false confidence. It doesn’t catch edge cases, track performance over time, or scale beyond one person looking at a few outputs.

Vibes are a starting point, but not an eval system.

The single-sample trap.

You run your eval suite once and report the results.

But LLMs are nondeterministic…

• A bad prompt might look good

• A good prompt might look bad

So run many samples and aggregate results. Remember, report variance, not just average scores.

Goodhart’s Law in disguise.

“When a measure becomes a target, it stops being a good measure.”

If you optimize for a metric too aggressively, the metric stops measuring what you care about…

• Reward confidence: you’ll get confident hallucinations

• Reward length: you’ll get long, low-quality answers

Metrics are only a proxy for quality. So don’t mistake them for the actual goal.

Eval-production mismatch.

Your golden dataset reflects what you expect.

Yet real users behave differently. They’re vague, unpredictable, and use your system in ways you didn’t anticipate. If your evals don’t reflect real usage, your scores are misleading. A high pass rate on unrealistic data doesn’t mean your system works.

Ignoring tail failures.

A 92% pass rate sounds good. But what about the 8%?

LLM failures are often catastrophic, not graceful. One harmful or incorrect response can matter more than hundreds of correct ones. So always review failure cases, not just averages.

The worst outputs tell you the most…

Get 20% savings by getting a group subscription right now:

Get 20% off a group subscription

Decision Framework

There’s no single tool that solves LLM evaluation.

What works is a layered system… Each layer catches problems the previous one misses…

29. Eval Stack

Layer 1: Heuristics.

Fast, deterministic, and cheap. These run on every output and catch structural issues such as incorrect format, missing fields, or banned content.

Layer 2: Semantic similarity and task-specific metrics.

These run automatically against reference answers. They help catch meaning-level failures without the cost of an LLM judge.

Layer 3: LLM-as-judge (offline).

This runs before deployment on your golden dataset using your best judge model. Its job is to catch quality regressions before they reach users.

Layer 4: LLM-as-judge (online, smaller model).

This runs in production on a sample of live outputs using a smaller, cheaper judge model. It helps catch failures at scale that offline evals missed.

Layer 5: Human spot checks.

These run periodically. And keep you calibrated, verify automated evals scoring still match human judgment, and add new failure cases back into your golden dataset.

Each layer has a different cost profile and catches a different failure type…

Eventually, you want all five. But you do NOT need all of them on day one.

So if you are starting from scratch, here is a simple three-step MVP:

Step 1: Build a golden set of 50 examples.

Don’t write all of them yourself.

Use real user queries if you have them. If not, create 30 representative examples and 20 edge cases. Also, include normal cases, adversarial inputs, and strange phrasings you expect users to try.

This dataset becomes the foundation of your eval system.

Step 2: Add one deterministic heuristic.

Pick the single most important structural requirement for your output.

That might be length, format, a required field, or a banned phrase. Then write a simple code check for it. This is quick to build and catches more failures than most people expect.

Step 3: Add one LLM judge prompt.

Take your rubric and turn it into a judge prompt.

Then use a strong model to score outputs on one important quality dimension, and run it on your golden dataset. Read the scores and the explanations carefully.

You will find something surprising…that’s the point.

That is your MVP eval system.

Everything else builds on top of it: pairwise comparisons, online monitoring, RAG scoring, regression testing, and more.

Start here…

Closing Thoughts

Evaluation isn’t something you do at the end.

Instead, it’s what makes iteration possible. Without it, you’re not doing LLM engineering. But guessing and shipping changes, only to find out from users that they didn’t.

Traditional software engineering learned this lesson the hard way: CI, automated tests, production monitoring…these aren’t optional. They’re what make fast, reliable development possible.

LLM systems need the same discipline. The difference is that the outputs are nondeterministic and hard to measure.

Now you have the vocabulary to deal with that…

Start simple and build your eval stack step by step. Over time, you’ll know whether a change improved quality before users ever see it.

👋 I’d like to thank Anshuman for writing this newsletter!

If you’re building LLM applications and want to go deeper, follow him on LinkedIn and X.

Don’t forget to check out his newsletter, AI Proof -- it’ll help you stay relevant in the AI era.

Louis and I launched the GENERATIVE AI MASTERCLASS (newsletter series exclusive to PAID subscribers) this month.

When you upgrade, you’ll get:

• Simple breakdown of real-world architectures

• Frameworks you can plug into your work or business

• Proven systems behind ChatGPT, Perplexity, and Copilot

👉 CLICK HERE TO JOIN THE GENERATIVE AI MASTERCLASS

(Golden members will get the next Generative AI newsletter in the first week of May.)

If you find this newsletter valuable, share it with a friend, and subscribe if you haven’t already. There are group discounts, gift options, and referral rewards available.

Subscribe now

Want to reach 200K+ tech professionals at scale? 📰

If your company wants to reach 200K+ tech professionals, advertise with me.

Thank you for supporting this newsletter.

You are now 210,001+ readers strong, very close to 210k. Let’s try to get 211k readers by 29 April. Consider sharing this post with your friends and get rewards.

Y’all are the best.

Share

• Share this post & I'll send you some rewards for the referrals.

You’ve built a search feature.

Then a user types a query, your application hits the database, and results come back ranked by relevance.

Simple enough, it works perfectly.

But you thought about how search needs to actually understand what users meant, not just match the exact words they typed.

A user searching for “how do I log in” should find results about “account access” and “sign in” even if those exact words never appear in their query.

This kind of search, where the database understands meaning rather than just matching words, is called semantic search. It requires a completely different approach to how you store and search your data.

Yet if you try adding this in as a feature, everything that made your traditional relational database fast will stop working.

The tools you relied on: indexes, WHERE clauses, keyword matching were all built for finding exact values.

Semantic search does NOT work with exact values.

Instead, it works with meaning, and meaning is harder to search efficiently. A search that took milliseconds before now takes seconds, and the bigger your dataset, the worse it gets.

Your database is NOT broken.

It’s doing exactly what you told it to do: comparing your search against every single item in your database, one by one, because there is no other way to find “similar” when there is NO exact match.

This is where vector databases come in.

Not because they are trendy, but because traditional databases were never designed to handle this kind of search at scale.

If you’re a software engineer encountering the term “vector database” for the first time, whether your team is adding AI features or you keep seeing it mentioned everywhere and want to understand what it actually means, this newsletter explains it from the ground up…

[Webinar] Stop babysitting your coding agents (Partner)

Agents can generate code. Getting it right for your system, team conventions, and past decisions is the hard part – you end up wasting time and tokens in correction loops.

More MCPs give agents access to information, but not understanding. The teams pulling ahead use a context engine to give agents exactly what they need.

Join Unblocked live on May 6 (FREE) to see:

• Where teams get stuck on the AI maturity curve

• How a context engine solves for quality, efficiency, and cost

• Live demo: the same coding task with and without a context engine

Register Now

(Thanks to Unblocked for partnering on this post.)

I want to introduce Maxine as the guest author.

She’s a cloud infrastructure engineer who spends her days scaling databases, debugging production incidents at 2 am, and writing about what actually works in production.

You can get a copy of her LLMs for Humans: From Prompts to Production (at 30% off right now). It’s 20 chapters of practical applied AI with real production context, not theory. And it’ll help you get smarter about using AI tools in infrastructure workflows.

Checkout her work:

• Substack - I Love DevOps

• LinkedIn

• Threads

Plus, if you’re thinking about making a career move into cloud or DevOps and want a structured path to get there, get a copy of her The DevOps Career Switch Blueprint.

Inside this newsletter, you’ll get:

• Why traditional search breaks. How exact-match databases fail with semantic meaning and why O(n) similarity search doesn’t scale.

• How vector search works. Embeddings, high-dimensional space, and why similarity is measured by distance instead of exact matches.

• The core architecture. Storage, indexing (HNSW), and query layers, with key tradeoffs in cost, latency, and accuracy.

• Query execution in practice. ANN search, metadata filtering (pre vs post), and ranking, including how real systems return results.

• Production realities. Scaling, replication, cost structure, monitoring, and failure modes teams hit in production.

• Decision framework. When pgvector is enough, when to use a dedicated vector DB, and when distributed systems become necessary.

Why Traditional Databases Struggle with Embeddings

Traditional relational databases are good at exact matches and range queries.

They use indexes and data structures so the database can jump directly to relevant rows without scanning the entire table.

When you query WHERE user_id = 12345, database uses an index to find that row instantly, the same way you’d look up a word in a dictionary: you don’t read every page, you jump straight to the right section. Also when you query, WHERE created_at > ‘2026-01-01’, it works the same way for dates.

But embeddings break this model because there is no exact value to look up!

An embedding is a list of numbers that represents the meaning of a text.

Words, sentences, or entire paragraphs get converted into these numbers by an AI model, and similar meanings produce similar numbers.

Instead of finding a specific row, you find rows closer in meaning, and “close” requires measuring distance in high-dimensional space rather than matching values. Think of it like a regular map with latitude and longitude, except you’re working with 1,536 dimensions instead of just 2.

Let’s walk through a concrete example to see why this creates a problem…

Imagine you have a document database where each document gets split into chunks.

And think of chunks like cutting a book into individual paragraphs so you can search each one separately. Each chunk gets converted into a 1,536-dimensional vector using an embedding model like OpenAI’s text-embedding-3-small:

1. A user submits a search query

2. App converts the query into a vector using the same embedding model

3. Now you have to find the “most similar” document chunks

Without specialized indexing, the only way to find “most similar” ones is to calculate the distance between your query vector and every chunk vector in your database, sort by distance, and return the top results.

Let’s break down what it means…

A vector is simply a list of numbers. In AI, those numbers represent meaning.

When you convert text into an embedding, you’re assigning it a location in a very high-dimensional space (imagine a map with 1,536 dimensions instead of the normal 2, latitude and longitude). Similar pieces of text end up close to each other on this map, while unrelated text ends up far apart.

When a user searches for something, their query also gets converted into coordinates on this map. To find the most relevant results, you need to measure the distance from the query’s location to every document chunk’s location in your database, then return the closest ones.

But here’s the problem…

Without an index, there’s no shortcut. You must measure the distance to each item.

Now imagine you’re standing in Times Square in New York City, and you need to find the 10 closest coffee shops. Without a map or any organized system, your only option is to:

1. Walk to every building

2. Check if it’s a coffee shop

3. Measure how far it is from Times Square

4. Keep a list of the 10 closest ones you’ve found

If there are 10k buildings, you check all 10k. And if there are 10 million buildings, you check all 10 million.

The more buildings that exist, the longer it takes…

Put simply, it takes O(n) time, and it grows linearly with the size of your dataset.

In practical terms, if you have 10,000 document chunks and each distance calculation takes 0.0001 seconds, one query takes 1 second. That’s not terrible.

But what if you have 10 million chunks?

That same query now takes 1,000 seconds, i.e., 16 minutes. And if your application serves 100 queries per second from different users? Your system immediately grinds to a halt!

This is why traditional databases struggle with embeddings…

Their indexes are built for exact matches.

They can jump straight to user_id = 12345 without checking every row. But with vectors, there are no exact matches, only “similar” matches, and similarity requires measuring distances.

Let’s now imagine finding a similar song to the one you already like…

You cannot just look it up by name; you have to compare it against every other song across hundreds of characteristics: tempo, key, energy, mood, and so on. The more songs in your library, the longer it takes.

Vector search works the same way, except instead of songs, you compare text, and instead of a few characteristics, you are comparing across 1,536 dimensions at the same time.

Without specialized techniques to avoid checking everything, the math simply doesn’t scale.

This is where people often bring up pgvector, a PostgreSQL extension that adds vector search capabilities to Postgres. It’s a good option for smaller datasets and lets you avoid adding another database to your stack. This keeps operational efforts very low.

For datasets with fewer than 100,000 vectors and moderate query volumes, pgvector works well and keeps your infrastructure simpler.

But as your dataset grows into the millions and your query latency requirements tighten, you hit the limits of what a general-purpose database can do for this specific workload.

You don’t need a specialized vector database just because you’re working with embeddings, but you need to understand when the operational trade-offs tip in favor of dedicated infrastructure.

This brings us to the core challenge that vector databases solve…

How do you find approximate nearest neighbors in high-dimensional space without comparing against every vector, while still returning results that are “good enough” in terms of accuracy?

The answer involves building specialized index structures that trade accuracy for massive speed improvements, and understanding those trade-offs is essential for operating these systems in production.

Architecture: How It Works Under the Hood

Vector databases have three key layers: storage, index, and query.

Each layer involves trade-offs that directly affect your operational concerns: cost, latency, accuracy, and scalability.

Let’s break down what matters from an infrastructure perspective…

Storage Layer: Where Vectors Live

Vectors are massive.

A single 1,536-dimensional vector using 32-bit floats takes up 6KB of storage. If you multiply that across millions of vectors, storage costs add up fast. This is a problem every vector database must solve at its foundation: the storage layer determines how vectors are physically stored, compressed, and retrieved from disk. This layer focuses on compression techniques that reduce both storage costs and the memory footprint while preserving as much accuracy as possible.

The most common approach is quantization.

Quantization reduces the precision of each dimension to use fewer bits.

Instead of storing each dimension as a 32-bit float, you quantize to 8-bit integers (scalar quantization) or binary representations (binary quantization).

This can reduce storage and memory requirements by 4x or more, but it introduces approximation error. Approximation error is the gap between the compressed version of data and the original.

When you round numbers to save space, your calculations become slightly less precise. In vector search, this means you might occasionally miss a result that was truly the closest match, but in practice, the difference is small enough that most applications never notice it.

In production, you’re constantly trading off storage costs for search accuracy, and the right balance depends on your use case.

Another critical decision is the trade-off between memory and disk…

For maximum query speed, you want your vector index in memory, but that’s expensive at scale. Some vector databases support tiered storage, where frequently accessed vectors stay in memory while cold data lives on disk, or hybrid approaches in which the index structure stays in memory while raw vectors get pulled from disk when needed.

These architectural choices directly affect your cost structure and query latency percentiles.

From an operational perspective, you need to understand how your chosen vector database handles durability and recovery:

• What happens if a node crashes mid-write?

• How long does it take to rebuild indexes after a restart?

• Are vectors written to disk synchronously or asynchronously?

These aren’t theoretical questions. They’re the difference between acceptable downtime and a major incident when something goes wrong.

Indexing Layer: Graph Structure

This is where the “magic” happens, and by magic I mean carefully engineered data structures that let you find approximate neighbors without exhaustive search.

The most popular algorithm in production vector databases is HNSW (Hierarchical Navigable Small World), and it’s worth understanding at a conceptual level even if you’re not implementing it yourself.

The core idea behind HNSW is based on “small world” networks, the same principle that says you’re only a few connections away from any other person on the planet.

Instead of comparing your query vector against every vector in your database, HNSW builds a multi-layer graph in which similar vectors are connected to each other. When you query, the algorithm starts at the top layer and navigates toward your target by following edges to similar neighbors, dropping through layers until it reaches the most similar vectors at the bottom layer.

Remember our map analogy?

Your vectors are locations on a high-dimensional map, and you’re trying to find the closest ones to your query without checking every single location. HNSW solves this by building a navigation system with different zoom levels.

Think of it like Google Maps navigating from New York to a specific coffee shop in San Francisco:

• Top layer (zoomed out): From New York, HNSW identifies the general direction of your target, the West Coast, and follows connections to get you to California. You’re not considering every city in America, just the major waypoints that get you closer to where you need to be.

• Middle layers (zooming in): Now you’re looking at California. The algorithm narrows down to the San Francisco Bay Area, following connections between nearby locations. You’re still not checking every street, just the neighborhoods that are directionally correct toward the coffee shop you want to visit.

• Bottom layer (street level): Finally, you’re at the specific neighborhood and can see individual addresses. The algorithm follows the last few connections to find the exact coffee shop location (or, in vector terms, the most similar document chunks).

At each zoom level, you’re following connections between “nearby” locations without checking everywhere.

You only need to follow the connections that get you closer to your target. This is how HNSW turns an O(n) problem into something much faster: it builds a navigation structure that lets you skip large portions of the map in the wrong direction.

The key trade-off in HNSW is speed versus accuracy, controlled by two parameters: the number of connections each vector has (called “M” or “max connections”) and the number of neighbors to check during search (called “ef” or “exploration factor”).

Higher values mean better accuracy, but slower queries and larger indexes…

Lower values mean faster queries, but you might miss some relevant results. In production, you tune these parameters based on your latency requirements and tolerance for accuracy.

Building the HNSW index is expensive.

Inserting vectors requires calculating distances and updating graph connections, which is why bulk index building often happens offline or during low-traffic periods. Index build time matters because it affects how quickly you can recover from failures, deploy updates, or onboard new data.

You do not need to understand the math behind HNSW to work with vector databases.

Just like driving a car, you might not know how the engine works, and that doesn’t affect how you drive. But knowing that the engine exists and what it does helps you figure out what is wrong when the car breaks down.

What you need to know is that HNSW produces approximate results, not perfect ones, and it’s possible to adjust its approximation. When search results seem wrong, queries suddenly slow down, or the index fails to build, this understanding helps you identify the problem.

Get a copy of LLMs for Humans: From Prompts to Production right now and save 30%.

Reminder: this is a teaser of the subscriber-only newsletter, exclusive to my golden members.

When you upgrade, you’ll get:

• High-level architecture of real-world systems.

• Deep dive into how popular real-world systems work.

• How real-world systems handle scale, reliability, and performance.

And much more!

Unlock Full Access

Query Layer: How Requests Move Through System

When a query hits your vector database, it follows a specific path:

Read more

• Share this post & I'll send you some rewards for the referrals.

When you build software with LLMs1, you need to decide how much control to give the model.

Does your code run every step, or does the model figure out the steps on its own?

Agentic patterns are design patterns for making that decision. They’re the same architecture choices you already make in any system: who controls what happens next, what happens on failure, and how data moves between components.

The difference is that some of those components are now language models.

This newsletter covers nine of those patterns…

On one end, workflow patterns where your code controls every step. On the other hand, agent patterns where the Large Language Model (LLM) decides what to do next.

That boundary is a decision about how much control to hand to the model: at some point, you stop telling the model what to do and start letting it figure that out.

The first question isn’t which pattern to use. It’s whether you need one at all.

Let’s start there...

Find out why enterprise leaders are doubling down on data foundations for AI (Partner)

AI is moving fast, but your data foundation isn’t keeping up?

That’s exactly why leaders from JPMorgan, Mercedes-Benz, Siemens, and Roche contributed to this AWS book on agentic analytics.

Here’s what you get:

• Real strategies from enterprise leaders: Learn how top companies are building data foundations for AI systems that actually scale.

• Practical frameworks you can apply: From data strategy to data products, ML, and agentic AI.

• Perspectives from 15+ enterprise leaders: Each chapter brings a unique view from senior leaders across industries.

• Complex topics broken down clearly: Understand what matters without getting lost in theory.

An AWS book designed to help you build data foundations for intelligent agents that scale.

Download it now and learn how to build systems ready for intelligent agents.

Get the AWS Data & AI Book

(Thanks to AWS for partnering on this post.)

Inside this newsletter, you’ll get:

• The shift from prompts to systems. Why simple LLM calls break down, and how workflows and agents introduce structured decision-making.

• A practical escalation ladder. When to stay simple, when to use workflows, and when agents are actually justified.

• Workflow patterns. chaining, routing, parallelization, and orchestrator-workers, with clear tradeoffs in cost, latency, and failure modes.

• Agent patterns. Reflection, tool use, ReAct, and planning, focusing on how the model makes decisions and where things go wrong.

• Evaluator-optimizer loop. How to add a reliability layer with evaluation, iteration limits, and cost control.

• Real-world case study. Combining multiple patterns in an AI code review pipeline, showing how to choose the simplest architecture that works.

Not everything needs an agent

Most tasks don’t need an agent2…

Most tasks don’t even need a workflow…The default should be the simplest setup that works, and you escalate only when that setup breaks.

Start with a direct API call to an LLM. You can do more with this than most people think:

• Summarization

• Classification

• Extraction

• Rewriting

• Translation

• Code generation with clear specs

Most people skip this level too quickly!

Workflow patterns are the next step up.

Escalate when a task has many steps, and when focused attention at each step improves the output. Between those steps, you define validation gates: checks that verify the output before passing it forward. The defining feature: you can write every step down before the system runs. Your code owns the control flow.

And the LLM handles the details.

Agent patterns are for when the number and type of steps are unknown until the system runs.

The model needs to look at results, decide what to do next, and choose its own path. For example, say you built a support agent that handles order cancellations. A customer asks to cancel, but the agent doesn’t know upfront whether it needs to check the refund policy, look up the shipping status, or pass the issue to a human.

The steps depend on what it finds…

Here’s how you know it’s time to switch: you write more code handling errors and exceptions than doing the actual work. Or you keep adding special cases for situations the LLM encounters you didn’t plan for.

If you can still write down all the steps before the system runs, stick with a workflow.

Agent patterns sit within multi-agent architectures, i.e., many agents cooperate on the same problem, each with its own role.

One common mistake: you get your system to 70-80% of a prototype and assume the architecture needs upgrading. It usually doesn’t…The real issue is usually prompt quality or missing validation gates. Move to a more complex pattern only after the simpler one has actually failed, not when it feels limiting.

With that baseline set, here are the four workflow patterns:

Workflow patterns: you control the flow

With workflow patterns, your code controls what happens.

You decide the steps, the order, and the checks between them. The LLM handles each step, but your code is in charge…

1. Prompt chaining

Prompt chaining breaks a task into a series of LLM calls.

Each call takes the output from the previous step, and a check runs between them to ensure the result is correct before moving on.

If you’ve ever set up a CI/CD (Continuous Integration/Continuous Deployment) pipeline, this is the same idea. A build pipeline compiles, lints, tests, and packages. Each step must pass before the next one runs.

If tests fail, the pipeline stops.

For example, in legal contract review, this looks like: extract all clauses from a contract, classify each clause by risk level, then generate a summary of high-risk items for human review. Products like Thomson Reuters CoCounsel and Robin AI run variations of this pipeline.

Each step gets a focused prompt with a narrow task, and a validation gate checks the output before passing it forward.

Tradeoffs

Latency grows linearly…

Five steps mean five round-trip. Errors carry forward: if step two misclassifies a clause, steps three through five process bad input. The gates between steps catch these failures early, but they only catch what the conditions you write can catch.

2. Routing

A routing pattern classifies the input and sends it to the right handler: a different prompt, a different model, or a different sub-workflow.

Each handler is built for one type of input.

This is like a hospital front desk. A patient walks in, the front desk checks their symptoms, and then sends them to the right doctor. The front desk doesn’t treat anyone. And the doctor doesn’t check in patients.

Sierra AI routes across 15+ models this way.

Classification tasks go to one model, tool calling to another, and response generation to a third. Customer support platforms like Intercom Fin route each customer request based on what the customer wants and how they feel about it, sending requests to either an AI handler or a human agent.

The cost argument is built into the pattern…

Simple queries hit cheap, fast models. While complex ones hit expensive, capable ones. You’re not paying the highest prices for work that a smaller model handles well.

Tradeoffs

In 1 word: misclassification.

A complex query routed to the cheaper model produces a poor answer. A simple query routed to the expensive model is just a waste of money.

Either way, the router is a single point of failure, and its accuracy sets the upper limit for the entire system.

3. Parallelization

In software, parallelization means running many tasks simultaneously rather than one after another. With LLMs, it has two variants that solve different problems…

Sectioning splits a task into independent parts, runs them in parallel, and merges the results. This is like a construction crew. Electricians, plumbers, and carpenters work in the same building at the same time. Their work gets combined at the end.

GitHub Advanced Security works this way: CodeQL and third-party tools like Snyk and Semgrep scan the same pull request in parallel, each producing independent security alerts.

Voting runs the same task many times with different prompts and combines the answers. This is like a jury. Each juror looks at the same evidence and gives their own opinion, and the group decides together.

You’d build this for something like a security review: three separate prompts analyze the same code change for vulnerabilities, and you flag an issue only if two or more agree. Running the task many times is more expensive, but it catches fewer false alarms.

In short, sectioning gives each worker a different job; voting gives every worker the same job.

Tradeoffs

Cost multiplies with every parallel branch.

Partial failures are a design decision you need to plan for up front: if one branch fails, do you retry it, proceed with the others, or fail the whole operation? There’s no universal answer, and a wrong choice is expensive.

4. Orchestrator-workers

The orchestrator-workers pattern uses one central LLM to break a task into subtasks, assign each subtask to a worker LLM, and combine the results.

The difference from the earlier patterns is that the subtasks aren’t known in advance. Orchestrator figures them out as it goes. This is like a general contractor building a house:

• They hire the right people, but don’t know every subcontractor they’ll need upfront

• Plumber finds a structural problem, so the contractor brings in a structural engineer

• The work plan changes as the project reveals new problems

Cursor’s agent mode works this way. It starts with different agents for different parts of the codebase:

• One adding tests

• Another updating documentation

• A third refactoring shared utilities

You’re trusting the LLM to figure out what work needs to be done, not just to do the work you’ve assigned.

This uses several LLMs, but it’s NOT a multi-agent architecture: a single central LLM remains in control of the entire process. In multi-agent systems, agents can call each other directly without a central coordinator, and control can transfer between them.

Tradeoffs

The orchestrator can lose track of the original goal.

It breaks simple tasks into too many subtasks. And when workers finish faster than it can combine results, the orchestrator itself becomes the bottleneck.

These four patterns get gradually more powerful and more expensive…

Chaining and routing are cheap and predictable: you know what will happen before it runs. Parallelization saves time but costs more. Orchestrator-workers handle problems you can’t predict upfront, but it introduces failure modes you can’t predict either. The first three patterns are simple to understand.

Most production systems shouldn’t need to go further than parallelization. When they do, the next four patterns hand control to the model itself...

Reminder: this is a teaser of the subscriber-only newsletter, exclusive to my golden members.

When you upgrade, you’ll get:

• High-level architecture of real-world systems.

• Deep dive into how popular real-world systems work.

• How real-world systems handle scale, reliability, and performance.

Unlock Full Access

Agent patterns: LLM controls the flow

With agent patterns, you STOP defining what happens and in what order.

Instead of defining steps, you define constraints: what tools are available, how much the model can spend, and when to stop. The model observes, reasons, and chooses what happens next.

That trust decision appears differently in each pattern: how much autonomy, over what actions, with what limits.

5. Reflection

Reflection is when the LLM generates output, reviews its own work, and then fixes the problems it found.

Read more

• Share this post & I'll send you some rewards for the referrals.

Most engineering resume advice you come across is junk.

At best, it comes from a tech recruiter who’s looked at 100s of resumes. At worst, it’s from yet another senior engineer cobbling together a blog post based on their experience applying to MAANGO.

This post is not like that.

I’ve made thousands of hiring and leveling decisions as mobile Hiring Committee Chair at Meta. I also led Meta’s engineering team responsible for finding and ingesting over 2M engineers’ resumes into their internal ATS (Applicant Tracking System).

I’ve seen a lot of resumes :)

And because of that, I know that you should treat your resume like a product. Think of it as a landing page for your career, designed to drive each target user who touches it to take action. And yes, your resume has multiple target users—four, in fact—and you should learn what each one is looking for.

(Oh, and btw, a lot of this applies to your LinkedIn profile too, since that is literally a landing page.)

Onward.

The tax you pay to run multiple agents (Partner)

If you’ve spent any time with coding agents, you know the feeling.

You start the morning with a clean plan. Spin up a few agents. One is refactoring the auth module. Another is writing tests. A third is scaffolding a new API endpoint. You’re flying.

Then, around 10:30 AM, you look up and realize you have 20 terminal windows open. One agent is blocked waiting for a decision you forgot to make. Another finished 40 minutes ago, and you never noticed. A third went sideways three commits back. You’re no longer flying. You’re drowning.

You’ve shifted from human as driver to human as director.

When running coding agents in parallel, the bottleneck isn’t just context. It’s your own attention trying to manage 10 agents across 10 terminals. You’re losing your mind to terminal chaos.

Meet Cline Kanban, a CLI-agnostic visual orchestration layer that makes multi-agent workflows usable across providers. Multiple agents, one UI. It’s the air traffic controller for the agents you’re already running, regardless of where they live.

• Interoperable: Claude Code and Codex compatible, with more coming soon.

• Full Visibility: Confidently run multiple agents and work through your backlog faster.

• Smart Triage: See which agents are blocked or in review and jump in to unblock them.

• Chain Tasks: Set dependencies so Agent B won’t start until Agent A is complete.

• Familiar UI: Everything in a single Kanban view.

Stop tracking agents and start directing them. Get a meaningful edge with the beta release.

Install Cline Kanban Today: npm i -g cline

Get Started Today

(Thanks to Cline for partnering on this post.)

I want to introduce you to Austen McDonald as a guest author.

Austen is a former Senior Engineering Manager and Hiring Committee Chair at Meta, where he spent almost 10 years and conducted over 1,000+ interviews. He’s the industry’s resident expert on behavioral interviews and writes about them on his Substack and in his book, Mastering Behavioral Interviews. As AI raises the bar for ownership and independent judgment, behavioral interviews matter more than ever.

For a few days only, use code SDN26 for 20% off Premium Access, which includes a print copy of his book and an AI copilot that helps you find your best career stories, shape them into high-signal answers, and deliver them well.

Part 1: Your Resume is a Product

Your resume is a landing page for your career search.

It’s the first thing these four personas see when they consider you for an engineering job:

• Machines (Applicant Tracking System),

• Recruiters (we’ll include sourcers for all you sticklers),

• Interviewers,

• And Hiring Managers.

Each of these personas looks progressively deeper into the content in the resume. Each one also extracts different, but overlapping, information.

Let’s consider the needs of each of these personas, what they look for in a resume, how they read it, and the decisions they make along the way…

Machines (ATS)

Unless your profile is handed directly to a recruiter, your hiring process starts with software. Whether you apply online, get referred, or turn up in a sourcer’s keyword search, you need to pass this automated filter first…

How ATS systems work:

When you submit a resume, the ATS parses it into structured data: your name, contact info, work history, education, and skills.

It then scores you against the job description on keyword relevance, years of experience, and other recruiter-configured criteria. High-scoring candidates float to the top of the pile; low-scoring ones may never be seen by a human.

Of course, companies use AI-powered search that goes beyond keyword matching. These systems understand that “built ML pipelines” and “machine learning infrastructure” mean the same thing. This is good news—it means you don’t need to stuff your resume with every keyword—but it also means your accomplishments need to be substantively relevant to the role and described as such.

What they need:

• Clean, parseable formatting. Fancy designs can break text extraction: tables, some multi-column layouts, text boxes, icons, or images. Even creative section titles like “My Journey” instead of “Professional Experience” are risky.

• Easy to find minimum qualifications since an automated tool is probably filtering first on whatever is stated in the job description.

• Keywords matching the job posting and role requirements.

• Technologies named where they can be easily indexed.

Hot Take: I have no data for this, but numerous companies have buttons on their career sites that say, “Upload your resume to see if you’re a fit.” Seems like it would be valuable to iterate on your resume until the site tells you you’re a good fit, lol.

Recruiters

Recruiters (or, more properly, sourcers) are speed readers scanning hundreds of resumes daily.

After basic machine filtering for experience, education, and past companies, they’ll skim your summary and a few accomplishment bullets, then make an in/out call in 5–7 seconds.

They’re looking for two things: candidates who meet the stated requirements, and candidates who fit the hiring manager’s mental model of a great hire. Those who clear both bars move forward to the next stage.

This is either a direct screening process, such as a phone call, or a final call to the hiring manager or committee before they begin screening process.

What they need:

• To immediately see that your profile fits the minimum requirements. They will reject you immediately if you don’t have the years of experience or the proper degrees, or have worked in similar roles (based mostly on title) that are identified in the job description.

• A quick visual path to something that excites them about you and convinces them you fit the idea of what the hiring manager is looking for. This could be a name-brand company, a project relevant to the hiring team, or frequent use of technology related to the role. Give them a reason to take action and move you to the next step.

• Brevity. Obviously, with 100s of these to review, the shorter the time all this takes, the better. So keep it to one page unless you’re a CEO.

Interviewers

There are a couple of places where interviewers or members of the engineering team read your resume.

Sometimes there is a layer of reviewers who approve the resume before they green-light a technical screen. It could be the hiring manager or a committee of ICs. You can see them more as a technical version of a recruiter, as they have the same needs.

Technical screeners are a different type of reader. They’re looking for conversation starters and signals that you’d contribute to the team.

Onsite interviewers (especially behavioral interviewers) are gauging your level based on the business impact you’ve driven. They’re also looking for red flags, such as performance challenges: short stints at a workplace or career gaps.

What they need:

• Easy to find accomplishments that look like they fit with the work of the company/team

• Evidence you can defend your claims under scrutiny: those accomplishments need technical details

• Signals of technical depth and leadership

• A minimum of reasons to be suspicious

Hiring Managers

Hiring managers are sometimes the first, as we talked about above in “Interviewers,” but almost always the last person to touch your resume.

At this stage, they’re reviewing a handful of finalists, which makes them the most thorough readers of your resume. They know what their team needs and look for specific evidence of alignment with those needs.

What they need:

• Evidence that your past work resembles what they’re hiring for

• Measurable impact with appropriate scope

• They want ownership and growth signals: can you self-direct, drive work to completion, and adapt fast, i.e., you’re ready for agentic coding.

• Clear visual hierarchy so they can find what matters

Now that you know who’s reading your resume and what they’re scanning for, there’s a natural question: what do you actually put on the page?

The answer isn’t “everything you’ve ever done.” Instead, curate a story of the impact you’ve made in your career. The next part walks you through that step by step...

Part 2: Resume Content that Converts

Now that we understand the users of your resume, let’s build a product that converts, moving you through the recruiting pipeline to a hire!

The Basics: What Goes In and What Stays Out of Your Resume

Before diving into content, we should get the basics out of the way.

Here’s the standard structure that readers expect, roughly in this order:

1. Contact Information – name, email, LinkedIn (maybe GitHub, if it’s good; more on that later)

2. Summary – Who you are and why you’re right for the role in 2-3 sentences

3. Work Experience – Your professional history with accomplishments

4. Education – Degrees, relevant coursework, honors

5. Projects – Side projects, open source, portfolio work

6. Skills – The keyword soup for machines, as we’ll see

This order should vary depending on who you are: experienced candidates can de-emphasize education; new grads should surface projects higher.

But readers expect to find things in roughly this order, so don’t make them hunt.

Contact Information

Of course, you should include basics like name and email.

If you want a phone call, include your phone number. Recruiters will ask for it later if you don’t give it now.

I also believe you should include a LinkedIn URL, linkified in the PDF form of the resume. Companies expect engineers to have LinkedIn profiles and may ingest that profile data automatically (don’t tell the LinkedIn lawyers). Much of the advice I give in this newsletter applies to LinkedIn profiles as well, especially the summary and bullet-point writing.

And please get a decent profile photo and cover photo for your LinkedIn profile.

Anything else you put in this contact information section should be additive to your candidacy. Like, don’t include a GitHub or personal website link if that’s there, it’s unimpressive. GitHub profiles need at least some recruiting-friendly repos that are deployed somewhere and well-documented. Personal websites should work (lol) and be a high-quality representation of yourself. Otherwise, don’t include them.

If you’re a US citizen applying to US roles and your citizenship isn’t obvious from your resume, note it in your contact section.

Write a Summary

Most resume advice is subjective.

Here’s my most opinionated take in this newsletter: always include a summary and put it at the top.

Two or three sentences max. And tailor it to the role. After reading it, I should immediately come away with the idea that you’d be a good fit for the role. Again, recruiters will heavily depend on this summary to help them decide.

⚠️ Warning: Don’t let AI write your summary. AI-generated summaries are an epidemic. They all sound the same—vague, buzzword-laden, and interchangeable. Recruiters and hiring managers can spot them instantly, and they leave readers with zero understanding of who you actually are.

These all say nothing:

Results-driven Front End Engineer with 4+ years of experience building scalable, user-centric web applications. Proven track record of delivering high-quality code in fast-paced environments. Passionate about creating seamless user experiences and collaborating with cross-functional teams.

Dynamic and detail-oriented Front End Engineer with expertise in modern JavaScript frameworks. Adept at translating complex requirements into elegant, performant solutions. Strong communicator with a passion for continuous learning and staying current with industry trends.

Innovative Front End Engineer bringing 4 years of hands-on experience crafting responsive, accessible interfaces. Committed to writing clean, maintainable code and driving technical excellence. Thrives in collaborative environments and passionate about mentoring junior developers.

My mind just numbs over reading these.

Phrases like “results-driven,” “passionate,” and “cross-functional teams” are filler. They fit any candidate, which means they describe none of them.

• What kind of products have you built? (B2B dashboards? Consumer apps? E-commerce?)

• What’s your technical specialty? (Performance optimization? Design systems? Accessibility?)

• What makes you different from the other 500 front-end engineers applying?

A Better Summary

Let’s build something better, remembering to tailor the summary to the role.

Suppose you were applying to some B2B SaaS company as this same front-end engineer who’s looking to replace their AI-built summary.

What about this one, that I typed with my own meat sticks:

I make complex data feel simple. I’m a front-end engineer with 4 years experience building B2B decision platforms at multiple fast-moving startups.

First, it’s shorter, and therefore more likely to be read.

My technical writing instructor in college told me, “Every line decreases your readership by half.” I don’t know if that’s true, but it’s a good principle to live by.

Next, it’s interesting and attractive. Who doesn’t want someone who makes complex data feel simple?

It also immediately communicates who you are: a front-end engineer with 4 years of experience at startups. Now I know whether your profile is relevant to me or not.

Finally, it connects to the role.

Note: Let’s talk about the “cute summary”. I actually prefer something like “Assistant to my local Claude Code instance” over a buzzwordy AI summary. At least it communicates some creativity and humanity. But I think you can do better and should do better unless it’s really obvious your resume matches the target role—like you’re applying for a senior ML safety engineering role at Big Tech X and just came from Big Tech Y with the same title.

Writing Bullets That Get Read

Resume readers are hiring you to deliver business value using technology.

That’s why they need an engineer. So give them confidence that you can do that with the bullets you list under your experience.

Here’s how you can do this:

• Lead with results, not activities. Start with what happened, not what you did. The activity is the supporting detail. Use a formula like: “Delivered [business value] using [technology/approach].”

• Quantify and show scope. Percentages, dollar amounts, user counts, latency improvements, and length of the project. Let the numbers imply how hard the work was.

• Your bullets become interview topics. The accomplishments you list will get asked about. Make them interesting enough to discuss and defensible under scrutiny.

• Include leadership signals. Highlight moments where you took initiative, managed others, or drove a project forward—even if you weren’t a manager.

• Lead with your best. Don’t sort accomplishments chronologically within a role. Place the most relevant and impactful ones first in each section.

• Embed tech in your accomplishments. Mentioning technologies within achievement bullets improves your ranking for those keywords, not just in a separate skills section.

Here are some examples of weak vs strong bullets:

Weak: Worked on the checkout system and fixed bugs.

Strong: Reduced checkout abandonment by 12% by identifying and resolving a race condition in the payment flow using React and Stripe’s API.

Weak: Responsible for migrating the database to a new system.

Strong: Led a 6-month migration of 50M records from MySQL to PostgreSQL, coordinating across 3 teams and achieving zero downtime during cutover.

Weak: Helped improve the CI/CD pipeline.

Strong: Cut deploy times from 45 minutes to 8 minutes by parallelizing test suites and implementing Docker layer caching, adopted by 12 engineering teams.

Weak: Mentored junior engineers on the team.

Strong: Mentored 3 junior engineers through their first production launches, with one promoted to mid-level within 8 months.

Projects & Proof of Work

I don’t think projects or other non-paid work are super valuable on a resume, unless you are in one of two positions:

• You’re a new grad with little paid experience, and projects are actually what you’ve done most of in your life,

• Or you’re trying to land a role in a new technical area where you haven’t technically worked (*cough* AI).

Outside of those two cases, projects are marginal. They might tip the scales if a recruiter is on the fence, but they rarely move the needle on their own.

Generally, though, prioritize contributions from paid work and don’t spend too much vertical space on projects.

GitHub can be useful. If your GitHub profile has a lot of green squares and/or a few polished repos deployed somewhere, with good READMEs, tests, and documentation, then great. Be sure to pin repos that showcase your best work. But if your contribution chart doesn’t look like a Christmas tree and/or you don’t have meaningful projects, just leave it off.

Side projects have some value. I don’t think side projects do a ton for you, since working by yourself with no expectation of production-level code is just a different environment than a real job. But they do show initiative and genuine interest. Skip the generic description (”AI-powered todo app in React Native”). Write something that shows personality: “Maximalist AI approach to running my life — a case study in build-fast-with-Claude-Code development.”

Technical blogs or writing are worth considering. Publishing shows communication skills and depth of understanding. If you provide anything, make sure it’s easy for me to find the 2-3 best pieces.

The Skills Section

If there’s one part of the resume that’s most often overlooked by readers, it’s the skills section. Even though that’s true, it still serves a purpose.

Include a skill soup section. A ranked list of technologies ensures you show up in keyword searches and reassures recruiters when hiring managers say, “They must know React.”

Keep it limited to what you actually know. Don’t list everything you’ve ever touched—only what’s relevant and what you can actually discuss. Some interviewers will probe for depth if you include an unholy number of skills. Also, I feel like I should charge you for my AI tokens if you include things like “HTML.”

Tailoring Your Resume Per Job

Honestly, most people don’t tailor their resume to roles because it’s just too much work. That’s probably true, especially if you’re applying to dozens of jobs.

Rather than tailoring per job, tailor for big-name companies, which are worth the extra effort, and keep a small set of versions for different role types. Perhaps you have one focused on your front-end work for tech-heavy senior IC roles, and another on your leadership skills for tech lead roles. Maybe one that highlights your AI personal projects and one that focuses on your full-time data engineering experience.

If you’re going to tailor, you need to know what to tailor to. Leverage the job description for this:

1. Look for repeated terms. If “distributed systems” appears three times, it matters more than “familiarity with GraphQL,” which only appears once.

2. Separate requirements from wish list. Most job postings have a “must have” and “nice to have” section—or they bury the real requirements in the first few bullets and pad the rest. Focus your tailoring energy on what appears at the top.

3. Identify the core problem they’re hiring for. “Looking for someone to own our data pipeline” means they want reliability and ownership signals. “Help us scale to 10x users” means they want performance and system design experience. Mirror this kind of language back to them.

4. Note specific technologies vs general skills vs filler. “Experience with Kafka” is a keyword you either have or don’t. “Experience leading teams” means they’re looking for some kind of people leadership and cross-functional signals. “Strong communication skills” is a filler that every job post includes. Don’t waste resume space on filler.

A quick trick: paste the job description into an LLM and ask, “What are the top 10 concepts in this job description and what are they looking for in this candidate?” Upload your resume and ask it to compare.

Now that you’ve identified what to focus on, here are some specific tactics to apply:

• Adjust your summary. Emphasize the aspects of your experience that align with the needs of that role or company that we just discussed, identifying.

• Reorder the sections. We’ll talk more about formatting below, but if you’re applying to AI roles but don’t have AI experience other than personal projects, leaving those projects at the bottom is a sure way to get ignored. Lift them up, even above the professional experience section if needed.

• Reorder your bullets. Lead with accomplishments most relevant to the target role.

• Expand or condense roles. Similar to reordering bullets, you can expand or condense on certain past roles that are most relevant, or even completely remove some experience if you think it’s distracting.

• Match the skills. If different skills are needed, then include those or reorder them to the front of the skills listing.

Now you have the raw materials: a summary, impact-driven bullets, and content tailored to your target roles. But even the best content fails if it’s buried in a cluttered design.

The next section is about making your resume visually effortless for each type of reader…

Part 3: Designing a Resume for Ease of Use

Your resume doesn’t have to be pretty, but it needs to fulfill the visual needs of the four readers. Aim for a simple, easy-to-use formatting:

• Most important stuff at the top: top of the page and the top of each new section

• Optimize for a clear visual flow, top to bottom

• Draw the reader’s eye to the most important parts with text size and weight. Be careful with color.

• Make it easy for machines to read

Formatting for Both Machines and Humans

Use simple formatting. Avoid overly creative designs. Distribute it as a PDF that exports cleanly to text. You can test this by uploading to Google Docs—if formatting breaks, the ATS may misread it too.

Keep it to one page. Admittedly, part of this is convention, but having everything on one page makes it easy for your readers to review. It’s like having the key elements “above the fold” on a product page. Here are some space-saving tips:

• Adjust margins, even all the way down to 1/2” on every side.

• Think about what you can fit on one line, such as the company name, title, years.

• Older or shorter experiences can sometimes be summarized in one line instead of a series of bullets. The farther the experience is down your resume, the less likely a reader is to actually read it, so feel free to reduce fidelity for those roles.

• Consider a two-column format, with the lesser-read parts like contact information, skills, and education in a smaller, side column.

Use formatting to guide the eye. Sub-bullets, bolding, and visual hierarchy. If every bullet looks equally weighted, eyes glaze over.

Don’t overthink templates. You’re not going to get hired because of your amazing resume template. Don’t spend too much energy on this. Find something that matches the principles here and then put your energy into the content.

Suggested Templates

I almost didn’t include templates at all, since there are so many good ones and the final formatting choices don’t matter that much. But how can this newsletter about resumes be complete without a couple of resume templates?

Here are a couple of templates that follow our guidelines: a single-column example and a two-column example. Notice how much more vertical space you save with the two-column format.

Making Your Experience Scannable

Make recognizable names visible. Make sure the eye jumps directly to household-name companies and high-impact projects. Recruiters notice brand recognition quickly, and readers only read the first part of each section.

Surface minimum qualifications. Many companies require recruiters to verify basic criteria, such as education or years of experience. Make this information prominent, as in the summary.

Prioritize your recent or flagship experience. Give more visual weight and spend more vertical space on the recent experience. Similarly, for experience at a name-brand company. Both of these will be the most-read content by reviewers. To accomplish this, compress older experience by removing bullets or shortening them.

Handling sticky situations: gaps and layoffs. Minimize these if you can. You don’t want to get screened out b/c of these before you can talk to someone:

• Remove the month indicators on roles that hide an intra-year gap.

• If you were recently let go, don’t update the current role with an end date yet. You can probably get away with this for 6-12 months since people expect resumes to be somewhat out of date.

• If you have recent stints at startups that folded after a year, add a brief one-liner explaining what happened. Otherwise, reviewers may assume it reflects on you.

At this point, you know what to write and how to format it.

But there are a lot of common mistakes I see in resumes. The good news is they’re fixable in five minutes.

Part 4: Common Mistakes & What to Avoid

Writing resumes can be tricky, so let’s consider some common advice I give coaching clients when we’re talking about resumes…

Common Resume Mistakes

Too many buzzwords. “Results-driven,” “synergy,” “leverage” all mean nothing really. If you can swap your bullet with any other candidate’s and it still makes sense, it’s too generic.

Zero impact. Describing activities without outcomes. “Worked on the payments team” tells me nothing. What shipped? What changed because you were there?

Hard-to-read templates. Fancy designs with unusual typefaces, lots of white space, etc., that prioritize aesthetics over scannability are pretty risky unless you’re applying to a design-heavy role. A reader should be able to make a decision about your resume in about 3 seconds.

Irrelevant information. Remove that summer lifeguarding job from 2015 when you’re a senior engineer in 2026. Every line that doesn’t serve the reader is a line they might stop reading at.

Wall of text. If I can’t immediately see where the key information is and there‘s no hierarchy, no bolding, and no visual breaks, then I just get overwhelmed and close your profile.

What NOT to Include

Unnecessary personal info. Age, photo (in the US), full address. These create opportunities for bias and waste space.

Irrelevant jobs. Unless they demonstrate transferable skills or fill a gap, cut them. Nobody hiring a backend engineer cares about your retail experience from a decade ago.

Social media links. Exception: LinkedIn. Your Twitter, Instagram, or TikTok is irrelevant noise unless you’re applying to a social media company and they showcase relevant work.

References available upon request. Everyone assumes this. It’s a filler.

Hobbies and interests. Unless directly relevant (you’re applying to a gaming company and list game development as a hobby), these just take up space.

ATS tricks. Don’t try things like packing your resume with white text to rock the ATS layer. Recruiters and engineers who write ATS systems know about these tricks and despise them.

Closing Thoughts

Your resume is never done.

That might sound exhausting, but it’s actually liberating. You don’t need to craft the perfect resume before you start applying. You need a good enough resume that you improve over time.

Treat your resume like a product. We started this newsletter with that idea, and it’s worth restating here. Products ship, get feedback, and iterate; your resume should too. If an interviewer or recruiter asks you about something on your resume, consider adding it. Did you bomb a question about an accomplishment you listed? Either cut it or prepare better.

Get human feedback early and often. Have friends, mentors, or peers review your resume. If you can, get feedback from someone who’s actually hired engineers.

Iterate ruthlessly. When I’ve worked with coaching clients on their resumes, it usually takes 3-4 revisions before it’s good.

The takeaway: Your resume has four readers—machines, recruiters, interviewers, and hiring managers—and each one needs something different. Build for all of them. Make it easy to scan, with the information each persona needs. Quantify your impact. Make it scannable. And then iterate.

Now go get that interview.

👋 I’d like to thank Austen for writing this newsletter!

Also, subscribe to his free newsletter, Mastering Behavioral Interviews, and check out his book by the same name.

It will help you prepare for the interview round you’re underinvesting in, the one that separates Senior engineers from Staff and above. He’s got a lot of practical advice similar to what he shared in this post.

Remember, you can claim 20% off Premium Access to his interview prep content with code SDN26, but only for a few days!

If you find this newsletter valuable, share it with a friend, and subscribe if you haven’t already. There are group discounts, gift options, and referral rewards available.

Subscribe now

Want to reach 200K+ tech professionals at scale? 📰

If your company wants to reach 200K+ tech professionals, advertise with me.

Thank you for supporting this newsletter.

You are now 210,001+ readers strong, very close to 210k. Let’s try to get 211k readers by 17 April. Consider sharing this post with your friends and get rewards.

Y’all are the best.

Share

• Share this post & I'll send you some rewards for the referrals.

I’ve built agents at Google to support their machine learning operations.

Then one thing became super clear to me: AI engineering is good software engineering with an understanding of a few key AI principles.

Just a year ago, hardly any software engineers were building with AI.

Now, engineers are expected to understand AI engineering and integrate AI into their products. This newsletter is a primer to bridge the gap from software to AI engineering.

Onward.

The tax you pay to run multiple agents (Partner)

If you’ve spent any time with coding agents, you know the feeling.

You start the morning with a clean plan. Spin up a few agents. One is refactoring the auth module. Another is writing tests. A third is scaffolding a new API endpoint. You’re flying.

Then, around 10:30 AM, you look up and realize you have 20 terminal windows open. One agent is blocked waiting for a decision you forgot to make. Another finished 40 minutes ago, and you never noticed. A third went sideways three commits back. You’re no longer flying. You’re drowning.

You’ve shifted from human as driver to human as director.

When running coding agents in parallel, the bottleneck isn’t just context. It’s your own attention trying to manage 10 agents across 10 terminals. You’re losing your mind to terminal chaos.

Meet Cline Kanban, a CLI-agnostic visual orchestration layer that makes multi-agent workflows usable across providers. Multiple agents, one UI. It’s the air traffic controller for the agents you’re already running, regardless of where they live.

• Interoperable: Claude Code and Codex compatible, with more coming soon.

• Full Visibility: Confidently run multiple agents and work through your backlog faster.

• Smart Triage: See which agents are blocked or in review and jump in to unblock them.

• Chain Tasks: Set dependencies so Agent B won’t start until Agent A is complete.

• Familiar UI: Everything in a single Kanban view.

Stop tracking agents and start directing them. Get a meaningful edge with the beta release.

Install Cline Kanban Today: npm i -g cline

Get Started Today

(Thanks to Cline for partnering on this post.)

I want to introduce Logan Thorneloe as the guest author.

He’s a software engineer at Google, building at the intersection of machine learning infrastructure, AI agents, and developer tooling.

He uses his background in machine learning research to teach software engineers the AI topics they should know via his newsletter, AI for Software Engineers.

You can also get an exclusive 25% off a paid AI for Software Engineers subscription as a reader of the System Design Newsletter.

1 Software 1.0, 2.0, and 3.0

“The hottest new programming language is English.” - Andrej Karpathy

There have been two recent AI-related shifts that have fundamentally changed software development and required software engineers to learn new concepts.

Both changes have been caused by the development of machine learning models: programs trained on data to recognize patterns and make predictions rather than follow explicit rules.

Andrej Karpathy details these changes by describing three different eras of software engineering:

• Software 1.0: hand-written logic via writing code. This was the paradigm from the inception of programming all the way to the advent of machine learning.

• Software 2.0: shift from explicitly writing logic via code to curating datasets and teaching algorithms to understand them. This shifts logic from a set of instructions to logic learned from data. A good example of this is Tesla’s Autopilot system, shifting from hand-written C++ code to neural networks1 for vision.

• Software 3.0: shift toward natural language2 being our new programming interface. This enables engineers to write code by speaking English (vibe coding). It also enables engineers to build agents3, or software that can think and act autonomously.

Software engineers need to understand these key AI principles to be productive in the software 3.0 paradigm…

2 Large Language Models (LLMs)

LLMs are natural language AI models trained to predict the continuation of text given a natural language input.

As with any AI model, understanding how it works best comes from examining its training data. LLMs are trained on a massive dataset of text from the internet. They’re trained to learn the relationship between words to best predict an output when given a specific string of inputs.

This relationship enables LLMs to use the knowledge stored in the semantics of language to understand relationships between concepts.

LLMs are also tuned to be non-deterministic; i.e., to provide different answers when an input is sent to the LLM more than once. This ensures an LLM’s output isn’t locked to a given input.

These two concepts make LLMs intelligent.

To relate LLMs to general software engineering, we can think of an LLM as a “non-deterministic database”.

This makes it easier to understand the impact of the knowledge encoded in its weights via semantic understanding. I like to compare it to a SQL database and a query: LLM is the database, and the prompt given to the LLM is the query that extracts the information in the format the user wants.

In AI engineering, it’s apt to think of an LLM as the central processing unit (CPU) of your application. By giving an LLM input information, it can make decisions on its own. Thus, the LLM becomes the CPU for your software, deciding what to do with the information provided.

This is also an apt comparison because choosing an LLM is like picking a CPU.

Each LLM has its own tradeoffs for latency, cost, functionality, and more. Choosing the right model (or models!) is important to meet the requirements of production software systems.

3 Context

Context is the information LLMs store in their memory and functions as the working memory for a task. For an LLM to perform a task or make a decision, it must have the relevant information in its context window4.

Context works by converting all input into tokens, which are small chunks of text (roughly 3-4 characters each) and feeding them into a model sequentially. The model uses an attention mechanism (a process that calculates how each token relates to every other token in the sequence) to determine which tokens are relevant to each other.

This is how LLMs ‘understand’ the relationships within the input mentioned above.

The tokens currently part of a conversation with an LLM are considered part of that LLM’s “context window”. Each time you interact with an LLM, the model processes the entire context window from scratch. It doesn’t remember the previous turn unless you feed the entire conversation history back in.

Similar to RAM, an LLM’s context window is finite…

There is a hard limit to the number of tokens that can be stored in context at a time. Even before hitting that limit, LLMs tend to bias their knowledge toward the start and end of their context window. This means an LLM’s understanding of the information it is given can degrade before it reaches the hard limit.

The size of an LLM’s context window varies between models.

We’re at a point where frontier models can process 1 million or more tokens, whereas many open models are closer to 256,000 tokens. This isn’t enough for many applications, so it’s important to design AI systems with context limitations in mind.

Also, like RAM, an LLM’s context doesn’t persist between sessions.

LLMs will only remember the context provided during a session. This means AI systems with long-term memory require a more persistent memory storage and retrieval system.

Context is managed by only keeping the tokens a model needs for a task.

To effectively manage context, consider:

• Using subagents for specific tasks. A separate agent means a separate context window. That agent can process and summarize the context, then return it to our primary agent.

• Summarizing all information currently within an LLM’s context window, clearing the context window, and adding the summary back in. This stores the same information in a shorter context.

• Using external systems to store information. Information is only needed in context when it’s being used. It should be stored in a separate, accessible datastore otherwise.

Every AI application built with LLMs needs to properly manage context.

4 Embeddings and Vector Databases

You can’t fit everything into a model’s context window, so you need a way to find relevant information efficiently.

An embedding represents the meaning of data as a list of numbers, called a vector. An embedding can be created from text, audio, video, or any other data format to convey the meaning of that data. Items with similar embedding vectors are considered similar.

Consider a coordinate system in which each axis corresponds to a learned semantic dimension.

“Deploy a microservice” and “ship a container to production” would be plotted close together. “Deploy a microservice” and “quarterly revenue forecast” would be far apart. Embeddings can do this across thousands of dimensions. Distance in this high-dimensional space equals similarity in meaning.

Vector databases5 are built explicitly to store and query embeddings.

They use algorithms optimized for quickly finding the most similar vectors in the database. Those vectors are then added to a model’s context. This is comparable to retrieving data from a cache.

The process follows these steps:

1. Index: Convert documents into embeddings, store in a vector database

2. Retrieve: Embed the query and find the most similar documents

3. Augment: Pass those documents into the LLM’s context window with the user’s question

4. Generate: LLM answers using retrieved information

Almost every production AI engineering system will need a working memory solution.

Context will need to be augmented with a persistent data store. Embeddings and vector databases.

5 Time to First Token

There are many factors to consider when choosing a specific LLM for your production AI system. One of these requirements is latency: your system must function fast enough to be useful.

AI systems must consider the key latency metrics of traditional software systems and the LLM latency metrics, such as:

• Time to First Token (TTFT): How long until the model starts responding. This determines perceived responsiveness. Smaller models respond in under a second, but larger models can take several seconds to generate anything.

• Tokens Per Second (Throughput): How fast the model generates after starting. This varies significantly across model sizes and compute units, and also affects how quickly the model's response is output to the user.

AI applications require concurrency to stream the output of an LLM call and reduce perceived latency.

Latency is an incredibly important metric to understand the real-world functionality of an AI system.

This is like streaming services, which play the start of a video while loading the rest in the background. This enables the viewer to watch the video before the entire video loads.

6 Evals

“Evals are a new toolset for any and all AI engineers, and software engineers should also know about them. Move from guesswork to a systematic engineering process for improving AI quality.” - Pragmatic Engineer, December 2025

In traditional software, tests are written to ensure code function reliability.

In AI, traditional tests are insufficient for many parts of the system because of their non-deterministic nature. Instead, we write evaluations (evals for short).

Since assert statements won’t suffice for non-deterministic applications, evals are written using LLM-as-a-judge6.

Essentially, a dataset of inputs and their successful outputs is curated.

This dataset will be used to tell the judge what we’re looking for in our model's output. The LLM-as-a-judge will score how closely the tested model’s outputs resemble the successful output based on given criteria such as relevance, hallucinations, and accuracy.

For example, an eval can be used to assess whether an LLM effectively pulls the important information from a document when summarizing. A set of documents and summarization pairs is gathered as a solution set. The documents are run through the LLM we’re testing, and our LLM-as-a-judge compares the solution for that document to our LLM’s output.

Evals are vital because even the smallest changes in a non-deterministic system can change the user's output. The most common of these changes is model tweaks. Even a small tweak in model performance can drastically change the output of a system.

Evals are written to catch these changes.

7 Agent Loops

Agent loops7 represent the shift from Software 2.0 to Software 3.0.

Software 1.0 to 2.0 was the change from logical control flow to probabilistic output. In the shift to Software 3.0, we see probabilistic output used as reasoning loops in an agentic system8.

Agent loops enable an LLM to continuously receive and act on information. An agent loop looks something like:

1. Thought: Model reasons about the current state and what it needs to do next

2. Action: The model decides to call a tool

3. Observation: Application executes the tool and returns the result

4. Repeat: Model then reasons about the observation, takes another action, or provides a final answer

This is the ReAct pattern (Reason + Act), and it has become the default agent loop architecture.

ReAct merges Chain of Thought reasoning, where the model works through a problem step by step, with tool execution into a single loop. This means a model can not only reason to act, but also take its previous actions and their outputs into account, continuously acting until a task is complete.

For example, consider an AI travel agent tasked with booking a trip within a budget.

It first searches for flights and finds the cheapest option. It then reasons about how much budget remains and searches for hotels accordingly. The initial results are all outside the city center, so it adjusts its search criteria and finds a better option. It calculates the total, confirms it fits the budget, and presents the full itinerary.

An agent loop relies on the LLM to determine the next step continuously until it completes a task. This looping process enables an LLM to use its knowledge and context to do meaningful work.

This enables complex workflows that aren’t possible with written code alone.

8 Tool Calling

Meaningful work is further enabled by tool calling9 capabilities.

This gives the LLM the ability to do something as part of its action. While LLMs can always output natural language, tool calling enables that output to call functions that perform work.

LLMs cannot execute code, access the internet, or perform other work natively. Instead, tool calling must be enabled:

Read more

• Share this post & I'll send you some rewards for the referrals.

I have a big big announcement for those of you who want to become good at AI engineering.

INTRODUCING: GENERATIVE AI MASTERCLASS

This is a monthly newsletter series that’ll teach you Generative AI system design.

By the end of this newsletter series, you’ll walk away with:

• Simple breakdown of real-world architectures

• Frameworks you can plug into your work or business

• Proven systems behind ChatGPT, Perplexity, and Copilot

And here’s the best part:

You’ll go from knowing how to “use AI” to understanding how to design, build, and ship AI.

No AI/ML background needed. Each newsletter explains concepts in plain English, with visuals and real product examples.

If you want the maximum AI career leverage, join below:

Subscribe now

New Open Source: Build and Scale Agents and Harnesses Like APIs (Partner)

One.harness() call and orchestrate hundreds of Claude Code, Codex, and Gemini instances into a coordinated team. They discover each other at runtime, split work, cross-reference results, and converge autonomously. Go deeper on harness engineering → blog.

No DAGs. No glue code. Agents work like APIs, Python, TypeScript & Go.

We built production systems you can clone and run today:

→ Autonomous Engineering Team: 175 agents ship code end-to-end

→ Deep Security Auditor: 250 agents scan every CVE

→ Adversarial Code Reviewer: 50+ agents debate your PRs

Each runs on any model. Apache 2.0.

Clone a Recipe

We’ve all used ChatGPT.

From our user perspective, it looks simple: type a question, get a response. But there’s a lot more behind it than a large language model (LLM) with a chat interface.

Building a product like ChatGPT means choosing the right model from a growing landscape of options, preparing the data that trains it, running a multi-stage training pipeline, evaluating whether the outputs are actually good, adding safety controls to prevent harm, and wrapping it all in serving infrastructure that can handle millions of requests. The same is true for Perplexity, GitHub Copilot, and Midjourney.

The model is one piece of a much larger system.

Understanding these components is what separates someone who can use GenAI from someone who can design and build it. That’s what this newsletter series is for.

This first newsletter covers the foundation.

We walk through the model landscape, the data layer, the training pipeline, evaluation, safety, and the engineering that turns a model into a product.

Let’s start with a glossary of key terms that come up throughout this newsletter and the rest of the series. If you’re already comfortable with terms like tokens, embeddings, and context windows, skip ahead to Part 1: The AI Model Landscape.

I want to introduce Louis-François Bouchard as the author of this newsletter.

He’s a best-selling author (Building LLMs for Production), the co-founder of Towards AI, and the creator of the YouTube Channel, What’s AI, where he helps people understand AI and learn how to apply it in the real world. Through his development works with clients and his content, teaching, and AI training programs on the Towards AI Academy, Louis focuses on making AI practical for builders, engineers, and curious learners alike.

At Towards AI, he and his team train AI engineers through courses built for every stage, from beginner to advanced. That educational mission and the real-world experience building for his clients are exactly why I wanted him in this newsletter series.

Key Terms

You don’t need to learn these upfront.

Skim to get familiar, then refer back whenever you hit an unfamiliar term:

Token: LLMs work with numbers, not words. Before processing any text, the model breaks it into small chunks, called tokens. A token can be a whole word (“the”), part of a word (“un”, “believ”, “able”), or punctuation. One token is roughly ≈ 4 English characters, though this varies by language and model. API pricing, context limits, and cost are all measured in tokens.

Embedding: A token ID alone (like 1023 for “What”) tells the model nothing about meaning. An embedding replaces that ID with a list of numbers (a vector) that captures what the word means and how it relates to other words. These vectors are what the model actually works with internally, processing and transforming them at every step to build up an understanding of the full input. Words with similar meaning end up with similar vectors: “king” and “queen” are close together, “king” and “banana” are far apart. This same property also powers search in GenAI products: when a user asks “How do I reset my password?”, the system can find documents on password resets, account recovery, and login issues, even if those documents never use that exact phrase.

Parameter: These are internal variables inside a model, billions of them. At the start of training, they’re random. Each time the model makes an incorrect prediction, settings are nudged slightly. After trillions of such adjustments, they encode language patterns. GPT-3 had 175B parameters. Llama 4 comes in Scout (17B active) and Maverick (17B active). More parameters mean more capability, but also more memory: a large model can need 140GB or more just to load.

Context Window: The maximum amount of text (in tokens) a model can see in one request. GPT-5 supports up to 400K tokens (272K input). Claude and Gemini support up to 1M tokens. Everything the model needs, system instructions, conversation history, retrieved documents, and the user’s question must fit within this window. Deciding what goes in and what gets cut is called context engineering.

Inference: Every time someone sends a message to ChatGPT, the model runs to produce a response. That’s one inference call. At scale, this can mean millions of calls per day, and the cost adds up. Two key metrics define inference performance: Time to First Token (TTFT), the time the user waits before the first word appears, and Tokens per Second (TPS), the rate at which the rest of the response streams in after that.

LLM (Large Language Model): A neural network (a program that learns patterns from data) trained on one objective: to predict the next token. GPT, Claude, Llama, and Gemini are all LLMs. The objective is simple, but at a massive scale (trillions of tokens of training data), it produces the ability to follow instructions, write code, reason through problems, and hold conversations.

Foundation Model: A large, general-purpose model trained on massive data that serves as the starting point for products. Both LLMs (GPT-5, Claude) and diffusion models (Stable Diffusion, Midjourney) are foundation models. These aren’t built from scratch for each product. You take one and adapt it through prompting, fine-tuning, or both.

RAG (Retrieval-Augmented Generation): A design approach where the system retrieves relevant documents at query time and includes them in the prompt, so the answer is grounded in real sources rather than what the model memorized during training. Products like Perplexity use this approach: search the web, feed the results to the model, and generate an answer with citations. It’s widely used in products that need up-to-date or domain-specific knowledge.

Fine-tuning: Taking a foundation model and training it further on a smaller, task-specific dataset to specialize its behavior. For example, a model fine-tuned on customer support conversations learns the right tone, format, and escalation patterns for that domain. Fine-tuning changes how the model responds (style, format, tone), not what it knows. If the model needs to answer questions using knowledge not in its training data, retrieval-augmented generation (RAG) is the standard approach.

Prompt: The full text input sent to a model for each request is a prompt. It typically includes a system prompt (developer-defined instructions that define the model’s role), the conversation history (previous messages), and the user message (the current question). In RAG systems, retrieved documents are also included. Prompt engineering is the practice of refining how this input is structured to get better results.

Hallucination: When a model generates confident but factually wrong output, like citing a paper that doesn’t exist or inventing a URL. This isn’t a bug. These models are optimized to produce text that sounds plausible rather than factually correct, so hallucinations are a built-in trade-off, not a malfunction. The main ways to reduce hallucinations are RAG (retrieving real sources to ground the answer) and evaluation (testing the model’s outputs for accuracy).

Transformer: The neural network architecture behind nearly all modern foundation models, introduced in 2017 (“Attention Is All You Need”). Its key innovation is the attention mechanism: instead of processing tokens sequentially, the model considers all tokens simultaneously and learns which are relevant to one another. In “The cat sat on the mat because it was tired,” attention connects “it” back to “cat” rather than “mat.” Some models extend this architecture with a Mixture of Experts (MoE), where only a subset of the model’s parameters is used for each input rather than all of them, making the model faster and cheaper to run while maintaining high overall capability.

GPU/TPU: The specialized hardware on which GenAI runs. GPUs (NVIDIA H100, A100, B200) handle the massive number of calculations required by models. A request that takes milliseconds on a GPU could take orders of magnitude longer on a regular CPU. Training a large model needs thousands of GPUs running for months. TPUs are Google’s custom chips for ML, used primarily within Google Cloud.

Latency: How long the user waits for a response. Different products have different targets depending on what the user is doing. Autocomplete has to keep up with typing, so it typically needs a response in under 300ms. Chat users expect a short pause before the reply starts streaming. Image and video generation is much slower; users wait 10 to 60 seconds. These targets influence which model to use, whether to add caching, and how much the system can do before it starts to feel slow.

The first decision in designing any GenAI system is choosing the right model…

That starts with understanding what kinds of models exist and what each one is good at.

Part 1: The AI Model Landscape

Not all generative models work the same way.

They differ in how they generate output, what kinds of output they produce, and how they’re built. This section breaks down the two core architectures, types of outputs they support, the internal structure of models, and how to choose between them.

Two Families of Generative Models

All generative AI models are built on one of two architectures:

1 Autoregressive models generate text one token at a time.

Given the input “The capital of France is”, the model predicts the most likely next token: “Paris” in this case. It appends “Paris” to the sequence, and now uses “The capital of France is Paris” to predict the next token, i.e., “.” Then it uses the full sequence to predict the token, and so on. Each output feeds back as input. This is how GPT, Claude, Gemini, and Llama models work.

When you see a chatbot “typing” its response word by word, that’s not a UI animation; the model is actually producing tokens one after another.

Because generation is sequential, latency scales with output length: a 500-token response takes roughly 500 generation steps. Cost scales, too: a 1,000-token response costs roughly 10x as much as a 100-token response.

Autoregressive generation: each token is produced one at a time, left to right.

2 Diffusion models work completely differently.

Instead of producing one piece at a time, they start with pure random noise and gradually remove it over many steps until a coherent image (or video) emerges.

During training, the model sees millions of images with noise added at various levels and learns to predict what the clean version looks like. At generation time, it reverses the process step by step, typically 20 to 50 steps, removing a little noise each time until a clear image forms.

This is how Midjourney, Stable Diffusion, DALL-E, and Sora work.

The serving model is very different from text: image generation can take roughly 10 to 60 seconds, and video can take minutes, so products typically run generation as batch jobs with a queue rather than streaming.

A single image generation also costs roughly 10-100x as much compute as a typical ChatGPT response.

Models Outputs

These two architectures cover a range of output types:

1 Text generation (autoregressive): chat responses, code, summaries, translations, structured data extraction. This powers ChatGPT, Claude, GitHub Copilot, Perplexity.

2 Text-to-image (diffusion): generate images from text prompts. Midjourney, DALL-E 3, Stable Diffusion.

3 Text-to-video (diffusion): generate video clips from text or image prompts. Sora, Runway Gen-3, Kling.

4 Text-to-audio (autoregressive + diffusion): generate speech, music, and sound effects. ElevenLabs, Suno, Bark.

5 Vision-language models (autoregressive) process images alongside text. Upload a photo and ask the model to describe it, extract data from a receipt, or analyze a chart. GPT-5, Claude, and Gemini all support this.

Multimodality

Each of the output types above used to require its own dedicated model.

That’s changing; models like GPT-5 and Gemini now handle text, images, audio, and video within a single architecture. The question is how. How does a text prompt like “a photo of a cat” actually guide an image model to generate the right image?

Through cross-modal embeddings.

An embedding converts content into a list of numbers that captures its meaning. Cross-modal embeddings do this across different content types, text and images, placing them on the same map.

CLIP, developed by OpenAI, is the most well-known example. It places both “a photo of a cat” (text) and an actual cat photo (image) side by side on this shared map. That shared representation enables a text description to guide image generation.

Model Architecture Variants

Not all foundation models are built the same way…

Most are based on the Transformer (covered in the glossary above), but they use it differently. The two main jobs of a language model are understanding input and generating output.

Different architectures handle these jobs in different ways:

1 Decoder-only models focus entirely on generation: they take an input and produce text left-to-right, one token at a time. GPT, Claude, Llama, and Gemini are all decoder-only. This is the dominant architecture for text generation today.

2 Encoder-only models (like BERT, an early Transformer‑based language model) focus entirely on understanding: they read input and produce a numerical representation of its meaning, but don’t generate new text. Used for classification, sentiment analysis, and search ranking. For example, when a support system reads “I’m furious about my broken order” and classifies it as “negative sentiment, high urgency”, that’s an encoder-only model at work. In GenAI products, they play supporting roles, such as classifying user intent or detecting toxicity.

3 Encoder-decoder models (like Google’s T5 and Meta’s BART) do both: one part reads and understands the input, the other generates the output. Suited for translation, summarization, and rewriting.

4 Mixture of Experts (MoE) is a different kind of optimization.

Instead of a single large model where every parameter is used for every input, MoE splits the model into multiple specialized groups, called “experts.” For each input, a router determines which experts are relevant and activates only those. A question about Python code might activate code-specialized experts, while a question about French history might activate different experts. Models like Mistral’s Mixtral 8x7B use this approach, with 46.7 billion total parameters but only around 12.9 billion active per token. You get more capability per unit of compute, but you need more GPU memory to store all the experts.

The Model Landscape: Proprietary, Open-Weight, and Open-Source

When choosing a model, there are three categories with different trade-offs:

1 Proprietary models (GPT, Claude, Gemini) are accessed via API. They offer the highest capability and are the easiest to start with, but give users the least control. You can’t see the model’s internal parameters, modify its architecture, or run it on your own servers.

2 Open-weight models (Llama, Mistral, Gemma) release their trained parameters publicly. You can download them, run them on your own GPUs, and fine-tune them. Training data and methods are typically not shared. This gives you full control over the model, but it also means you have to manage the infrastructure yourself.

3 Open-source models go further by sharing not just weights but training code, data, and the full methodology behind how the model was built.

The gap between proprietary and open-source is narrowing…

By early 2025, models like Llama 4 Maverick and DeepSeek-V3 were competitive with proprietary models on many tasks. Most production systems use a mix: a proprietary model for complex tasks and an open-weight model for high-volume, simple ones.

Model Selection

Each model family offers a range of sizes (e.g., GPT-5 vs. GPT-5 mini, Claude Opus vs. Haiku, Llama 4 Maverick vs Scout).

Smaller models are faster and cheaper, but less capable. Beyond size, some models offer specialized variants. OpenAI’s GPT-5, for example, includes configurable reasoning effort, and Claude offers an extended thinking mode. These reasoning capabilities require more compute during inference to work through problems step by step, breaking complex questions into smaller parts before answering. This improves accuracy on tasks such as math, logic, and multi-step analysis, but it costs more per request and takes longer to respond.

This trade-off is another reason model routing matters: route simple questions to a faster model, and complex reasoning tasks to a model with these capabilities.

Public benchmarks help narrow down which specific model is best suited to a task. MMLU tests general knowledge across 57 subjects; HumanEval and MBPP focus on code generation; Chatbot Arena by LMSYS aggregates human preference scores from real conversations; and benchmarks like GPQA and ARC-AGI evaluate reasoning and problem-solving abilities.

But benchmarks have a well-known problem: contamination.

Models can be trained on benchmark test data, inflating scores. It’s like a student who memorized the answers to last year’s exam; they score high, but can’t solve newer problems. A model that tops MMLU may underperform on a specific task. Benchmarks are useful for narrowing the field, but final selection requires evaluating on your own data.

That covers the model landscape, what models exist, how they differ, and how to pick between them. All of these models, though, start with random parameters. What turns them into something useful is data.

Reminder: this is a teaser of the subscriber-only newsletter series, exclusive to my golden members.

When you upgrade, you’ll get:

• Simple breakdown of real-world architectures

• Frameworks you can plug into your work or business

• Proven systems behind ChatGPT, Perplexity, and Copilot

Unlock Full Access

Part 2: The Data Layer

A model’s capability comes entirely from the data it was trained on.

The quality of that data, how it’s collected, cleaned, and organized directly determines how well the model performs.

Training Data

Foundation models learn from massive datasets: trillions of tokens scraped from books, websites, code repositories, scientific papers, and forum posts. For diffusion models, training data consists of millions of images, each paired with a text description of its contents.

Raw data is never used directly.

It passes through a data pipeline that cleans, filters, deduplicates, and transforms the data before training begins. For example, the same news article might appear on hundreds of websites. Without deduplication, the model sees the same phrases hundreds of times and overlearns them.

The pipeline handles problems like near-duplicate web pages that inflate certain patterns, toxic or biased content the model would absorb, personally identifiable information (PII) that shouldn’t be memorized, and low-quality text that adds noise without a useful signal.

The quality of this pipeline directly determines the model's quality.

Synthetic data generation is increasingly used to fill data gaps. An existing model generates additional training examples for areas where real data is scarce, such as specialized medical terminology, underrepresented languages, or edge-case code patterns. The quality of synthetic data depends on the model that generates it and how it’s validated.

Bias detection starts here. If the training data overrepresents certain perspectives, demographics, or viewpoints, the model will reflect those imbalances. Data teams audit for representation gaps and offensive content before training begins, but detection is imperfect. Some biases only surface after the model is trained and tested.

NSFW and harmful content requires explicit handling during this stage. Some pipelines filter it entirely. Others retain it in a controlled form so the model can later learn to recognize and refuse it. The approach depends on the product’s intended use.

Data Storage and Retrieval

Training and inference have very different data needs.

During training, the dataset can be terabytes or more in size, containing text, images, and code. Teams typically store this in distributed file systems like HDFS (Hadoop Distributed File System), which splits large files across a cluster of machines so they can be read in parallel, or cloud object storage like S3 (Amazon’s Simple Storage Service), which stores files as objects in the cloud and is widely used because it’s cheap, scales without limits, and requires no infrastructure to manage. Training pipelines stream data from these systems to GPU clusters in batches, feeding the model continuously throughout training.

At inference time, the training data is no longer needed.

What matters is the model itself. The model’s parameters need to be loaded into GPU memory before it can serve any requests, and for large models, that means hundreds of gigabytes. Once loaded, the model takes in a prompt and generates tokens one at a time, streaming them back to the user.

The challenge is keeping sufficient GPU capacity available to handle many users simultaneously without long wait times.

Some GenAI systems also retrieve data at runtime, pulling in documents, search results, or database records when a user asks a question. This is how products like Perplexity stay current beyond the model’s training data.

In Part 6, we cover how this retrieval works, embeddings, vector databases, and structured data handling when we design the full product architecture.

But having clean, well-organized data is only the starting point.

The next question is what happens to that data, and how a pile of text becomes a model that can actually follow instructions…

Let’s keep going!

Read more

• Share this post & I'll send you some rewards for the referrals.

• Block diagrams created using Eraser.

Object storage isn’t just “upload a file, get a URL back.”

That’s true for a small side project. It stops being true when you’re storing 100 trillion objects like AWS S3 does today.

Amazon Simple Storage Service (S3) launched in 2006.

In 2013, Amazon reported storing 2 trillion objects. By 2021, that number crossed 100 trillion. The system handling all of that doesn’t look anything like a file server or a relational database. It’s a fundamentally different class of infrastructure, built around a different set of tradeoffs.

This problem shows up in senior- and staff-level interviews at companies building storage-heavy products…

It tests the exact skills that separate mid-level engineers from seniors: understanding why different storage types exist, designing for durability at a scale where hardware failures occur daily, and making smart trade-offs between consistency, cost, and performance.

We’re going to build this from scratch:

We’ll start with what object storage actually is and how it differs from other storage systems, then work through requirements, capacity estimation, the high-level architecture, disk-based data persistence, durability strategies, metadata design, object versioning1, large-file uploads, and garbage collection.

At each step, we’ll explain the why behind each decision…

Find out why 150K+ engineers read The Code twice a week (Partner)

Tech moves fast, but you’re still playing catch-up?

That’s exactly why 150K+ engineers working at Google, Meta, and Apple read The Code twice a week.

Here’s what you get:

• Curated tech news that shapes your career - Filtered from thousands of sources so you know what’s coming 6 months early.

• Practical resources you can use immediately - Real tutorials and tools that solve actual engineering problems.

• Research papers and insights decoded - We break down complex tech so you understand what matters.

All delivered twice a week in just 2 short emails.

Sign up and get access to the Ultimate Claude code guide to ship 5X faster.

Join 150K+ Engineers

(Thanks for partnering on this post and sharing the ultimate claude code guide.)

I want to reintroduce Hayk Simonyan as a guest author.

He’s a senior software engineer specializing in helping developers break through their career plateaus and secure senior roles.

If you want to master the essential system design skills and land senior developer roles, I highly recommend checking out Hayk’s YouTube channel.

His approach focuses on what top employers actually care about: system design expertise, advanced project experience, and elite-level interview performance.

What is Object Storage?

Before we design anything, we need to understand what object storage actually is…

Engineers often confuse the three main storage categories. Each one exists for a reason, and choosing the wrong one at design time is expensive to fix later.

Block Storage

Block storage is the oldest and lowest-level.

When you plug a hard drive or SSD into a server, the operating system (OS) sees it as a sequence of raw blocks, each typically 4KB in size. The OS decides how to format those blocks and build a file system on top of them. Some applications, such as databases and virtual machine engines, skip the file system entirely and manage blocks directly, which gives them maximum control and performance.

Block storage doesn’t have to be physically attached…

You can connect to block storage over a network using protocols like Fibre Channel or iSCSI2. The server still sees raw blocks, as if the drive were directly attached, but the data resides elsewhere on the network. This is how cloud providers like AWS offer Elastic Block Store (EBS): you attach a network drive to your VM, and it behaves like a local disk.

Block storage is fast and flexible, but it’s also expensive and doesn’t scale cheaply to petabytes.

File Storage

File storage is built on top of block storage.

It adds a layer that handles the complexity of managing blocks and gives you a familiar directory hierarchy: folders, subfolders, and files. You don’t deal with blocks at all. You just read and write files using paths like /documents/report.pdf.

File storage becomes especially useful when many servers need to share the same files.

Protocols like NFS and SMB/CIFS3 allow many machines to mount the same file share and read and write to it concurrently. This is how shared drives inside organizations typically work, and it’s how many legacy enterprise applications store data.

File storage is easier to use than block storage, but it still doesn’t scale to the level object storage does. Hierarchical directory structures become slow and complex when you have billions of files.

Object Storage

Object storage is the newest of the three and the most different…

It makes a deliberate tradeoff: give up performance and mutability in exchange for near-unlimited scalability, very high durability, and low cost.

There are no directories in object storage.

Everything lives in a flat namespace inside containers called buckets. Every object is accessed via a RESTful HTTP API using a unique key. You can’t partially update an object. If you want to change a file, you replace the entire object or create a new version. This immutability4 constraint sounds limiting, but it’s actually what makes object storage so cheap to operate at scale, because it simplifies replication and consistency considerably.

AWS S3, Google Cloud Storage, and Azure Blob Storage are all object storage systems. They’re the foundation of most modern cloud architectures: video files, backups, data lake storage, machine learning datasets, static website assets, and more.

Comparison Table

The key constraint to internalize: objects are immutable.

You cannot edit part of an object. You replace the whole thing, or you version it. This constraint is not an accident. It’s a deliberate design choice that enables the durability and scale properties that make object storage useful in the first place.

Key Terms

These are the concepts you need to know before we get into the design:

Bucket

A bucket is a logical container for objects.

Think of it like a top-level folder, except it’s not really a folder since there’s no hierarchy inside it. Bucket names must be globally unique across all system users, not just within your account.

You have to create a bucket before you can store anything in it.

Object

An object is an individual piece of data stored in a bucket.

It has two parts:

• Payload is the actual data bytes, which can be anything: a photo, a video, a CSV file, a binary blob.

• Metadata is a set of key-value pairs that describe an object, such as content type, creation timestamp, custom tags, and anything else the application needs to store alongside the data.

The metadata is stored separately from the payload and is much smaller.

Object Key

Every object is identified by a key, which is just a string.

In S3, that key looks like a file path: photos/2024/vacation.jpg. But there are no actual directories. The entire string, slashes and all, is just the key. S3 lets you use slashes as a convention to simulate folders, but under the hood, it’s still a flat namespace.

This distinction matters when we design the listing feature later.

Versioning

Versioning is a bucket-level feature that keeps all previous versions of an object instead of overwriting them.

When versioning is enabled, uploading an object with the same key as an existing object doesn’t replace it. Instead, it creates a new version alongside the old one. You can retrieve, restore, or delete any version at any time.

This protects against accidental overwrites and deletions.

Durability SLA

S3 Standard storage class is designed for 99.999999999% durability, also known as eleven nines.

In practical terms, if you store 10 million objects for 10,000 years, you’d expect to lose one. That’s not an accident. It comes from specific engineering decisions around replication and error correction, which we’ll get into in the durability section.

Clarifying Requirements

Before touching any design, you need to understand what you’re actually building and at what scale. Candidates who skip this step in interviews fail immediately, because they end up designing the wrong thing at the wrong scale…

Questions to Ask

• What are the core operations? Upload, download, delete, list?

• Do we need versioning?

• How much data do we need to store in year one?

• What durability and availability targets do we need?

• Do we need to support large file uploads, such as files over multiple gigabytes?

• Any access control requirements? Do different users own different buckets?

Our Assumptions

For this design, we’ll assume:

• Core operations: bucket creation, object upload and download, object versioning, and listing objects in a bucket by prefix

• 100 petabytes of total data

• Durability target: six nines, which is 99.9999%

• Availability target: four nines, which is 99.99%

• Must handle both small objects (tens of kilobytes) and large objects (several gigabytes)

Now, let’s figure out what these requirements actually mean for infrastructure…

Capacity Estimation

Math matters in system design because it tells you what kind of infrastructure you need. Vague statements like “we’ll need a distributed database” aren’t useful without concrete numbers.

Let’s work through the estimates:

Object Size Distribution

In practice, object storage systems see a mix of object sizes. A reasonable assumption for a general-purpose system:

• 20% of objects are small, under 1MB, with a median size of about 0.5MB. These might be thumbnails, config files, or short documents.

• 60% of objects are medium-sized, ranging from 1MB to 64MB, with a median of about 32MB. These might be images, audio files, or compressed datasets.

• 20% of objects are large (>64MB), with a median of about 200MB. These might be videos, database backups, or large archives.

Total Object Count

We’re targeting 100 petabytes of stored data.

In practice, storage systems don’t fill to capacity, so let’s assume 40% utilization, meaning we provision enough storage to hold 100PB when 40% full.

100 PB = 10^11 MB

Weighted average object size: (0.2 x 0.5MB) + (0.6 x 32MB) + (0.2 x 200MB)

= 0.1 + 19.2 + 40.0

= 59.3 MB per object (average)

Total objects at 40% utilization: (10^11 x 0.4) / 59.3

= approximately 680 million objects

680 million objects are a lot.

This tells us immediately that no single machine can store or index all of this. We need distributed storage and a distributed metadata index from day one.

Metadata Storage

Each object needs a metadata record. If we assume roughly 1KB per record (object name, bucket, timestamps, UUID, tags), then:

680 million objects x 1KB = ~680GB of metadata

680GB of metadata is manageable in a database, but the access patterns will require sharding as we scale. The metadata store is separate from the data store, which we’ll cover in the design.

IOPS Constraints

A standard SATA hard drive5 spinning at 7200 RPM can handle roughly 100-150 random seeks per second.

This is called IOPS6, or input/output operations per second. At 680 million objects spread across many disks, the IOPS constraint becomes a real bottleneck, especially for small object workloads where you’re doing many small reads and writes rather than a few large sequential ones.

This is one reason we’ll later choose to merge many small objects into a single larger file on disk, instead of storing each object as its own file.

Design Philosophy: Separating Metadata from Data

Before we look at the full architecture, there’s one core design principle that shapes everything else: metadata and data are stored separately, and for good reason.

This idea comes from how UNIX file systems work:

In UNIX, when you save a file, the filename and the actual data bytes are not stored together. The filename and other file information (size, permissions, timestamps, disk location) live in a data structure called an inode7. The actual data bytes live in separate disk blocks that the inode points to.

Object storage works the same way.

The metadata store is like the inode layer. It holds the object name, bucket, size, and a UUID that points to the location of the actual bytes. The data store is like the disk. It holds the raw bytes, and it only knows about UUIDs, not names or paths.

So why separate them?

Because they have completely different characteristics. The data is immutable. Once written, it never changes. The metadata is mutable. You can update tags, rename objects (in some implementations), or add versioning records. They need different consistency guarantees, different storage engines, and different scaling strategies.

Keeping them separate lets us optimize each independently.

High-Level Architecture

With that principle in mind, here’s how the full system is structured:

Components

Upload Flow

Let’s trace exactly what happens when a user uploads a file named report.pdf to a bucket called company-docs:

1. Client sends a HTTP PUT /company-docs/report.pdf request with the file bytes in the request body.

2. Request hits the load balancer and gets routed to one of the API service instances.

3. API service calls the IAM service8 to confirm the user has WRITE permission on the company-docs bucket. If not, the request is rejected immediately with a 403 Forbidden.

4. API service forwards the file bytes to the data store. The data store persists the bytes and returns a UUID, a unique identifier for this specific object.

5. API service then writes a metadata record to the metadata store. This record contains the object name (report.pdf), bucket ID, UUID returned from the data store, file size, creation timestamp, and any metadata tags the user provided.

6. A 200 OK response is returned to the client.

The metadata record now serves as the bridge between the human-readable path (company-docs/report.pdf) and the actual bytes stored under the UUID in the data store.

Download Flow

Now, let’s trace what happens when someone requests the same file:

1. Client sends a GET /company-docs/report.pdf request.

2. API service calls IAM to verify READ permission.

3. API service queries the metadata store: “What is the UUID for the object named report.pdf in the bucket company-docs?”

4. API service then uses that UUID to fetch the actual bytes from the data store.

5. The bytes get returned to the client.

Notice that the data store never knows the file was called report.pdf.

From its perspective, someone asked for the object with a specific UUID, and it returned the bytes. The translation from name to UUID always happens in the metadata store.

Reminder: this is a teaser of the subscriber-only newsletter series, exclusive to my golden members.

When you upgrade, you’ll get:

• High-level architecture of real-world systems.

• Deep dive into how popular real-world systems actually work.

• How real-world systems handle scale, reliability, and performance.

Subscribe now

Deep Dive: Data Store

The data store is where most of the interesting engineering happens…

Let’s break down its internal architecture:

Internal Components

Data Routing Service

The data routing service is the entry point into the data store…

It’s stateless, meaning it holds no state itself, so you can scale it horizontally by adding more instances. When a write comes in, the placement service determines which data node should receive the data, then sends the data there.

When a read comes in, it asks the placement service where the data lives, then fetches it.

Placement Service

The placement service9 is responsible for knowing the physical layout of the entire storage cluster…

It maintains a virtual cluster map10, which is essentially a registry of every data node, including the rack and availability zone it’s in, how many disks it has, and how much space is used on each disk.

Placement service continuously receives heartbeat messages from every data node.

A heartbeat is a small message a node sends every few seconds, saying, “I’m alive, here’s my current state.” If the placement service doesn’t hear from a node within a configurable grace period (typically 15 seconds), it marks that node as down and stops sending new data to it.

Because the placement service is so critical, you run it as a cluster of 5 or 7 nodes using a consensus algorithm11 like Raft or Paxos.

A consensus algorithm ensures the cluster agrees on a single consistent view of the world, even if some nodes fail. With a 7-node cluster, you can lose 3 nodes simultaneously, and the service keeps running. With a 5-node cluster, you can lose 2. You never run this as a single instance, because if it goes down, the entire storage cluster becomes unavailable for writes.

Data Nodes

Data nodes are where the actual bytes live…

Each data node manages one or more physical disks. Each node runs a daemon process that sends heartbeats to the placement service with information about disk count and available space. When the placement service receives a heartbeat from a new node it hasn’t seen before, it assigns that node an ID, adds it to the virtual cluster map, and tells the node where to replicate data.

Durability is the central promise of object storage and the hardest engineering problem in this design.

We'll get there.

But before we can talk about how data survives hardware failures and entire node outages, we need to understand exactly how a write moves through this system.

The decisions made in the next few steps determine whether durability can be guaranteed…

Read more

• Share this post & I'll send you some rewards for the referrals.

Since people are always asking me for more “practical” system design content:

Following is the free edition of a premium 3-part newsletter series... This covers how mobile apps work under the hood. And it’s explained in a simple, realistic, and useful way.

On with part 1 of the newsletter:

Trying to debug a request that touched 5 services? (Partner)

Distributed systems help you move faster…until you have to debug them.

This blog shows how to use Sentry tracing and logging to follow a request end-to-end. You don’t need prior microservices experience to follow these steps.

Read the blog and start fixing

(Thanks to Sentry for partnering on this post.)

I want to reintroduce Shefali Jangid as a guest author.

She’s a web developer, technical writer, and content creator with a love for frontend architecture and building things that scale.

Check out her work and socials:

• Shefali.dev

• GitHub

• Twitter

You’ll often find her writing about web development, sharing UI tips, and building tools that make developers’ lives easier.

Building “scalable mobile apps” is not just about writing frontend code and calling APIs…

It’s actually one of the toughest parts of software engineering!

Mobile apps don’t run in ideal environments. The network can go down, batteries run low, the operating system can close your app in the background, devices vary, and users still expect everything to work instantly, even offline.

When building mobile apps, you always have to think about limits. Every decision affects speed, stability, data safety, security, and user experience simultaneously. And making the right choices turns a simple app into a strong, production-ready package.

In this newsletter, we’ll focus on 19 concepts as the foundation:

1. Client-Server Architecture,

2. WebSockets & Persistent Connections,

3. Push Notifications,

4. Polling, Long Polling & SSE,

5. REST vs GraphQL vs gRPC,

6. Network Resilience,

7. Idempotency in APIs,

8. Request Batching & Payload Optimisation,

9. Resumable Uploads,

10. Handling Intermittent Connectivity,

11. On-Device Caching,

12. HTTP Caching (ETag, Cache-Control),

13. Offline-First Architecture,

14. CDN Strategy & Media Optimisation,

15. Cache Invalidation Strategies,

16. Local Database Design,

17. Schema Migration Strategy,

18. Pagination (Cursor vs Offset vs Page Number),

19. Data Modeling for Mobile Constraints.

(…and much more in parts 2 & 3!)

For each concept, I’ll cover:

• What it is and how it works

• Real-world example

• The tradeoffs

• Why it matters for mobile

But before we get into the concepts, let’s first understand what mobile system design really means…

What is Mobile System Design?

Mobile system design is about planning how a mobile app works in the background.

It includes how the app connects to servers, saves and reads data, stays fast, keeps user data safe, and handles errors.

It’s not just about designing screens. It’s about understanding how data flows within the app, what happens when the network goes down, how the app runs in the background, and how it works across different devices.

Now that we understand what mobile system design means, let’s start with one of the most important parts: how your app connects and communicates with the outside world.

Networking & Real-Time Communication

This section explains how a mobile app connects to servers and handles real-time data…

You’ll learn how apps send and receive data, choose the right APIs, and handle slow or unstable networks so the app stays fast and reliable.

1. Client-Server Architecture (Thick vs Thin Clients)

When building a mobile app, an important question to ask is: how much work should the phone handle, and how much should the server do?

There are two main approaches:

a. Thick Client

A thick client means the app does most of the work on the device.

It can:

• Save data on the phone

• Handle some logic by itself

• Sync data later in the background

• Work without network connectivity

Most modern apps use this approach because:

• The network is not always stable

• Phones are powerful now

• Users expect apps to work offline

b. Thin Client

A thin client means the app does very little on the device.

It:

• Sends requests to the server

• Waits for the server to process everything

• Shows whatever the server returns

If the network fails, the app mostly stops working.

Why It Matters

This decision affects your entire app design:

• Whether your app can work offline

• How difficult syncing will be

• How easily you can update features

• How much important logic stays on the server

Real-World Example

• Gmail is a “thick” client. You can read emails and write drafts even without network connectivity. It syncs later.

• A simple app that only loads a website is a thin client. It’s useless without the network.

Trade-offs

• Thick client: Faster and works offline, but syncing and updates can be harder.

• Thin client: Easy to update and simpler to build, but it doesn’t work well with a bad network.

Practical Advice

For most apps:

• Keep the main app logic thick (use local storage and background sync).

• Keep sensitive logic on the server (like payments, login, pricing, fraud detection).

This gives you the best balance between speed, reliability, and security.

Once you decide what the app and the server should handle, the next question to ask is: How they should communicate, especially when you need real-time updates?

2. WebSockets & Persistent Connections

Usually, mobile apps use HTTP requests to talk to a server.

That means:

1. App asks for data

2. Server sends a response

3. Connection closes

If the app needs new data, it has to ask again.

WebSocket works differently…

It creates a long-lasting connection between the app and the server.

Once connected:

• App can send data anytime

• Server can also send data anytime

• Connection stays open

This is called real-time communication.

Why It Matters

With normal HTTP, the app must keep asking:

• “Any new messages?”

• “Any updates?”

With WebSockets, the server sends updates immediately when something happens.

This is very important for:

• Chat apps

• Live sports scores

• Stock price updates

• Multiplayer games

• Collaborative tools

If updates are slow, users feel like the app is broken.

Real-World Example

Apps like Slack, WhatsApp, and Figma use WebSockets.

When someone sends a message, you see it instantly because the server pushes it directly to your app.

Important Things to Remember (For Mobile)

On mobile devices, keeping a connection open all the time can drain the battery, use more data, and cause the operating system to close it.

Instead:

• Send small ‘heartbeat’ signals to keep it alive

• Reconnect if the network drops

• Close the connection when the app goes to the background

Otherwise, the connection may appear active but stop working.

How It Works

1. App starts with a normal HTTP request.

2. Server upgrades it to a WebSocket connection.

3. After that, both sides can send messages at any time.

4. Connection stays open until one side closes it.

Trade-offs

Pros

• Very fast (low latency)

• Instant updates

• Two-way communication

Cons

• Harder to manage

• Needs reconnect logic

• Can affect the battery if not handled properly

In simple words, use WebSockets when your app needs instant updates. Just manage the connection carefully on mobile devices.

But not all updates require a constant open connection.

What if the app is completely closed?

That’s when push notifications become useful…

3. Push Notifications (APNs & FCM)

Push notifications let your app send updates, such as “Your order has been picked up”, “You got a new message”, “Your payment was successful”, to users even when the app is closed.

But here’s the important part: Your app does not stay connected to your server all the time.

Instead, special services handle this for you:

• APNs (Apple Push Notification Service) for iPhones

• FCM (Firebase Cloud Messaging) for Android

These services act like messengers between your server and the user’s phone.

How It Works

1. Something happens on the server (e.g., a new message).

2. Server sends the notification to APNs (iOS) or FCM (Android).

3. APNs/FCM deliver it to the user’s device.

4. The mobile phone’s operating system shows the notification.

5. If needed, the app wakes up in the background.

Your app does NOT keep a permanent connection.

The phone’s operating system manages a “single” shared connection for all apps to conserve battery.

Why It Matters

Without push notifications, your app would need to stay connected all the time and keep checking for new updates. Neither iOS nor Android allows this because it drains battery.

Push services solve this problem in a battery-efficient way.

Real-World Example

Imagine a food delivery app. When your order is picked up:

1. The backend sends a push notification using FCM or APNs.

2. Your phone receives it instantly.

3. Even if the app is fully closed, you still get the update.

Trade-offs

Pros

• Saves battery

• Works even if the app is closed

• Delivers updates quickly

• No need for a constant background connection

Cons

• Delivery is not 100% guaranteed

• Can fail on a poor network

• Limited message size

• Should not be used for very critical data

You should never assume the user received the notification. Always refresh or sync data when the app opens.

Pro Tip

You can send a silent push notification (background push).

This shows nothing to the user. It simply wakes the app in the background to fetch fresh data. So when the user opens the app, everything updates without a loading spinner.

Push notifications are useful for sending updates to users even when the app is closed. But in many cases, the app itself needs to fetch updates from the server while it’s running.

In such situations, use techniques such as polling, long polling, or Server-Sent Events.

4. Polling, Long Polling & SSE

Sometimes apps need server updates, but using WebSockets isn’t always necessary or practical. In those cases, use simpler methods such as polling, Long Polling, or Server-Sent Events (SSE) to receive updates from the server.

Each method works a little differently.

Why It Matters

Not every feature needs instant real-time updates:

• Metrics dashboard might refresh every 30 seconds.

• Notification count might update every few seconds.

Using WebSockets for these cases can be unnecessary and complex. Choosing the right approach can save server resources and keep the system simple.

a. Polling (Short Polling)

Polling is the simplest method.

The app requests updates from the server at fixed intervals:

1. Every 10 seconds, the app sends a request.

2. The server responds with new data (if any).

So the process looks like this:

Problem:

Most of the time, there's no new data, yet the app keeps asking the server. This wastes network and server resources.

b. Long Polling

Long polling improves on basic polling.

Instead of responding immediately, the server keeps the request open until new data is available.

How it works:

1. App sends a request.

2. Server waits until something changes.

3. When new data arrives, the server sends the response.

4. The app immediately sends another request.

This reduces unnecessary requests compared to short polling.

Yet it still requires managing many open connections on the server.

c. Server-Sent Events (SSE)

SSE allows the server to push updates to the client over a single open HTTP connection.

How it works:

1. Client sends a request.

2. Connection stays open.

3. Server sends updates whenever something happens.

The client keeps listening for these updates.

SSE also has automatic reconnection built into browsers through the EventSource API.

Real-World Example

• GitHub uses SSE to stream live action logs.

• Many analytics dashboards use polling every few seconds.

• Early versions of Twitter used long polling for real-time updates.

Trade-offs

• Short Polling: Simple to implement, but wastes resources if there are no updates.

• Long Polling: More efficient than polling, but harder to manage on the server.

• SSE: Efficient for real-time updates and simple compared to WebSockets, but communication is one-way (server → client only).

The bottom line:

• Polling: Client repeatedly asks for updates.

• Long Polling: Server waits until new data appears.

• SSE: Server pushes updates over a single open connection.

These approaches are useful when WebSockets are unnecessary or unavailable, which helps you keep the system simple and efficient.

Once you understand how apps receive updates from the server, the next step is deciding how the app requests data from the server.

This is where API design becomes important.

APIs define how the mobile app and server communicate, what data is requested, and how that data is returned.

5. REST vs GraphQL vs gRPC

There are different ways to design APIs.

Three common approaches are REST, GraphQL, and gRPC. Each one has its own advantages depending on the type of app you’re building.

REST

REST (Representational State Transfer) is the most common and widely used API style:

1. App calls specific endpoints like /users, /orders, /products.

2. Server returns predefined data.

It is simple and easy to understand, which is why many mobile apps use it.

But REST can sometimes return too much or too little data.

For example, a screen needs only the user’s name and profile photo, but the API returns email, address, preferences, and more. This is called over-fetching.

Sometimes the opposite happens: the app needs multiple API requests to load one screen. This is called under-fetching.

GraphQL

GraphQL was designed to address the problems of over-fetching and under-fetching.

With GraphQL:

1. Client asks for exactly the data it needs.

2. Server returns only that data.

Example request:

GetUser {
  name
  profilePhoto
}

This makes GraphQL very useful for complex mobile interfaces where different screens need different types of data. Yet GraphQL can be harder to cache and secure compared to REST.

gRPC

gRPC is another approach focused on high performance.

Instead of sending JSON data like REST or GraphQL, gRPC uses a binary format called Protocol Buffers, which is smaller and faster.

It runs over HTTP/2, which allows faster communication between services. Because of this, gRPC is often used for internal microservices, backend communication, and high-performance mobile systems.

But it requires special tooling and is less common for public APIs.

Why It Matters

The API style you choose affects network speed, battery usage, number of requests, and app performance.

If an API returns too much data, it wastes bandwidth and battery. If it returns too little, the app needs more requests, which increases latency. Choosing the right approach helps keep the app fast and efficient.

Real-World Example

• Facebook created GraphQL because its mobile apps needed many REST requests to load a single screen.

• GitHub moved its public API to GraphQL.

• Google uses gRPC internally for inter-service communication.

Trade-offs

• REST: Simple, widely supported, and easy to cache, but sometimes returns too much or too little data.

• GraphQL: Clients request exactly the data they need and great for complex UIs, but caching and security can be more complicated.

• gRPC: Very fast, efficient, and great for internal services, but requires HTTP/2 and special tooling.

Practical Advice

A common approach in real systems is:

• REST for simple CRUD APIs.

• GraphQL for complex UI data requirements.

• gRPC for internal communication between services where performance matters most.

Once you decide how the app will request data from the server, the next step is making sure those requests still work properly when the internet connection is unstable.

6. Network Resilience (Exponential Backoff & Retry Strategy)

Mobile networks can be slow, unstable, or unavailable.

As a result, requests may fail due to weak signals, timeouts, or temporary server issues. Instead of failing immediately, most apps retry the request after a short delay.

Yet retrying requests often can cause serious problems.

Why It Matters

Imagine thousands of mobile apps trying to reconnect at the same time after a server outage.

If every app retries instantly, the server suddenly receives a huge number of requests at once. This can overload the server again, causing another failure. This situation is called the thundering herd problem, when many clients try at the same time and overwhelm the system.

A good retry strategy spreads these retries over time so the server can recover…

Exponential Backoff (The Common Solution)

To avoid retrying too aggressively, apps use exponential backoff. This means the app waits longer between each retry:

• First retry → wait 1 second

• Second retry → wait 2 seconds

• Third retry → wait 4 seconds

• Fourth retry → wait 8 seconds

Each retry waits longer than the previous one. Thus reducing pressure on the server.

Adding Jitter

If every app waited the exact same time, they would still retry together.

To avoid that, add jitter, which is a small random delay. For example, retry after 4 seconds + random delay.

This spreads the requests over time instead of sending them all at once.

Real-World Example

Many popular services, such as AWS SDK, Google Cloud client libraries, and Stripe API clients, use this strategy by default. This approach is considered the standard for reliable network communication.

Trade-offs

Retry strategies must balance reliability and user experience.

If the app takes too long to retry, the user may wait too long for a response. If it retries too quickly, it might overload the server.

To handle this, many systems use a circuit breaker.

A circuit breaker stops retrying after several failures and shows a clear error message instead of retrying forever.

After adding retries to handle network failures, another challenge arises: what happens if the same request is sent many times?

Sometimes a request reaches the server successfully, but the response never reaches the app because the network drops.

If the app retries the request, it could perform the same action again…

7. Idempotency in APIs (Safe Retries)

In mobile apps, network requests can fail halfway through.

For example:

1. App sends a request to the server

2. Server successfully processes it

3. But the response never reaches the app because the internet connection drops

From the app’s perspective, the request appears to have failed, so it retries it. If the server processes the request again, the action may happen twice.

This is where idempotency becomes important.

Idempotency means sending the same request many times produces the same result, rather than repeating the action.

How Idempotency Works

To make retries safe, the client sends a unique identifier with each request, called an idempotency key. This key is usually a unique ID (UUID) generated by the app.

Example request header:

Idempotency-Key: 8f3c2c2e-92f1-4f9b-b8b0-1b0f1b1a3e3f

When the server receives a request:

1. It checks whether this key was used before.

2. If the key is new, the server processes the request normally.

3. If the same key appears again, the server does NOT repeat the action. Instead, it returns the previous response.

This prevents duplicate operations.

Why It Matters

Imagine a user taps the “Pay” button.

1. The payment request reaches the server.

2. Server processes the payment successfully.

3. But the response never reaches the phone because the network drops.

So the user taps “Pay” again!

Without idempotency, the user might be charged twice.

This is why idempotency is extremely important for operations that change data.

Real-World Example

Many large systems rely on idempotency:

• Stripe requires an Idempotency-Key for payment API calls.

• Uber uses idempotency keys for ride requests.

• Most financial systems use this approach to prevent duplicate transactions.

Trade-offs

To support idempotency, the server must store previously used keys and their responses for a period of time (typically 24 hours).

This requires some storage, but the cost is small compared to the problems caused by duplicate operations.

It’s also important to design the key carefully, usually combining:

• user ID

• operation type

• unique request ID

Practical Tip

The idempotency key should be generated on the client before the request is sent. The app should also save the key locally.

If the app crashes or restarts, it can retry the request using the same key, ensuring the action is not repeated.

After making retries safe, the next step is to improve the app's efficiency when communicating with the server. One important way to do this is by reducing the number of network requests the app sends.

8. Request Batching & Payload Optimisation

Every time a mobile app sends a request to the server, it adds latency and uses battery.

The total time for a request to reach the server and its response to return is called a network round-trip.

If an app sends many small requests separately, it can slow down the app and drain the device’s battery.

To solve this, use these two techniques:

• Request batching

• Payload optimisation

a. Request Batching

Request batching combines multiple requests into a single request.

Instead of doing this:

App → Request 1
App → Request 2
App → Request 3

The app sends one combined request:

App → Batch Request (Request 1 + Request 2 + Request 3)

This reduces the number of times the app needs to communicate with the server.

b. Payload Optimisation

Payload is the data sent between the app and the server.

Payload optimisation focuses on reducing the amount of data being transferred.

This can be achieved by:

• Compression (like gzip or Brotli)

• Sending only the needed fields

• Using efficient formats like binary data instead of large text formats

Reducing the app's payload size helps it load faster and use less network data.

Why It Matters

On mobile devices, sending a request wakes up the device’s network radio (the hardware responsible for communication).

This wake-up process takes time and consumes battery.

For example:

• Sending 10 separate requests means the radio wakes up 10 times.

• Sending 1 batched request wakes the radio only once.

So batching requests can significantly reduce both latency and battery usage.

Real-World Example

Many popular apps use batching to improve performance.

• Facebook batches many GraphQL queries into a single HTTP request.

• Instagram loads images gradually but fetches data for the next screen in a single batched request.

This makes the app feel faster and smoother.

Trade-offs

Batching improves efficiency but also introduces challenges.

• If the app waits too long to collect requests before batching them, it can add extra delay.

• Another challenge is handling errors. For example, if a batched request contains 10 operations and 3 fail, the system must handle those failures properly.

To balance this, many systems use a short batching window, usually around 50–100 milliseconds, to collect requests before sending them together.

After reducing the number of network requests, another challenge appears when apps need to upload large files, such as photos, videos, or documents.

Large uploads can easily fail on mobile networks because connections may drop or become unstable. To handle this, apps use a technique called resumable uploads.

9. Resumable Uploads (Chunked Uploads)

When a mobile app uploads a large file, such as a 100MB video, sending the entire file in a single request is risky.

If the network disconnects halfway through the upload, the whole upload may fail. The app would have to start again from the beginning.

To avoid this problem, use resumable uploads, also called chunked uploads.

How Chunked Uploads Work

Instead of uploading the entire file at once, it’s split into smaller parts, called chunks.

For example, a 100MB file might be divided into 10 chunks of 10MB each, and each chunk is uploaded separately.

The process usually works like this:

1. App starts the upload and creates an upload session.

2. Server returns a session ID or upload URL.

3. App uploads the file chunk by chunk.

4. Server keeps track of how much data it has already received.

5. If the network fails, the app asks the server where the upload stopped.

6. The upload continues from that point instead of starting from the beginning.

This makes large uploads much more reliable on mobile networks.

Why It Matters

Mobile networks are often unstable.

Imagine uploading a 200MB video that reaches 99% when the network suddenly disconnects, causing the upload to fail…

Without resumable uploads, the app would have to restart the entire upload from the beginning, which wastes time, data, and battery.

With resumable uploads, the app can resume uploading from where it left off instead of starting over.

Real-World Example

Many popular apps use resumable uploads:

• YouTube for video uploads

• Google Drive for file uploads

• Dropbox for syncing files

Any app that allows users to upload media usually uses this approach.

Even with reliable upload systems, mobile apps still face a major challenge: network connections can drop at any time.

Users may lose connectivity when entering a subway tunnel, switching to airplane mode, or moving to an area with poor network coverage. Because of this, mobile apps must be designed to handle unstable or missing network connections without breaking.

10. Handling Intermittent Connectivity

Mobile apps should continue working as smoothly as possible even when the internet connection is weak or temporarily unavailable. Instead of failing immediately, the app should adapt to the network situation and recover when the connection returns.

Use these techniques to handle this:

a. Local Request Queuing

When the device is offline, the app can store user actions locally rather than sending them to the server immediately.

For example:

1. A user sends a message.

2. The app saves the message locally.

3. When the internet connection returns, the app sends it to the server.

This is called request queuing. The app keeps a list of pending actions and processes them once the network is available again.

b. Optimistic UI

Another common technique is optimistic UI.

With optimistic UI, the app updates the interface immediately, assuming the request will succeed.

For example:

• A user likes a post.

• App shows the post as liked instantly.

• Request is sent to the server in the background.

If the request fails later, the app can correct the state.

This makes the app feel fast and responsive, even when the network is slow.

c. Network State Awareness

Apps can also monitor the device’s network status.

This allows the app to know when it is online, offline, or on a slow connection.

Based on this information, the app can adjust its behaviour. For example, it might delay data syncing while offline and resume syncing when connectivity returns.

Why It Matters

Users frequently lose internet access during normal usage.

For example, in subway tunnels, on airplanes, or in areas with weak network coverage. If an app crashes, freezes, or loses user data, it feels unreliable.

Properly handling connectivity problems makes the app feel stable and trustworthy.

Real-World Example

Many modern apps handle intermittent connectivity this way:

• Notion queues edits locally and syncs them later.

• Google Docs allows you to continue editing even when offline.

• Messaging apps show a “sending” or “pending” status until the message is delivered.

Practical Tip

Mobile operating systems provide built-in tools to monitor network connectivity:

• NWPathMonitor on iOS

• ConnectivityManager on Android

These tools notify the app when the network status changes.

This approach is event-driven and battery-efficient, unlike constantly checking the network in a loop.

Even with reliable networking strategies, mobile apps cannot always depend on an internet connection. To keep the app fast and usable even without internet access, developers use caching.

Caching & Offline

This section focuses on how mobile apps store and reuse data to reduce network requests and improve performance.

You’ll learn how caching strategies and offline-first approaches help apps stay fast and usable even when the internet is slow or unavailable.

11. On-Device Caching (Memory vs Disk)

Mobile apps usually store cached data in two places on the device:

• Memory (RAM)

• Disk (device storage)

Using both together creates a two-layer caching system.

a. Memory Cache (RAM)

Memory cache stores data in device RAM, which is extremely fast to access.

When data is stored in memory:

• App can load it almost instantly.

• It avoids making a network request.

Yet memory has a limitation:

• Data disappears when the app closes.

• Operating system may remove cached data if the device runs low on memory.

So memory caching is fast but temporary.

b. Disk Cache (Storage)

Disk cache stores data on the device’s permanent storage.

This means:

• Data remains available even after the app restarts.

• App can load previously downloaded data without contacting the server again.

But disk access is slower than memory access.

So disk caching is slower than RAM but more persistent.

Why It Matters

Fetching data from the server can take 50–500 milliseconds, depending on the network. But reading data from memory happens in nanoseconds, which is almost instantaneous.

So if the app can load data from the cache instead of the network, the user interface feels much faster. For example, profile pictures, app configurations, or recently viewed content.

These are perfect candidates for caching.

Real-World Example

Many image-loading libraries use this two-layer caching approach. For example, SDWebImage and Kingfisher.

When an image is requested:

1. App first checks the memory cache.

2. If it’s not there, it checks the disk cache.

3. Only if both are missing does it requests image from the network.

This makes images load almost instantly.

Trade-offs

Both types of caching have advantages and limitations:

• Memory Cache: Extremely fast, but cleared when the app closes.

• Disk Cache: Data remains available after restarting the app, but reading from disk is slower than memory.

Cache size also needs to be controlled carefully.

If too much memory is used for caching, the operating system may remove the cached data or close the app to free memory.

Practical Tip

Mobile platforms provide built-in caching tools that automatically manage memory. For example, NSCache on iOS and LruCache on Android.

These systems automatically remove older items from memory when the device is running low on resources. This helps prevent the app from using too much memory.

Using simple data structures like dictionaries or hash maps for caching is NOT ideal, because they do not automatically respond to memory pressure. As a result, they may keep too much data in memory, increasing the risk that the app will be terminated by the operating system.

Local caching stores data on the device, which helps apps load faster.

But the internet itself also has built-in caching mechanisms that can reduce unnecessary network requests. This is where HTTP caching becomes useful.

12. HTTP Caching (ETag, Cache-Control)

HTTP caching allows apps and browsers to reuse previously downloaded responses instead of requesting the same data again.

This is controlled using special HTTP headers, Cache-Control, and ETag. These headers tell the app two important things:

• How long can cached data be reused

• How to check if the data has changed

a. Cache-Control

This header tells the app how long it can reuse the cached data before asking the server again.

Cache-Control: max-age=300

This means the response can be reused for 300 seconds (5 minutes).

During those 5 minutes, the app can load the data directly from cache instead of making a new request to the server.

This improves speed and reduces network usage.

b. ETag

An ETag is a unique identifier that represents a specific version of the data.

When the server sends a response, it also includes an ETag value:

ETag: “abc123”

When the app requests the same data again, it sends that value back to the server:

If-None-Match: “abc123”

The server checks whether the data has changed:

• If the data has NOT changed, the server returns 304 Not Modified with no response body.

• If the data has changed, the server sends the new data.

This saves bandwidth because the server does NOT need to resend the entire response.

Why It Matters

HTTP caching reduces unnecessary network requests.

For example, if an API response is 200 KB, returning a 304 Not Modified response avoids downloading that 200 KB again.

This helps improve app performance, network efficiency, and battery usage. The best part is that HTTP caching works automatically when the correct headers are set.

Real-World Example

A news app might load a list of articles using an API. The server might return a header like this:

Cache-Control: max-age=60, stale-while-revalidate=300

This means cached response can be used for 60 seconds. After that, the app can still show the cached content while checking for updates in the background.

As a result, the articles appear instantly without showing a loading spinner.

Trade-offs

Different caching strategies have different advantages:

• Strong caching (long max-age): Faster loading and fewer network requests, but the data may become outdated.

• Weak caching (ETag validation): Always checks if data is fresh, but still requires a small network request.

Because of this, cache durations should match the frequency of data changes.

Caching helps apps load data faster, but it still depends on the network at some point. When the network connection disappears completely, caching alone cannot solve the problem.

To keep apps usable even without a network connection, many apps use a design approach called offline-first architecture.

13. Offline-First Architecture

Offline-first architecture means the app is designed to work normally even without a network connection.

Instead of always asking the server for data, the app mainly works with data stored locally on the device. The app shows and updates local data first, and then synchronises changes with the server when the network becomes available.

This approach helps the app stay fast and responsive, even when the network is slow or unavailable.

How It Works

In an offline-first system, the app reads and writes data from local storage first.

The process usually works like this:

1. The app reads data from the local database instead of requesting it from the server.

2. When the user makes a change, the app updates the local database immediately.

3. A background process then sends those changes to the server when a connection is available.

4. If the server has new updates, the app downloads them and updates the local database.

5. When the local data changes, the app automatically updates the user interface.

Because the UI always reads from local data, the app feels fast and responsive.

Why It Matters

Apps that depend entirely on the network often feel slow.

Every action requires a request to the server, which can take 100–300 milliseconds or more. This delay often appears as loading spinners.

Offline-first apps avoid this problem.

Since the UI reads from local data, interactions feel instant, while the network syncing happens quietly in the background.

Real-World Example

Many popular apps use an offline-first architecture:

• Spotify downloads playlists so they can be played without internet.

• Notion saves edits locally and syncs them later.

• Google Maps stores map tiles for offline navigation.

Apps dealing with media, productivity, or travel benefit from this approach.

Trade-offs

Offline-first systems provide a better user experience, but they are complex to build.

They require:

• Local database to store data

• Sync system to send and receive updates

• A way to handle conflicts when local and server data differ

• Plan for updating the data structure over time

Because of this complexity, an offline-first design usually needs to be planned early in app development.

Caching helps apps reuse data locally, but another challenge is delivering large files like images and videos efficiently. These files often make up most of the data in app downloads.

To improve speed and reduce data usage, use techniques like CDNs and media optimisation.

14. CDN Strategy & Media Optimisation

Images and videos usually make up a large part of the data used by mobile apps.

To deliver this content quickly and efficiently, apps often rely on Content Delivery Networks (CDNs) and various optimisation techniques. A CDN is a network of servers distributed around the world. Instead of downloading content from a central server, the app downloads it from the nearest server, reducing load time.

While media files can be optimised, the app only downloads what it actually needs.

What a CDN Does

When a user opens an app and requests an image/video:

1. The request is sent to the closest CDN server.

2. CDN returns the file from a nearby location instead of a distant server.

3. The content loads faster because the distance and network delay are reduced.

This speeds up and improves the reliability of content delivery, especially for users in different parts of the world.

Media Optimisation

Another important improvement is optimising the size and format of images and videos.

Some common techniques include:

• Image resizing: Instead of sending a large image (e.g., 4K), the server sends a smaller version that fits the device screen, such as 200×200 pixels.

• Modern image formats: Newer formats like WebP or AVIF store images more efficiently than older formats like JPEG or PNG. They provide similar image quality with smaller file sizes.

• Lazy loading: Images are loaded only when they are about to appear on the screen. This avoids downloading content that the user might never see.

Together, these techniques significantly reduce data usage and improve loading speed.

Why It Matters

Large media files can slow down apps and consume a lot of mobile data.

For example, loading a 4MB image on a small phone screen wastes most of its pixels because the device only displays a small portion of the image’s resolution.

Serving images that match the device size can reduce total data usage by up to 80%, making the app faster and more efficient.

Real-World Example

Many popular apps rely on CDN and media optimisation:

• Cloudinary and Imgix dynamically resize images based on URL parameters.

• Instagram serves images that match the exact resolution of the user’s device.

• On modern Android devices, Instagram also uses the AVIF format to reduce image size.

Trade-offs

Using a CDN improves performance but introduces additional cost, as CDN providers charge based on data transfer.

As a result, CDN assets are usually cached for long periods. Apps often use versioned or hashed URLs to update assets. This process ensures that the new version gets downloaded when needed.

Practical Tip

To make images appear faster, apps often show a temporary low-quality preview while the full image loads.

Techniques like BlurHash or LQIP (Low Quality Image Placeholder) display a blurred or low-resolution version first. This makes the page feel fast, even before the high-quality image finishes loading.

Even with strong caching and efficient content delivery, one important challenge remains: keeping cached data accurate and up to date.

15. Cache Invalidation Strategies

Caching helps apps load data faster by storing it locally.

But cached data can become outdated when the original data on the server changes. Because of this, apps need a way to update or remove old cached data. This process is called cache invalidation.

In simple terms, cache invalidation means deciding when cached data should no longer be used and needs to be refreshed.

Why It Matters

If outdated data stays in the cache for too long, the app may show incorrect information.

For example, a user might see an old product price, a deleted message, or an outdated address. In these situations, showing stale cached data can be worse than not using a cache at all.

That’s why managing cached data correctly is very important!

Common Cache Invalidation Strategies

There are several common ways to control when cached data should expire or update:

a. TTL (Time-To-Live)

TTL means cached data is only valid for a certain amount of time.

For example, if a cache has a TTL of 5 minutes, the app will reuse the cached data for five minutes. After that, it must fetch fresh data from the server.

This approach is simple but may briefly show outdated data.

b. Event-Driven Invalidation

In this approach, the server actively tells the app when cached data is no longer valid.

For example, a product price changes, and the server sends an event to clear or update the cache.

This ensures the app always shows the latest data, but it requires more complex systems to send these updates.

c. Stale-While-Revalidate

This strategy allows the app to show cached data immediately, even if it is slightly outdated.

The app quietly requests fresh data in the background. When the new data arrives, the cache is updated.

This approach improves the user experience by loading the app instantly while keeping data up to date.

d. Versioned URLs

Sometimes cached assets like images or style sheets are stored for a very long time.

To update them safely, the filename or URL includes a version or hash.

Example:

style.abc123.css

When the file changes, the version in the URL changes. Because the URL is different, the cache automatically downloads the new file.

Real-World Example

An e-commerce app might use different caching strategies depending on the data.

For example:

• Product listings may use a TTL of 5 minutes, since a short delay in updates is acceptable.

• Inventory or pricing data may use event-driven invalidation because it must always be accurate.

• Static assets, such as images or stylesheets, can use versioned URLs, allowing them to be cached for a long time.

Trade-offs

Each caching strategy has its advantages and limitations:

• TTL-based caching: Simple to implement, but cached data may briefly become outdated.

• Event-driven invalidation: Keeps data accurate but requires more complex infrastructure.

• Stale-while-revalidate: Makes the app feel fast, but temporarily shows slightly outdated data.

Important Tip

Caches should never have infinite expiration times for data that changes frequently.

If cached data never expires, the app may continue showing outdated information, making it difficult to detect and debug.

Once data is cached or fetched, the app also needs a reliable way to store and organise that data locally, which leads to the next part: storage and database design.

Storage & Data

Mobile apps need efficient ways to store and manage data on the device.

This section explains how local databases, schema design, and pagination help apps handle large datasets while maintaining smooth performance on resource-limited devices.

16. Local Database Design (Schema Modeling)

Mobile apps often store data locally on the device so they can load information quickly and continue working even when the network is slow or unavailable.

To do this, apps use local databases such as SQLite, which is commonly used on both Android and iOS. A database schema simply describes how data is organised in the database. It defines tables, columns, and how different pieces of data are connected.

Good schema design is important because it helps the app load and display data quickly.

How Data Is Often Structured on Mobile

On servers, databases are usually designed to avoid duplicating data across many places. This approach is called normalisation.

But mobile apps often use a slightly different approach called denormalisation.

Denormalisation involves storing related data in the same table, enabling quick reads without complex queries.

This helps mobile apps load data faster because the app does NOT need to combine multiple tables every time it displays information.

Why It Matters

Mobile apps need to display data quickly.

For example, imagine an app showing a list of 50,000 messages. If the database is poorly designed, finding the right messages might take much longer than expected.

A missing index (a structure that helps the database locate data quickly) can cause the database to scan every row in a table. This can turn a 1-millisecond query into a 500-millisecond query, causing visible delays or lag in the user interface.

So the way a database is designed directly affects how smooth the app feels.

Real-World Example

Consider a messaging app.

Instead of storing only a user ID with each message, the app may also store the sender’s name and profile image directly inside the message record.

This means the app can display messages immediately without looking up additional data from another table.

This approach speeds up loading when thousands of messages need to be shown quickly.

Trade-offs

Each database design approach has advantages and disadvantages:

• Denormalised structure: Faster data reading and works well for displaying lists in the UI, but updating repeated data can be harder. For example, if a user changes their name, multiple records may need to be updated.

• Normalised structure: Cleaner, more organised, and easier to maintain consistent data, but requires more complex queries, which may slow down UI rendering.

Because mobile apps mostly read data to display it on the screen, they often prefer denormalised structures for faster reads.

Practical Tip

When writing many records to the database, it’s usually faster to group them into a single transaction.

Instead of saving each item one by one, the app saves many items together in a single operation. This reduces overhead and makes database writes much more efficient.

As apps grow and evolve over time, the database structure may need to change as well. Because of this, apps must handle database schema updates carefully, which leads to the next topic: schema migration strategies.

17. Schema Migration Strategy

Mobile apps often store data in a local database on the device.

Over time, the app may change how this data is structured. For example, a new version of the app might add a new column, change a table, or introduce new data fields.

When this happens, the database structure must be updated safely. This process is called schema migration.

A schema migration ensures that the existing database on a user’s device is updated to match the new version of the app.

Why It Matters

Unlike a web server, mobile databases exist on millions of individual devices that developers cannot directly control.

Some users may update the app immediately, while others may skip several versions before updating. For example, a user installs version 1 of an app and later updates directly to version 5.

The database must correctly apply every change that happened between those versions. So the migrations must run step-by-step:

1 → 2 → 3 → 4 → 5

If a migration fails, the app may crash on launch. Because the database exists on the user’s device, fixing it later can be difficult…

Real-World Example

Many mobile frameworks provide tools to manage database migrations.

For example, Room on Android requires every database version to define how the schema should be updated.

This helps prevent mistakes where the app changes the database structure without providing a safe upgrade path.

Trade-offs

There are two main ways to handle schema changes:

a. Destructive Migration

In this approach, the app deletes the existing database and creates a new one.

This is simple and safe for the app structure, but it removes all locally stored data.

b. Additive Migration

In this approach, the database is updated without deleting existing data.

For example, adding new columns, adding new tables, and keeping old data intact.

This method is usually preferred because it preserves user data.

Practical Tip

Sometimes migrations may fail due to issues like a corrupted database.

In these situations, it is better for the app to reset the database and re-download the data from the server rather than crash. Losing cached data can be recovered, but an app that keeps crashing when it starts creates a much worse experience.

As apps handle larger amounts of stored data, another important challenge appears: loading and displaying large datasets efficiently, which is where pagination becomes useful.

18. Pagination (Cursor vs Offset vs Page Number)

Mobile apps often display long lists of data, such as messages, posts, products, or search results.

Sometimes these lists can contain thousands of items.

Loading all of that data at once would be slow and would use a lot of memory. Because of this, apps usually load data in smaller chunks called pages. This technique is called pagination.

Pagination helps apps load content gradually as users scroll, keeping the app fast and responsive.

How It Works

Instead of loading everything at once, the app requests a small set of items at a time.

For example:

1. App requests page 1 of data.

2. The server returns the first group of items.

3. When the user scrolls further, the app requests the next page.

This process continues until the user reaches the end of the list…

Common Pagination Methods

There are three common ways to implement pagination:

a. Page Number Pagination

This is the simplest method. The client requests data using page numbers.

For example:

?page=1
?page=2
?page=3

Each page contains a fixed number of items.

This approach is easy to understand and works well for things like search results or product listings.

b. Offset Pagination

Offset pagination retrieves data starting from a specific position.

For example:

?limit=20&offset=40

This means skip the first 40 items and return the next 20 items.

This method works well for static data, but dynamically adding new items while the user is browsing can cause problems.

For example, if new posts appear at the top of a feed, the list may show duplicate items or skip items.

c. Cursor Pagination

Cursor pagination uses a cursor token that represents the last item the user has seen. Instead of requesting a page number, the app requests the next items after a specific position.

For example:

next_cursor = “abc123”

Then the next request might look like:

?after=abc123

The cursor usually represents a stable value, such as the ID or timestamp of the last item. This approach works well for data that constantly changes, such as social media feeds.

Why It Matters

Imagine a social feed with 10,000 posts. If the app tried to load everything at once:

• The request would take a long time

• Rendering the UI would be slow

• Device might run out of memory

Pagination solves this by loading only a small portion of the data at a time, which keeps scrolling smooth.

Real-World Example

Many popular apps, such as Twitter, Instagram, and TikTok, rely on cursor-based pagination.

These apps load new posts as the user scrolls, using a cursor that points to the last item that was displayed.

This ensures the feed remains stable even when new posts appear…

Trade-offs

Each pagination method has advantages and limitations:

• Page number pagination: Simple to understand and works well for search results, but less efficient for large datasets.

• Offset pagination: Easy to implement and allows jumping to any position, but it can break when new data is inserted.

• Cursor pagination: Stable and efficient for live feeds and prevents duplicates and skipped items, but does not easily support jumping directly to a specific page.

Because mobile devices have limited memory and storage, loading and storing data must be done carefully! Therefore, it’s important to design efficient data models that work well within these constraints.

19. Data Modeling for Mobile Constraints

Mobile apps often store data locally on the device so they can load information quickly and continue working even when the network is slow.

Yet mobile devices have limited storage, CPU power, and battery, so the way data is structured must be carefully designed.

Because of these limits, mobile apps usually store and organise data to help the UI load quickly, even if it means repeating some information.

Things Mobile Data Models Must Consider

When designing data models for mobile apps, several important constraints should be considered:

a. Limited storage

Mobile devices cannot store unlimited data.

Apps should avoid caching everything and instead remove old or unused data after a period of time. For example, an app might store cached data for a few hours or days, then automatically delete it.

b. CPU performance

Complex database queries can slow down the app.

Queries that join many tables may take longer to process. To keep the app fast, store related data together so the app can read it quickly without performing complicated queries.

c. Battery usage

Every time an app writes data to disk, it uses battery power.

Frequent database writes can drain the battery over time. Because of this, apps try to minimise unnecessary writes and batch operations when possible.

Why It Matters

Data models used on backend servers are usually designed to save storage space and keep data perfectly structured.

But mobile apps have a different goal: fast rendering on the screen.

This means the data model on mobile may look different from the backend data model, because it’s optimised for speed and user experience rather than strict structure.

Real-World Example

Imagine a contacts app that shows a list of people.

Instead of calculating everything each time the list loads, the app may store some information in advance, such as display name, profile picture URL, and the first letter used for alphabetical sections.

Because this information is already stored, the app can render the list instantly without performing extra calculations.

Trade-offs

Storing extra or pre-computed data improves speed, but it also has some downsides.

Pre-computed data uses more storage space and must be updated whenever the original data changes.

Yet this trade-off is usually acceptable because storage is relatively cheap, while slow or laggy interfaces are immediately noticeable to users.

Final Words

None of these concepts works on its own.

The choices you make in one area affect everything else. Networking affects caching, caching affects sync, and sync affects how conflicts are handled.

It’s all connected.

The best engineers aren’t the ones who know the most concepts. They’re the ones who understand the tradeoffs and know when to use what.

So remember this:

Start simple. Add complexity only when you really need it. And when that time comes, you’ll know what to use.

👋 I’d like to thank Shefali for writing this newsletter!

Plus, don’t forget to check out her work and socials:

• Shefali.dev

• GitHub

• Twitter

You’ll often find her writing about web development, sharing UI tips, and building tools that make developers’ lives easier.

Get the full premium newsletter series and max your system design career leverage:

Subscribe now

Plus, there are group discounts, gift options, and referral rewards available.

Want to reach 210K+ tech professionals at scale? 📰

If your company wants to reach 210K+ tech professionals, advertise with me.

Thank you for supporting this newsletter.

You are now 210,001+ readers strong, very close to 210k. Let’s try to get 211k readers by 27 March. Consider sharing this post with your friends and get rewards.

Y’all are the best.

Share

• Block diagrams created using Eraser.

• Share this post & I'll send you some rewards for the referrals.

You’ve probably been there: You’re sitting across from two engineers (or staring at them through a webcam), and they’ve just asked you to design Instagram’s feed. Or a ride-sharing app. Or some chat system with offline support and end-to-end encryption.

The clock is ticking…

You have 45 minutes to turn a vague prompt into something that looks like you know what you’re doing.

Your mind races: Where do I even start? Do I draw boxes first? Ask about the user numbers? Talk about databases? What if I forget something critical?

Most mobile engineers don’t realize this about system design interviews: They’re NOT testing whether you can build Instagram. They’re testing whether you can take something fuzzy and turn it into something concrete.

This is the same skill you use every day when your PM says, “We need push notifications,” or your designer drops a Figma file in Slack with the message “Thoughts?” in your day job.

12% OFF MiniMax M2.7 - The SOTA Cowork Agent Model That Just Outranked Opus and Gemini 3.1. (Partner)

MiniMax M2.7 is here, and it’s unlike any agent model released before it.

Built on their proprietary SOTA Cowork Agent Model architecture, M2.7 shows early echoes of self-evolution: the ability to follow complex, layered instructions across long workflows without losing the thread.

At 5 tools, every model looks sharp.

At 40, almost all fall apart silently.

Wrong tool invoked. Step missed. No error thrown. You find out 3 days post-deploy.

M2.7 was tested across 40 complex skills, each exceeding 2,000 tokens.
Skill adherence rate: 97%.

Here’s what that looks like in production:

• Alert fires → metrics correlated → root cause traced across systems

• DB queried to verify → fix submitted

• Recovery time: under 3 minutes. Manual process: 4+ hours.

Most models are built to write code.

M2.7 is built to understand your production systems.

And it ships inside something new: the world’s first subscription plan for an all-modality-capable model. One API key covers M2.7, Speech, Image, Video, and Music generation; the full stack, no provider switching.

MiniMax Token Plan 12% OFF

I want to introduce Tjeerd in ‘t Veen as the guest author.

He recently released the Mobile System Design book bundle based on years of interviewing mobile engineers and building large-scale mobile apps. This newsletter distills the core process (framework) from the books into something you can use to prep for interviews or plan features at work.

If you want to go deeper, the full book bundle has more examples, detailed architecture patterns, and code-level implementation strategies.

What this newsletter covers

There are plenty of great general system design guides. This one is about mobile constraints and how they shape the architecture.

This newsletter walks through the actual process:

• How to take a vague prompt and figure out what the interviewer actually wants

• How to uncover the hidden requirements nobody mentioned (because they’re testing if you’ll think of them)

• How to structure your thinking so you don’t forget the mobile-specific stuff (offline state, battery drain, app lifecycle, memory constraints)

• How to turn abstract architecture talk into concrete implementation details

This works whether you’re in an interview or planning a real feature at work.

The process is the same.

Who this is for

If you’re a mobile engineer (iOS, Android, Flutter, React Native, etc.) and you need to:

• Prepare for system design interviews

• Stop feeling like you’re fumbling when asked to “design something.”

• Learn how to think through mobile architecture in a structured way

• Get good at turning vague requirements into actual plans

Then this is for you…

I’m assuming you already know how to develop mobile apps. You understand the basics of networking, data persistence, and why you shouldn’t do heavy work on the main thread. You’ve shipped something to production and dealt with the consequences.

What you might not have is a structured way to think about system design. That’s what we’re covering here…

Interview Structure

Let’s talk about what actually happens in a mobile system design interview:

The format can vary wildly between companies. Some give you 45 minutes, others give you 60. Some want you on a whiteboard, others use tools like CodeSignal or HackerRank. Some companies even ask backend system design questions in mobile interviews, which is a separate discussion entirely.

But most mobile-focused interviews follow a similar arc.

Here’s what you can typically expect…

Typical flow

Most interviews break down roughly like this:

At a high level, the flow is:

• Requirements and scope (5-10 minutes): clarify what you’re designing and the constraints

• API and data needs (5-10 minutes): define what the app needs from the server

• Architecture (10-15 minutes): outline client components and data flow

• Deep dive (15-20 minutes): walk one critical flow or adjust for a new constraint

• Wrap-up (1-5 minutes): summarize decisions and tradeoffs

I’ll go deep on how to execute each phase in the next section…

What’s expected at different levels

Interviewers don’t judge all mobile engineers the same way.

Here’s what interviewers are actually looking for based on your level:

Junior to mid-level engineers

You’re expected to:

• Ask clarifying questions (even if they seem obvious)

• Draw a reasonable architecture diagram without major gaps

• Explain common mobile patterns (caching, background tasks, handling lifecycle)

• Discuss basic tradeoffs - ”This approach is simpler but uses more memory.”

You’re not expected to:

• Know every edge case off the top of your head

• Have opinions on monorepo versus multi-repo

• Design for 100 million users on day one

If you get stuck on something complex (like offline conflict resolution), just say so. “I haven’t built a system that merges offline edits before, but I’d start with last-write-wins and ask about requirements,” is fine.

Honesty beats bullshitting.

Senior engineers

Everything from junior/mid, plus:

• Proactively uncover hidden requirements without prompting - ”What about users on limited data plans?”

• Propose architecture with clear reasoning for your choices - ”I’m using a repository pattern here because we need to support offline-first”

• Demonstrate deep knowledge of mobile-specific challenges (app lifecycle, memory pressure, battery drain, offline sync)

• Adapt when requirements change mid-interview - ”Okay, if we need to support 50MB video uploads, we’ll need background upload with chunking…”

You should drive the conversation.

The interviewer shouldn’t have to pull answers out of you.

Staff and principal engineers

Everything from senior, plus:

• Frame the problem: “Before we design this, let’s talk about whether we even need offline support for this use case.”

• Consider team structure: “If we structure this as a separate module, we can have one team own it independently.”

• Think long-term: “This works for v1, but if we add feature X later, we’ll need to refactor. Let’s design for that now.”

• Make architectural tradeoffs at the system level: modularization strategy, build times, deployment, backward compatibility

You’re expected to think like a tech lead or architect, not just an IC implementing a feature. The interviewer wants to see if you can make decisions that affect the entire team or org.

Now that you know what the interview structure looks like, let’s talk about the actual process you’ll use during those 45 minutes.

Core Process

The interview structure gives you the timeline. But what do you actually do in each phase?

This is the process you’ll follow, roughly in order, during the first 30-40 minutes of the interview. Think of it as a checklist for making sure you forget nothing critical.

You won’t always do these steps in exactly this order. Sometimes you’ll jump back and forth. Sometimes the interviewer will steer you in a different direction. That’s fine.

But this is the general flow that works:

A reusable pattern (applies to every step)

Most steps look different, but the rhythm is the same:

• Ask 2-3 high-impact questions before designing.

• Restate what you heard and get a clear “yes.”

• Write down decisions: constraints, must-haves versus nice-to-haves, and open questions.

• Call out red flags early instead of waiting.

• You don’t have to solve everything on the spot. Show you see the issue, pick a reasonable tradeoff, and move on.

Capturing the briefing

Here’s how most system design interviews start:

The interviewer says something like, “Design a feed for a social media app,” or “Build a way for users to share photos with friends.”

And your brain immediately starts racing: Should I ask about the backend? Should I start drawing? What if I forget to mention caching?

Before you do any of that, pause.

Your first job is not to design anything. It’s understanding what you’re being asked to design.

This sounds obvious, but most candidates blow past this step in 60 seconds and then spend 40 minutes designing the wrong thing…

What you’re actually trying to do

In the first 5-10 minutes, you’re trying to turn a vague prompt into something concrete enough that you know what problem you’re solving.

You’re not trying to get every detail. You’re trying to get enough clarity that you’re not completely guessing.

Think of it like this: If someone asks you to “design a messaging feature,” do they want:

• WhatsApp (end-to-end encrypted, real-time, requires phone number)

• Slack (threaded, searchable, integrates with other tools)

• Twitter/X DMs (simple, text-only, tied to social graph)

All of those are “messaging.” They’re also completely different architectures.

So your job is to ask questions that reveal which one they actually want.

Questions that actually matter

Here’s what you should ask:

About success criteria:

• “What does success look like for this feature?” (Are we optimizing for speed? Reliability? Simplicity?)

• “What’s the most important thing this needs to do well?” (If we could only do one thing right, what would it be?)

About scope:

• “Is this the whole app, or are we focusing on one part?” (Don’t design all of Instagram if they just want the feed)

• “What platforms are we targeting?” (iOS only? Android? Cross-platform?)

About constraints:

• “What network conditions should we plan for?” (Always online? Intermittent? Fully offline?)

• “Are there any technical constraints I should know about?” (Existing systems we need to integrate with? Legacy code?)

About users:

• “Who are the primary users?” (Consumers? Enterprise? Internal tools?)

• “What devices are they on?” (Flagship phones? Low-end Android? Tablets?)

Don’t ask everything.

Pick the questions that will most change your design. You have limited time.

A bad and a good example

Bad example:

Interviewer: “Design a feed for a social media app.”

You: “Okay, so I’ll use MVVM for the architecture. We’ll have a ViewModel that fetches data from a Repository, which talks to both a local database and a REST API. UI will be built with—”

Interviewer (internally): They didn’t ask a single question. Do they even know what they’re designing?

You just spent 3 minutes designing an architecture without knowing:

• Whether it needs to work offline

• Whether it’s a text-based feed or heavy on images and video

• Whether it requires pre-caching for a smoother scrolling experience

That’s a red flag. Interviewers are explicitly looking for whether you ask questions.

Good example:

Interviewer: “Design a feed for a social media app.”

You: “Okay, let me make sure I understand what we’re designing. Is this feed mostly images and video, or mostly text? That affects caching and pre-fetching on mobile.”

Interviewer: “Mostly images, some video.”

You: “Got it. For mobile, I’d plan on disk caching images (and maybe short videos) and pre-fetching upcoming content so the feed loads fast and works offline. Is that aligned with what you want?”

Interviewer: “Yes, basic offline support for previously loaded content is important.”

You: “Makes sense. One more thing: do we need live updates, or is pull-to-refresh enough for v1?”

Interviewer: “Pull-to-refresh is fine.”

You: “Perfect. So, to summarize: we’re designing a media-heavy feed with basic offline support. We’ll cache images (and maybe short videos), pre-fetch upcoming content for fast loads, and start with pull-to-refresh for updates. Does that sound right?”

Interviewer: “Yep, that’s it.”

Reminder: this is a teaser of the subscriber-only newsletter series, exclusive to my golden members.

When you upgrade, you’ll get:

• High-level architecture of real-world systems.

• Deep dive into how popular real-world systems actually work.

• How real-world systems handle scale, reliability, and performance.

Subscribe now

Briefing wrap-up (red flags + next steps)

Sometimes the briefing is incomplete or contradictory. Call it out early instead of guessing…

Common red flags:

• Requirements that are vague or overly broad - ”make it fast.”

• Contradictory statements - ”we want it simple, but also highly customizable.”

• Missing critical info (no mention of offline, auth, or data sources)

When you spot these, call them out:

“You mentioned we want it to be ‘fast.’ Can you be more specific? Are we talking about load time under 1 second, smooth scrolling, or something else?”

“I’m noticing we haven’t talked about where the data comes from. Should I assume we’re pulling from a REST API, or is there a different data source?”

Don’t rush ahead, hoping it’ll become clear later. It won’t. You’ll end up designing something that doesn’t match what they wanted…

Next steps once the briefing is clear:

• Write down key constraints in the shared doc or on the whiteboard (”Offline support required”, “100k users”, “Existing auth system”)

• Mentally categorize what is a must-have versus a nice-to-have

• Identify the biggest unknowns you’ll need to address

Lesson: Don’t start designing until you know what you’re designing.

This can feel slow, but it’s faster than designing the wrong thing…

Defining scope and requirements

Okay, so you’ve asked your questions, and you have a basic understanding of what you’re designing. Now comes the hard part: figuring out what you’re actually going to design.

Here’s the problem: If someone asks you to “design a messaging feature,” there are about 47 different things they could mean. Do they want read receipts? Typing indicators? Group chats? Voice messages? End-to-end encryption? Reactions? Threads?

You can’t fit all of that into v1. You probably can’t even fit half of it.

So your job is to separate what’s essential from what’s nice-to-have.

This is where most people mess up. They either:

1. Try to include everything at once and run out of time

2. Design the wrong things and miss what actually matters

3. Don’t ask at all and just guess

All three are bad. Let’s talk about how to do this right:

What you’re trying to figure out

You need to answer these questions:

Read more

• Share this post & I'll send you some rewards for the referrals.

Every large language model1 (LLM) you use has lied to you with confidence, fluency, and frequency…

Ask any model about something that happened last week. It doesn’t know. It can’t know. Its knowledge was frozen months ago. They might try, and if they do, they will hallucinate.

This isn’t a bug.

It’s a fundamental architectural limitation. LLMs keep knowledge in their parameters2. These are billions of numerical weights learned during training. Once training ends, the knowledge is locked. The model doesn’t know what it doesn’t know, so it fills the gaps with confident fabrication. Studies show that hallucination3 rates can be as low as 1% for simple summarization tasks. However, they can exceed 58% for complex work.

This is exactly what Retrieval-Augmented Generation4 (RAG) solves.

RAG doesn’t bake knowledge into the model. Instead, it pulls in relevant context when you ask a question. The model stays smart, your data stays current, and every answer is traceable back to its source.

This is the architectural pattern that makes AI actually useful.

Here’s how it works, why it works, and how to build it:

Still juggling 10 different tools? Learn AI workflows and replace 80% of them (Partner)

Perplexity’s new computer thinks, designs, codes, and manages projects — all without you lifting a finger to switch tools.

This is where AI is headed: one place to get everything done. Smarter, faster, and built to make you 10x more productive.

Now is your chance to enter into your most productive era!

That’s why we recommend joining Outskill- the world’s first AI learning platform where over 10+ Million Learners have learnt from top industry leaders like Microsoft, NVIDIA, and Google.

They are hosting a 2-day LIVE AI Mastermind where you’ll build automations, create personalized agents, and learn to turn AI into your ultimate competitive edge.

START LEARNING NOW

You will also unlock exclusive bonuses for free when you show up: A Prompt Bible, AI monetization roadmap, and a personalized toolkit builder - for which you would have to pay $1000+ outside.

🧠 Happening LIVE- Saturday and Sunday
🕜 10 AM EST to 7PM EST

Register here before they run out of seats. (free for the next 72 hours only!)

I want to introduce Eric Roby as a guest author.

He’s a senior backend and AI engineer focused on building real-world systems and teaching developers how to do the same. He runs the YouTube channel @codingwithroby, where he focuses on backend engineering, and created the platform The Backend OS.

Through his content and courses, he helps engineers go beyond tutorials, think in systems, and develop the skills that actually matter for real backend roles.

Check out The Backend OS, built to close the knowledge gaps you don’t even know you have.

The Knowledge Problem

We need to understand why LLMs struggle with real-world knowledge before we fix them. The problem has three layers, and none of the obvious solutions work…

LLM knowledge is frozen at training time

Every LLM has a cutoff date for its knowledge.

Everything the model knows was scraped, processed, and compressed into parameters during training. After that date, the model is blind. It isn’t aware of your new product launch, yesterday’s security issue, or this morning’s HR policy change.

This isn’t a minor inconvenience.

In business, accuracy is crucial. This includes areas like customer support, legal analysis, internal search, and compliance. A model that can’t access current information is a liability. It doesn’t help the business at all.

Context Windows Aren’t the Answer

The brute-force approach is tempting: dump everything into the prompt.

Context windows5 have increased a lot. Some models can now handle over a million tokens6. But this approach has three fatal problems.

First, it’s expensive.

You pay for each token. Sending your entire knowledge base with every query will wipe out your budget.

Second, there are hard limits.

A million-token window can’t fit all of a large enterprise’s documents or its databases.

Third, and this is the one most people miss, models get worse with more context.

Research from Stanford and UC Berkeley shows LLM performance follows a U-shaped curve. Models do best when key information is at the start or end, but accuracy drops sharply when important facts are buried in the middle. In Liu et al.’s multi-document QA experiments, accuracy for some models dropped to roughly 25% when key information was placed in the middle of a 20-document context.

It’s clear: adding more context to the prompt doesn’t guarantee the model will use it.

Fine-tuning isn’t the answer either

Fine-tuning7 is when you take a pre-trained model and continue training it on your own specific data, so it learns new knowledge or behavior.

Fine-tuning changes the model’s weights to incorporate new knowledge. Think of it like sending someone back to school for a specialized course. In theory, this lets you teach the model about your domain.

In practice, it creates more problems than it solves…

It requires GPU compute, machine learning expertise, and carefully prepared training data. It takes days or weeks to complete. The result is a snapshot. Once your underlying data changes, your fine-tuned model becomes stale.

What’s Actually Needed

The key is to provide the AI with the right information at the right time for each query. And it should be done without changing the model itself.

That’s RAG.

Now let’s investigate RAG deeper:

What RAG actually is

Retrieval-Augmented Generation is an architectural pattern, not a product. The concept is straightforward, and the best analogy is an open-book exam.

A student taking a closed-book exam relies on what they have memorized.

That’s a standard LLM. It’s smart and can reason, but it’s limited to what’s in its parameters. A student with an open-book exam has the same reasoning skills. They can check relevant pages before answering.

That’s RAG.

The formal definition comes from a 2020 paper by Patrick Lewis and his team at Facebook AI Research and University College London. RAG models combine parametric memory8 with a pre-trained language model and non-parametric memory9. This non-parametric memory uses an external knowledge index. It’s accessed by a neural retriever10.

You can still update the model’s knowledge by swapping the retrieval index. No one needs to retrain.

In practice, RAG follows a three-step loop:

1. Query comes in: a user asks a question.

2. Retrieve: System looks in an external knowledge base for the best information chunks.

3. Question and context go to the LLM. It then creates an answer based on the retrieved documents.

You’re not changing the model. You’re changing what it sees. That distinction makes RAG so powerful and so practical.

The open-book analogy makes sense at a high level. But understanding why it works so well means diving deeper into the topic…

How RAG Works Under the Hood

The three-step loop sounds simple. The engineering that makes it work is where things get interesting.

A RAG system has two main parts:

• An offline ingestion pipeline that gets your data ready.

• An online retrieval pipeline that answers queries.

Let’s walk through each one…

Embeddings: Core Mechanism

Before anything else, you need to understand embeddings11. This is the concept that enables semantic search12.

Text embeddings turn text into dense numerical vectors13. These are arrays of floating-point numbers, usually with 1,536 or 3,072 dimensions. They capture the text meaning. The magic is in what “capture meaning” means. Words and sentences with similar intent are close together in vector space. This happens even with different words.

Consider this: “How do I reset my password?” and “I can’t log into my account” use completely different words.

But when converted to embeddings, they produce nearly identical vectors. The distance between them is tiny because their meanings are similar. This is measured using cosine similarity14, dot product, or Euclidean distance.

The key insight for RAG is clear.

System finds relevant content. It does this even if the user’s question doesn’t use the exact words from the source document. It searches by meaning, not by keywords.

Data Ingestion Pipeline (Offline Phase)

First, process your knowledge base. Then, index it. Only then can your RAG system answer questions.

This happens in five steps:

What triggers the offline phase?

In traditional workflows, this pipeline runs when new data is available. This can happen when documents are added or updated. It can also occur when a database changes or on a regular schedule, like nightly or weekly. Some teams trigger re-ingestion if retrieval quality drops.

They also do this when a new data source connects:

1. Load documents from anywhere: PDFs, databases, APIs, wikis, Slack channels, and Confluence pages. Frameworks like LangChain15 and LlamaIndex16 offer ready-made connectors for many common sources.

2. Chunking17: Split documents into semantically meaningful pieces. This is the single highest-leverage step to get right, and we will cover it in depth later.

3. Embedding: Convert each chunk into a vector using an embedding model.

4. Storage: Store the vectors in a vector database18 optimized for similarity search.

5. Metadata tagging: Tag each chunk with source, timestamp, category, and access control information. This metadata becomes critical for filtering, attribution, and security later.

Retrieval Pipeline (Online Phase)

When a user asks a question, the retrieval pipeline kicks in:

1. Query embedding: User’s question turns into a vector. This uses the same embedding model from ingestion. This is critical: the query and the documents must live in the same vector space.

2. Similarity search: The system finds the top-K19 similar chunks. It compares the query vector to each vector in the database.

3. Retrieval strategy: This is where the real engineering decisions happen. Three primary approaches exist:

   1. Sparse retrieval20 uses a statistical method that matches exact keywords. It weighs these matches using term frequency and inverse document frequency.

   2. Dense retrieval (embeddings): Semantic search via vectors. Finds conceptually relevant content even when the wording differs.

   3. Hybrid search (combining both): Use both sparse and dense retrieval. Then, merge the results with Reciprocal Rank Fusion21 (RRF). This boosts documents that rank high in both systems.

4. Re-ranking: After retrieving results, a cross-encoder model rescans them. It processes the query and each document as one input. This captures fine-grained relevance that bi-encoders miss.

Generation Phase

With the most relevant chunks in hand, the system assembles the final prompt:

1. Prompt Construction:

   1. System Prompt: Combine the system prompt (the instructions that tell the LLM how to behave, like “You are a helpful customer support agent”) with the user’s request.

   2. Retrieved Context Chunks: Integrate the relevant pieces of text pulled from your knowledge base. For instance, if someone asks, “What’s your refund policy?” You might pull two paragraphs from your company’s policy document.

   3. User’s Original Query: Include the user’s actual question or task — exactly as they typed it.

2. LLM Call: The model generates an answer grounded in the retrieved documents. The key facts are in the prompt. The model uses these facts to reason instead of relying on its trained memory. Think of it like giving someone an open-book exam instead of asking them to answer from memory.

3. Citation and Attribution: The system shows which source documents were used and provides verifiable citations. For example, the response might say, “Based on Section 3.2 of the Employee Handbook...” so the user knows exactly where the answer came from. This is one of RAG’s biggest advantages over fine-tuning: transparency.

Now that you know how RAG works mechanically, the real question is whether it’s the right tool. That depends on what you’re comparing it to…

Why RAG over alternatives?

RAG isn’t the only approach to giving AI access to knowledge.

You need to know when to use it and when not to. This means comparing it to other options.

RAG vs. Fine-Tuning

This is the comparison most teams face first. Here’s how they stack up:

The key nuance: fine-tuning and RAG are complementary, not competing.

Fine-tuning is the best option when you want to adjust the model’s behavior, style, or output format, not just its knowledge. The best production systems do two things: they fine-tune style and behavior, and then they use RAG for knowledge. You can adjust a model to produce structured JSON in a certain format.

Then, use RAG to fill it with up-to-date data.

RAG vs. Long Context Windows

Context windows keep growing. Why not dump everything in the prompt?

Cost and precision. RAG retrieves only the 5-20 most relevant chunks. Long context stuffs everything in and hopes the model finds the needle. RAG is cheaper per query. It’s also more precise in what it finds. Plus, it can search millions of documents, unlike others that have a token limit.

They work well together: first, use RAG to get the right 20 chunks.

Then, use long context to process them all at once. The “lost in the middle” research shows that retrieving small, relevant chunks works better than using large context windows.

Reminder: this is a teaser of the subscriber-only newsletter series, exclusive to my golden members.

When you upgrade, you’ll get:

• High-level architecture of real-world systems.

• Deep dive into how popular real-world systems actually work.

• How real-world systems handle scale, reliability, and performance.

Subscribe now

Traditional RAG vs. Agentic RAG

Traditional RAG follows the straightforward three-step loop described above: retrieve, then generate. It works well for simple question-answering but has limitations. What happens when the first retrieval doesn’t return good results? Traditional RAG just pushes forward with whatever it finds.

Agentic RAG adds a reasoning layer on top.

An AI agent decides how to handle each query. It can change the search, link several retrievals, pick a knowledge base, or skip retrieval completely. Think of Traditional RAG as a student who looks up one page and writes their answer. Agentic RAG is like a student. They check several sources. They re-read sections that seem off and cross-reference information before writing.

Multi-Step RAG Pipeline

Production RAG systems often go beyond the simple retrieve-and-generate loop. A multi-step pipeline adds intelligence before and after retrieval:

1. Query Intent Parsing: Before searching, the system analyzes what the user actually wants. Is it a factual question? A comparison? A request for a summary? Understanding intent helps the system choose the right retrieval strategy and knowledge base.

2. Query Reformulation: The system may rewrite the user’s question to improve retrieval quality. For example, “Why is my app slow?” might become “application performance bottleneck causes and solutions.”

3. Retrieval: System searches for relevant chunks (as described above).

4. Live Web Search: If internal documents aren’t enough, some RAG systems can do live web searches. This helps them get current information from the internet.

5. Reranking22 and Filtering: Results are scored, filtered, and reranked for relevance.

6. Generation: LLM produces an answer grounded in all the gathered context.

This multi-step approach is what separates demo-quality RAG from production-quality RAG.

RAG Limitations

RAG is powerful, but it’s not a silver bullet…

Understanding its limitations helps you decide when to use it and when to look elsewhere:

Read more

• Share this post & I'll send you some rewards for the referrals.

Everyone is talking about AI agents, but only very few people are actually building them.

Plugging a chatbot into your website is just a UI update. The biggest opportunity is in agentic AI, and most people are missing it…

Onward.

The best way to build any app (Partner)

The real tax on vibe builders right now isn’t effort, it’s cost.

Every tool wants you to pay for its model usage and hosting forever. Orchids.app lets you bring your own and use tools/SDKs that already exist elsewhere.

Plug in your ChatGPT, Claude Code, Gemini, Copilot, GLM, or any API key you already pay for.

You keep control of the bill. And the best part is: you don’t get price locked because you shipped with Orchids. Deploy your code straight to Vercel with one click.

Get Started Today

(Use this discount code to get a onetime 15% off during checkout: MARCH15)

I want to introduce Fran Soto as a guest author.

Fran is a software engineer at Amazon. He’s responsible for most of the “Add to Cart” and other buttons you see in the search results of the Amazon marketplace, and now works at Ring.

Fran helps software engineers grow their careers. He teaches them productivity and the AI automation skills they need to grow to the next level.

To celebrate this collaboration, Fran is giving away his premium System Design Template completely free to all new subscribers. This template is the perfect tool to structure your architectural decisions before you write any code. You can practice filling it with the case studies and deep dives from this newsletter!

For the next few days, you can also claim a 25% discount on the StrategizeYourCareer annual subscription from the links in this newsletter. Whether you grab the annual discount or just want the free template, join today to become a productive engineer.

Check out his newsletter: StrategizeYourCareer.

Imagine you have just bought some furniture and now you have to assemble it…

The fixed set of instructions that comes in the box is a workflow. It is a linear path that works perfectly as long as everything goes right. It’s the equivalent of scripts and programming languages. But if you make a mistake and break a wooden part, the instructions cannot help you.

There is no recovery from that.

An LLM would be like having a customer support line available for you to call.

You can describe the broken piece, and they will give you a brilliant, empathetic answer. They might even explain why the wood snapped. But at the end of the call, you are still sitting on the floor with a broken piece. No matter how nice the other person is, they don’t have hands to help you with your broken piece.

You are the agent in this scenario.

You are the one with the tools, like a hammer, nails, glue, and a drill. You are the one with the brain to make decisions on how to best use those tools and how to recover from errors that were never in the initial instructions.

In this newsletter, I want to show you how to build AI agents, providing the tools and knowledge they need to do work on your behalf.

I will use an incident response agent as an example, but the principles of giving an AI both a brain and a set of hands apply to almost any process.

In this newsletter, you’ll learn:

• The differences between LLMs, workflows, and agents.

• Why the non-deterministic nature of AI is a good feature.

• A 10-step framework to evolve from manual work to autonomous agents.

• How and when to keep the human in the loop.

Let’s get started:

Why the lack of determinism in AI is good for reliability and self-healing

I see a lot of skepticism about AI because it is NOT as reliable as traditional code.

We are used to code that does exactly the same thing every time. However, I believe this lack of determinism is actually a good thing for certain tasks.

Humans are not as reliable as a machine, yet we are the ones who fix machines when they break. We prefer humans using machines because it brings the best of both worlds. Machines give us the speed and precision of determinism, while humans make things not go as expected.

An agent mimics this human ability to adapt to new situations. When a system fails in a way you did not anticipate, an agent can review the logs and adjust its approach.

It’s self-healing.

However, to get the most out of agents, instead of using the out-of-the-box generic tool, you need to create your own agents tailored to your use case.

How to build your first AI agent

AI companies provide us with generic agents that have planning capabilities and built-in tools. While these are useful, I do not think we want a generic assistant for specialized tasks.

They also offer integration points to our own memories and tools.

This is where the opportunity is. The same way a company would hire a software engineer rather than a “generic human”, we want that same level of expertise in our agents.

Building an agent is not a one-step process where you just flip a switch and everything works perfectly. It is an iterative process in which you gradually offload parts of your cognitive load to the machine while keeping a close eye on its performance.

I’ve followed this 10-step framework that starts with pure manual work and ends with a fleet of autonomous agents working together…

(This process starts with the way we have always worked: Solving it manually.)

Step 1. Do the work manually

The first step is always to do the work yourself…

I recommend outlining the steps as if you were documenting the runbook for someone else to follow:

1. You open the monitoring dashboard and check any metric outside the usual pattern.

2. You check the runbooks for troubleshooting info for that specific metric.

3. You query the error logs for the last 15 minutes, looking for requests that resulted in an HTTP 500 error.

Current state: At this stage, you are the bottleneck in the entire process. Every step requires your manual intervention and your attention. This is the baseline that we want to improve.

Step 2. Start using an LLM

Now that you have a solid manual process, you can use it as a guide for an LLM.

This moves us from manual labor to having an assistant who can analyze the data for you. You can prompt the AI by copy-pasting the logs from your terminal or the metrics from your dashboard.

I suggest paying close attention to the prompts you usually use.

• “Review these logs to find what error is happening: <copy-paste of logs>”

At this point, it’s important to notice the areas where the AI does not do what you want, and how you steer it back to the right path when it goes off track.

Current state: The AI can start analyzing patterns, but you still have to orchestrate everything and move data around manually.

Step 3. Add MCP Tools

At this stage, you have an AI partner helping with the analysis, but you are still stuck in the middle, acting as a bridge.

The next step is to stop copy-pasting data. You can use the Model Context Protocol (MCP1) to allow the AI to take actions for you. MCP is like HTTP for AI agents. It’s an API that enables the model to execute operations in other systems.

Once you connect to the right MCP servers and enable the MCP tools, your prompts will look like this:

• “Obtain the alarms that fired in the last 15 minutes. You must give me only the alarm name.”

• “Check the logs for requests ending with an HTTP 500 error. You must give me the message of the log, the error message, and the stack trace...”

The AI can get the incident details on its own!

Current state: The human can drive the entire process through the AI interface without switching between tabs and tools.

Step 4. Add Agent Skills

Up to this point, you still have to be an expert in incident response…

AI is useful for doing everything from one place. Now we’ll start looking at the notes we have been taking on how to improve the process and where AI usually goes wrong.

We have to onboard the AI to incident response the same way we’d onboard a junior engineer recently hired:

• Teach exactly how to use each tool.

• Teach how you want the work done.

We can combine guides and Standard Operating Procedures.

These provide step-by-step instructions for routine tasks. We’d write them in human language, using small scripts, to address a problem with determinism.

For example, if the MCP tools require a query as input, we can create the Skill of building a query. We’ll combine a markdown guide containing the fields from the logs with a few scripts to build queries from a template. This is much better than having the AI write complex queries every time. Plus, it reduces the hallucinations of fields that don’t exist in your logs.

We call these reusable pieces of logic Agent Skills.

Agent Skills are great because they’re loaded into the model ONLY when needed. The agent may have skills for metrics and logs. If we only ask about fired alarms, the agent ignores the log data. It just doesn’t need it.

This saves memory.

With agent skills, our interaction would be like this (Slash is the syntax used in IDEs like Cursor2 to force the agent to load a particular skill):

• /Metrics-and-alarm-management Find the alarms fired

• <LLM response with metrics information>

• /My-service-logs-management Find the logs with errors in these endpoints

• <LLM response with the logs information>

• /Code-expert Where in code is this log emitted?

• <LLM response with the code information>

Current state: We have now taught AI how we want certain tasks done, such as metrics and log information. The human is orchestrating the process to do things in order.

Step 5. Add memory

We have given our agent tools and domain expertise, but every time you start a new chat, it’s like the agent has amnesia. It knows how to solve an incident, but it doesn’t remember the one you fixed yesterday.

An agent needs context to be effective…

You can add simple short-term memory and long-term memory files to your system.

The short-term memory is built by design into the agent. This is essentially the AI’s ability to remember the previous messages in your current conversation. It helps it stay on track without you having to repeat yourself. When making an API call to get the model’s response, it sends the previous messages along with the new ones.

For an incident response agent, we also want some long-term memory.

We have added some memory thanks to the skills, but we can add more context by creating files that the AI can access.

Some time ago, each company had its own format, but now they have agreed on a standard called Agents.md3. This means the information inside Agents.md file will be read by the AI regardless of which AI it is.

Also, each implementation allows you to point to which folders/files to use for the agent’s context.

Current state: We still prompt AI the same way. It uses data from skills and logs. It also reads files to understand the system it is debugging. This helps the AI find past errors and the solutions that worked.

Step 6. Orchestrate with Agent SOPs

Up to this point, we have given our agent a brain, a set of hands through MCP tools, and a memory.

But you are still the one directing every move. While the agent is now more capable, it still requires you to decide manually what to use and when. This is the bottleneck that we solve with Agent SOPs.

Instead of prompting the AI to use a series of skills one by one, you can create a step-by-step process for the AI to follow. This is what’s called a Standard Operating Procedure (SOP) for your agent.

An Agent SOP is a file written in accordance with RFC 21194. This RFC defines key capitalized words (MUST, SHOULD, MAY, etc.) to specify requirement levels. Besides, it contains the steps in order.

I learned about this concept at work, as Amazon uses it in its Kiro5 and Strands6 agents. However, I’m also using Agent SOPs with Cursor and Claude Code7 easily. You just have to write the markdown file as an Agent Skill.

Now, instead of prompting step by step, we just execute our agent with that SOP. For example, if we wrap our SOP in a skill:

• /Incident-response-agent-sop Investigate the incident reported at JIRA-1234

Current state: This makes the agent much more autonomous. The human just invokes the right SOP, and the agent handles the rest. It’s like a high-level manager finding the right engineer to loop into a call.

Step 7. Define an agent

At this point, we have many MCP tools, skills agent SOPs that need to be configured and made available for our agent to work.

This is easy if this is your first agent, but when you have dozens of them, each using their own set of artifacts, things get hard…

We solved this problem in traditional software with dependencies. We declare which packages our software depends on, so it can run in different environments.

Now it’s time to declare those dependencies with a new artifact called an agent. This way, we’d have our incident response agent, and we can define other agents for other purposes.

This guarantees the AI has all the tools it needs, so you do not need to provide them manually. Currently, there isn’t a proper package manager to install all of these, but it’s just a matter of time…

Current state: We now have our agent, and we can delegate the entire incident response investigation to it. We just invoke it and wait for the results.

Step 8. Execute your agent periodically

Up to this point, we have a fully defined agent that knows exactly what to do, but it only moves when you tell it to.

You have built a great tool, but you are still the one holding the power button…

Now that we can delegate most work to AI, you may ask: Why am I still getting paged at 3 am just to invoke the agent and wait?

It’s time to set the AI agent to run autonomously…

I recommend using CLI tools to run your AI agent independently. We can start with a simple script that checks for work and runs the AI agent when found; otherwise, sleep for N minutes.

Current state: Now the human isn’t involved in triggering the agent, but it will find the alarm and the agent’s output together.

Step 9. Improve state management

The agent is now running on its own, but it’s still isolated on your local machine using basic loops. To make it a true part of your team, it needs to be integrated into your production environment…

The goal is to define an end-to-end workflow that triggers AI agents at the right time without human involvement. For example, a webhook from PagerDuty could automatically trigger the AI agent in Lambda.

For periodic reports, you may want to set up a cron8 job that triggers the AI agent at regular intervals.

The AI agent performs the investigation and then posts a summary of the root cause or a periodic report to a Slack channel. This means your agent is working while you sleep, even if it only does one specific thing well.

Current state: Now our agent is like a real engineer involved when an incident happens, or an engineer who has scheduled work in their calendar.

Step 10. Scale to multiple agents

At this stage, you have a specialized agent working 24/7.

But as your system grows, one agent can become overloaded with too many responsibilities. Just as you wouldn’t ask one engineer to handle every single microservice, we shouldn’t ask one agent to do it all.

The final step is to have multiple agents collaborating on the same incident. Each agent can have a different function based on its expertise:

• One agent gathers the raw logs while another checks for known infrastructure issues.

• Another agent could review the recent code deployments for errors.

There are different orchestration paradigms you can use, like sequential, routing, or swarms.

Even for the same function, there’s no limit to how much you specialize. If an incident response agent has too many details, you can create agents to check different parts of the system.

You need a bit of abstraction to manage your AI workforce, just as a manager coordinates a team of specialists. You just do the same: multiple specialist agents, and a manager that invokes each subagent.

You can use different triggers, such as schedules or events, to make agents collaborate via the file system. No need to start with complex agent-communication protocols; instead:

1. Start with an orchestrator agent triggered by events, schedule, or polling.

2. The orchestrator invokes other subagents that you have also defined. Most AI companies already implement subagents.

3. All agents write to the filesystem, so they see each other’s work.

I suggest running agents on your own computer and letting them work together by sharing files.

Current state: At this point, the AI performs complex work with multiple steps without your involvement. Humans are no longer the bottleneck.

We did it!

Now we have our autonomous AI agent for incident response.

I know it looks easier in diagrams than in practice, and the AI doesn’t work as expected. Let me also share some lessons from building AI agents.

Lessons from the trenches

I’ve spent a lot of time building these agents, and I want to share some practical advice that will save you hours of frustration:

• Make sure you have enough examples to build a robust agent. You wouldn’t want to automate a task that isn't frequent, so you're likely to have multiple instances to iterate on. I asked my team to handle all intakes of a service for a month. I’m sure your manager will be happy you're taking on more work.

• Iterate multiple times per step. Don’t move to the next step of the automation until you have solved several tickets or tasks to find the real patterns and failure modes.

• Don’t switch tabs while AI works. Watch the agent work live to catch its mistakes instead of letting it run in the background.

• Use tmux9 for persistent loops. Run your agents from your computer in a tmux session so they keep working even if you lose your terminal connection.

• Log everything for observability. Pipe all the agent’s output into a dedicated log file to give you the same level of visibility when you move to executing it 24/7.

• Refine prompts based on observations. Even after you consider the 10 steps done, new edge cases will arise. Make AI keep learning, either by updating it yourself or by letting the AI update its memory files between iterations.

Defining the role of the human in the loop

I want to be clear that this is not about all-human or all-AI systems…

I’ve focused the incident response agent examples on read-only tasks, such as checking alarms, metrics, and logs.

Each of us has to define where and when to involve humans for the highest leverage. You can have the agent triage the incident and suggest a fix, then wait for your approval and execution.

Maybe someone else wants to plug the AI into the codebase to create a PR with the fix and wait for a human to approve it and deploy the hotfix.

I still think we don’t want AI making changes in production without human oversight. I have read postmortems where AI ran unauthorized production operations in an AWS account without any human review.

If we think about the incident response process and a code change, there are many steps before we change something in the system…

My current stance on agents is to push the human to the right as much as possible. This is all about making the human do less work, but still overseeing. This is important because no one will ask the AI for ownership or accountability.

They’ll ask the humans for it!