• Share this post & I'll send you some rewards for the referrals.

I’ve built agents at Google to support their machine learning operations.

Then one thing became super clear to me: AI engineering is good software engineering with an understanding of a few key AI principles.

Just a year ago, hardly any software engineers were building with AI.

Now, engineers are expected to understand AI engineering and integrate AI into their products. This newsletter is a primer to bridge the gap from software to AI engineering.

Onward.

The tax you pay to run multiple agents (Partner)

If you’ve spent any time with coding agents, you know the feeling.

You start the morning with a clean plan. Spin up a few agents. One is refactoring the auth module. Another is writing tests. A third is scaffolding a new API endpoint. You’re flying.

Then, around 10:30 AM, you look up and realize you have 20 terminal windows open. One agent is blocked waiting for a decision you forgot to make. Another finished 40 minutes ago, and you never noticed. A third went sideways three commits back. You’re no longer flying. You’re drowning.

You’ve shifted from human as driver to human as director.

When running coding agents in parallel, the bottleneck isn’t just context. It’s your own attention trying to manage 10 agents across 10 terminals. You’re losing your mind to terminal chaos.

Meet Cline Kanban, a CLI-agnostic visual orchestration layer that makes multi-agent workflows usable across providers. Multiple agents, one UI. It’s the air traffic controller for the agents you’re already running, regardless of where they live.

• Interoperable: Claude Code and Codex compatible, with more coming soon.

• Full Visibility: Confidently run multiple agents and work through your backlog faster.

• Smart Triage: See which agents are blocked or in review and jump in to unblock them.

• Chain Tasks: Set dependencies so Agent B won’t start until Agent A is complete.

• Familiar UI: Everything in a single Kanban view.

Stop tracking agents and start directing them. Get a meaningful edge with the beta release.

Install Cline Kanban Today: npm i -g cline

Get Started Today

(Thanks to Cline for partnering on this post.)

I want to introduce Logan Thorneloe as the guest author.

He’s a software engineer at Google, building at the intersection of machine learning infrastructure, AI agents, and developer tooling.

He uses his background in machine learning research to teach software engineers the AI topics they should know via his newsletter, AI for Software Engineers.

You can also get an exclusive 25% off a paid AI for Software Engineers subscription as a reader of the System Design Newsletter.

1 Software 1.0, 2.0, and 3.0

“The hottest new programming language is English.” - Andrej Karpathy

There have been two recent AI-related shifts that have fundamentally changed software development and required software engineers to learn new concepts.

Both changes have been caused by the development of machine learning models: programs trained on data to recognize patterns and make predictions rather than follow explicit rules.

Andrej Karpathy details these changes by describing three different eras of software engineering:

• Software 1.0: hand-written logic via writing code. This was the paradigm from the inception of programming all the way to the advent of machine learning.

• Software 2.0: shift from explicitly writing logic via code to curating datasets and teaching algorithms to understand them. This shifts logic from a set of instructions to logic learned from data. A good example of this is Tesla’s Autopilot system, shifting from hand-written C++ code to neural networks1 for vision.

• Software 3.0: shift toward natural language2 being our new programming interface. This enables engineers to write code by speaking English (vibe coding). It also enables engineers to build agents3, or software that can think and act autonomously.

Software engineers need to understand these key AI principles to be productive in the software 3.0 paradigm…

2 Large Language Models (LLMs)

LLMs are natural language AI models trained to predict the continuation of text given a natural language input.

As with any AI model, understanding how it works best comes from examining its training data. LLMs are trained on a massive dataset of text from the internet. They’re trained to learn the relationship between words to best predict an output when given a specific string of inputs.

This relationship enables LLMs to use the knowledge stored in the semantics of language to understand relationships between concepts.

LLMs are also tuned to be non-deterministic; i.e., to provide different answers when an input is sent to the LLM more than once. This ensures an LLM’s output isn’t locked to a given input.

These two concepts make LLMs intelligent.

To relate LLMs to general software engineering, we can think of an LLM as a “non-deterministic database”.

This makes it easier to understand the impact of the knowledge encoded in its weights via semantic understanding. I like to compare it to a SQL database and a query: LLM is the database, and the prompt given to the LLM is the query that extracts the information in the format the user wants.

In AI engineering, it’s apt to think of an LLM as the central processing unit (CPU) of your application. By giving an LLM input information, it can make decisions on its own. Thus, the LLM becomes the CPU for your software, deciding what to do with the information provided.

This is also an apt comparison because choosing an LLM is like picking a CPU.

Each LLM has its own tradeoffs for latency, cost, functionality, and more. Choosing the right model (or models!) is important to meet the requirements of production software systems.

3 Context

Context is the information LLMs store in their memory and functions as the working memory for a task. For an LLM to perform a task or make a decision, it must have the relevant information in its context window4.

Context works by converting all input into tokens, which are small chunks of text (roughly 3-4 characters each) and feeding them into a model sequentially. The model uses an attention mechanism (a process that calculates how each token relates to every other token in the sequence) to determine which tokens are relevant to each other.

This is how LLMs ‘understand’ the relationships within the input mentioned above.

The tokens currently part of a conversation with an LLM are considered part of that LLM’s “context window”. Each time you interact with an LLM, the model processes the entire context window from scratch. It doesn’t remember the previous turn unless you feed the entire conversation history back in.

Similar to RAM, an LLM’s context window is finite…

There is a hard limit to the number of tokens that can be stored in context at a time. Even before hitting that limit, LLMs tend to bias their knowledge toward the start and end of their context window. This means an LLM’s understanding of the information it is given can degrade before it reaches the hard limit.

The size of an LLM’s context window varies between models.

We’re at a point where frontier models can process 1 million or more tokens, whereas many open models are closer to 256,000 tokens. This isn’t enough for many applications, so it’s important to design AI systems with context limitations in mind.

Also, like RAM, an LLM’s context doesn’t persist between sessions.

LLMs will only remember the context provided during a session. This means AI systems with long-term memory require a more persistent memory storage and retrieval system.

Context is managed by only keeping the tokens a model needs for a task.

To effectively manage context, consider:

• Using subagents for specific tasks. A separate agent means a separate context window. That agent can process and summarize the context, then return it to our primary agent.

• Summarizing all information currently within an LLM’s context window, clearing the context window, and adding the summary back in. This stores the same information in a shorter context.

• Using external systems to store information. Information is only needed in context when it’s being used. It should be stored in a separate, accessible datastore otherwise.

Every AI application built with LLMs needs to properly manage context.

4 Embeddings and Vector Databases

You can’t fit everything into a model’s context window, so you need a way to find relevant information efficiently.

An embedding represents the meaning of data as a list of numbers, called a vector. An embedding can be created from text, audio, video, or any other data format to convey the meaning of that data. Items with similar embedding vectors are considered similar.

Consider a coordinate system in which each axis corresponds to a learned semantic dimension.

“Deploy a microservice” and “ship a container to production” would be plotted close together. “Deploy a microservice” and “quarterly revenue forecast” would be far apart. Embeddings can do this across thousands of dimensions. Distance in this high-dimensional space equals similarity in meaning.

Vector databases5 are built explicitly to store and query embeddings.

They use algorithms optimized for quickly finding the most similar vectors in the database. Those vectors are then added to a model’s context. This is comparable to retrieving data from a cache.

The process follows these steps:

1. Index: Convert documents into embeddings, store in a vector database

2. Retrieve: Embed the query and find the most similar documents

3. Augment: Pass those documents into the LLM’s context window with the user’s question

4. Generate: LLM answers using retrieved information

Almost every production AI engineering system will need a working memory solution.

Context will need to be augmented with a persistent data store. Embeddings and vector databases.

5 Time to First Token

There are many factors to consider when choosing a specific LLM for your production AI system. One of these requirements is latency: your system must function fast enough to be useful.

AI systems must consider the key latency metrics of traditional software systems and the LLM latency metrics, such as:

• Time to First Token (TTFT): How long until the model starts responding. This determines perceived responsiveness. Smaller models respond in under a second, but larger models can take several seconds to generate anything.

• Tokens Per Second (Throughput): How fast the model generates after starting. This varies significantly across model sizes and compute units, and also affects how quickly the model's response is output to the user.

AI applications require concurrency to stream the output of an LLM call and reduce perceived latency.

Latency is an incredibly important metric to understand the real-world functionality of an AI system.

This is like streaming services, which play the start of a video while loading the rest in the background. This enables the viewer to watch the video before the entire video loads.

6 Evals

“Evals are a new toolset for any and all AI engineers, and software engineers should also know about them. Move from guesswork to a systematic engineering process for improving AI quality.” - Pragmatic Engineer, December 2025

In traditional software, tests are written to ensure code function reliability.

In AI, traditional tests are insufficient for many parts of the system because of their non-deterministic nature. Instead, we write evaluations (evals for short).

Since assert statements won’t suffice for non-deterministic applications, evals are written using LLM-as-a-judge6.

Essentially, a dataset of inputs and their successful outputs is curated.

This dataset will be used to tell the judge what we’re looking for in our model's output. The LLM-as-a-judge will score how closely the tested model’s outputs resemble the successful output based on given criteria such as relevance, hallucinations, and accuracy.

For example, an eval can be used to assess whether an LLM effectively pulls the important information from a document when summarizing. A set of documents and summarization pairs is gathered as a solution set. The documents are run through the LLM we’re testing, and our LLM-as-a-judge compares the solution for that document to our LLM’s output.

Evals are vital because even the smallest changes in a non-deterministic system can change the user's output. The most common of these changes is model tweaks. Even a small tweak in model performance can drastically change the output of a system.

Evals are written to catch these changes.

7 Agent Loops

Agent loops7 represent the shift from Software 2.0 to Software 3.0.

Software 1.0 to 2.0 was the change from logical control flow to probabilistic output. In the shift to Software 3.0, we see probabilistic output used as reasoning loops in an agentic system8.

Agent loops enable an LLM to continuously receive and act on information. An agent loop looks something like:

1. Thought: Model reasons about the current state and what it needs to do next

2. Action: The model decides to call a tool

3. Observation: Application executes the tool and returns the result

4. Repeat: Model then reasons about the observation, takes another action, or provides a final answer

This is the ReAct pattern (Reason + Act), and it has become the default agent loop architecture.

ReAct merges Chain of Thought reasoning, where the model works through a problem step by step, with tool execution into a single loop. This means a model can not only reason to act, but also take its previous actions and their outputs into account, continuously acting until a task is complete.

For example, consider an AI travel agent tasked with booking a trip within a budget.

It first searches for flights and finds the cheapest option. It then reasons about how much budget remains and searches for hotels accordingly. The initial results are all outside the city center, so it adjusts its search criteria and finds a better option. It calculates the total, confirms it fits the budget, and presents the full itinerary.

An agent loop relies on the LLM to determine the next step continuously until it completes a task. This looping process enables an LLM to use its knowledge and context to do meaningful work.

This enables complex workflows that aren’t possible with written code alone.

8 Tool Calling

Meaningful work is further enabled by tool calling9 capabilities.

This gives the LLM the ability to do something as part of its action. While LLMs can always output natural language, tool calling enables that output to call functions that perform work.

LLMs cannot execute code, access the internet, or perform other work natively. Instead, tool calling must be enabled:

Read more

• Share this post & I'll send you some rewards for the referrals.

I have a big big announcement for those of you who want to become good at AI engineering.

INTRODUCING: GENERATIVE AI MASTERCLASS

This is a monthly newsletter series that’ll teach you Generative AI system design.

By the end of this newsletter series, you’ll walk away with:

• Simple breakdown of real-world architectures

• Frameworks you can plug into your work or business

• Proven systems behind ChatGPT, Perplexity, and Copilot

And here’s the best part:

You’ll go from knowing how to “use AI” to understanding how to design, build, and ship AI.

No AI/ML background needed. Each newsletter explains concepts in plain English, with visuals and real product examples.

If you want the maximum AI career leverage, join below:

Subscribe now

New Open Source: Build and Scale Agents and Harnesses Like APIs (Partner)

One.harness() call and orchestrate hundreds of Claude Code, Codex, and Gemini instances into a coordinated team. They discover each other at runtime, split work, cross-reference results, and converge autonomously. Go deeper on harness engineering → blog.

No DAGs. No glue code. Agents work like APIs, Python, TypeScript & Go.

We built production systems you can clone and run today:

→ Autonomous Engineering Team: 175 agents ship code end-to-end

→ Deep Security Auditor: 250 agents scan every CVE

→ Adversarial Code Reviewer: 50+ agents debate your PRs

Each runs on any model. Apache 2.0.

Clone a Recipe

We’ve all used ChatGPT.

From our user perspective, it looks simple: type a question, get a response. But there’s a lot more behind it than a large language model (LLM) with a chat interface.

Building a product like ChatGPT means choosing the right model from a growing landscape of options, preparing the data that trains it, running a multi-stage training pipeline, evaluating whether the outputs are actually good, adding safety controls to prevent harm, and wrapping it all in serving infrastructure that can handle millions of requests. The same is true for Perplexity, GitHub Copilot, and Midjourney.

The model is one piece of a much larger system.

Understanding these components is what separates someone who can use GenAI from someone who can design and build it. That’s what this newsletter series is for.

This first newsletter covers the foundation.

We walk through the model landscape, the data layer, the training pipeline, evaluation, safety, and the engineering that turns a model into a product.

Let’s start with a glossary of key terms that come up throughout this newsletter and the rest of the series. If you’re already comfortable with terms like tokens, embeddings, and context windows, skip ahead to Part 1: The AI Model Landscape.

I want to introduce Louis-François Bouchard as the author of this newsletter.

He’s a best-selling author (Building LLMs for Production), the co-founder of Towards AI, and the creator of the YouTube Channel, What’s AI, where he helps people understand AI and learn how to apply it in the real world. Through his development works with clients and his content, teaching, and AI training programs on the Towards AI Academy, Louis focuses on making AI practical for builders, engineers, and curious learners alike.

At Towards AI, he and his team train AI engineers through courses built for every stage, from beginner to advanced. That educational mission and the real-world experience building for his clients are exactly why I wanted him in this newsletter series.

Key Terms

You don’t need to learn these upfront.

Skim to get familiar, then refer back whenever you hit an unfamiliar term:

Token: LLMs work with numbers, not words. Before processing any text, the model breaks it into small chunks, called tokens. A token can be a whole word (“the”), part of a word (“un”, “believ”, “able”), or punctuation. One token is roughly ≈ 4 English characters, though this varies by language and model. API pricing, context limits, and cost are all measured in tokens.

Embedding: A token ID alone (like 1023 for “What”) tells the model nothing about meaning. An embedding replaces that ID with a list of numbers (a vector) that captures what the word means and how it relates to other words. These vectors are what the model actually works with internally, processing and transforming them at every step to build up an understanding of the full input. Words with similar meaning end up with similar vectors: “king” and “queen” are close together, “king” and “banana” are far apart. This same property also powers search in GenAI products: when a user asks “How do I reset my password?”, the system can find documents on password resets, account recovery, and login issues, even if those documents never use that exact phrase.

Parameter: These are internal variables inside a model, billions of them. At the start of training, they’re random. Each time the model makes an incorrect prediction, settings are nudged slightly. After trillions of such adjustments, they encode language patterns. GPT-3 had 175B parameters. Llama 4 comes in Scout (17B active) and Maverick (17B active). More parameters mean more capability, but also more memory: a large model can need 140GB or more just to load.

Context Window: The maximum amount of text (in tokens) a model can see in one request. GPT-5 supports up to 400K tokens (272K input). Claude and Gemini support up to 1M tokens. Everything the model needs, system instructions, conversation history, retrieved documents, and the user’s question must fit within this window. Deciding what goes in and what gets cut is called context engineering.

Inference: Every time someone sends a message to ChatGPT, the model runs to produce a response. That’s one inference call. At scale, this can mean millions of calls per day, and the cost adds up. Two key metrics define inference performance: Time to First Token (TTFT), the time the user waits before the first word appears, and Tokens per Second (TPS), the rate at which the rest of the response streams in after that.

LLM (Large Language Model): A neural network (a program that learns patterns from data) trained on one objective: to predict the next token. GPT, Claude, Llama, and Gemini are all LLMs. The objective is simple, but at a massive scale (trillions of tokens of training data), it produces the ability to follow instructions, write code, reason through problems, and hold conversations.

Foundation Model: A large, general-purpose model trained on massive data that serves as the starting point for products. Both LLMs (GPT-5, Claude) and diffusion models (Stable Diffusion, Midjourney) are foundation models. These aren’t built from scratch for each product. You take one and adapt it through prompting, fine-tuning, or both.

RAG (Retrieval-Augmented Generation): A design approach where the system retrieves relevant documents at query time and includes them in the prompt, so the answer is grounded in real sources rather than what the model memorized during training. Products like Perplexity use this approach: search the web, feed the results to the model, and generate an answer with citations. It’s widely used in products that need up-to-date or domain-specific knowledge.

Fine-tuning: Taking a foundation model and training it further on a smaller, task-specific dataset to specialize its behavior. For example, a model fine-tuned on customer support conversations learns the right tone, format, and escalation patterns for that domain. Fine-tuning changes how the model responds (style, format, tone), not what it knows. If the model needs to answer questions using knowledge not in its training data, retrieval-augmented generation (RAG) is the standard approach.

Prompt: The full text input sent to a model for each request is a prompt. It typically includes a system prompt (developer-defined instructions that define the model’s role), the conversation history (previous messages), and the user message (the current question). In RAG systems, retrieved documents are also included. Prompt engineering is the practice of refining how this input is structured to get better results.

Hallucination: When a model generates confident but factually wrong output, like citing a paper that doesn’t exist or inventing a URL. This isn’t a bug. These models are optimized to produce text that sounds plausible rather than factually correct, so hallucinations are a built-in trade-off, not a malfunction. The main ways to reduce hallucinations are RAG (retrieving real sources to ground the answer) and evaluation (testing the model’s outputs for accuracy).

Transformer: The neural network architecture behind nearly all modern foundation models, introduced in 2017 (“Attention Is All You Need”). Its key innovation is the attention mechanism: instead of processing tokens sequentially, the model considers all tokens simultaneously and learns which are relevant to one another. In “The cat sat on the mat because it was tired,” attention connects “it” back to “cat” rather than “mat.” Some models extend this architecture with a Mixture of Experts (MoE), where only a subset of the model’s parameters is used for each input rather than all of them, making the model faster and cheaper to run while maintaining high overall capability.

GPU/TPU: The specialized hardware on which GenAI runs. GPUs (NVIDIA H100, A100, B200) handle the massive number of calculations required by models. A request that takes milliseconds on a GPU could take orders of magnitude longer on a regular CPU. Training a large model needs thousands of GPUs running for months. TPUs are Google’s custom chips for ML, used primarily within Google Cloud.

Latency: How long the user waits for a response. Different products have different targets depending on what the user is doing. Autocomplete has to keep up with typing, so it typically needs a response in under 300ms. Chat users expect a short pause before the reply starts streaming. Image and video generation is much slower; users wait 10 to 60 seconds. These targets influence which model to use, whether to add caching, and how much the system can do before it starts to feel slow.

The first decision in designing any GenAI system is choosing the right model…

That starts with understanding what kinds of models exist and what each one is good at.

Part 1: The AI Model Landscape

Not all generative models work the same way.

They differ in how they generate output, what kinds of output they produce, and how they’re built. This section breaks down the two core architectures, types of outputs they support, the internal structure of models, and how to choose between them.

Two Families of Generative Models

All generative AI models are built on one of two architectures:

1 Autoregressive models generate text one token at a time.

Given the input “The capital of France is”, the model predicts the most likely next token: “Paris” in this case. It appends “Paris” to the sequence, and now uses “The capital of France is Paris” to predict the next token, i.e., “.” Then it uses the full sequence to predict the token, and so on. Each output feeds back as input. This is how GPT, Claude, Gemini, and Llama models work.

When you see a chatbot “typing” its response word by word, that’s not a UI animation; the model is actually producing tokens one after another.

Because generation is sequential, latency scales with output length: a 500-token response takes roughly 500 generation steps. Cost scales, too: a 1,000-token response costs roughly 10x as much as a 100-token response.

Autoregressive generation: each token is produced one at a time, left to right.

2 Diffusion models work completely differently.

Instead of producing one piece at a time, they start with pure random noise and gradually remove it over many steps until a coherent image (or video) emerges.

During training, the model sees millions of images with noise added at various levels and learns to predict what the clean version looks like. At generation time, it reverses the process step by step, typically 20 to 50 steps, removing a little noise each time until a clear image forms.

This is how Midjourney, Stable Diffusion, DALL-E, and Sora work.

The serving model is very different from text: image generation can take roughly 10 to 60 seconds, and video can take minutes, so products typically run generation as batch jobs with a queue rather than streaming.

A single image generation also costs roughly 10-100x as much compute as a typical ChatGPT response.

Models Outputs

These two architectures cover a range of output types:

1 Text generation (autoregressive): chat responses, code, summaries, translations, structured data extraction. This powers ChatGPT, Claude, GitHub Copilot, Perplexity.

2 Text-to-image (diffusion): generate images from text prompts. Midjourney, DALL-E 3, Stable Diffusion.

3 Text-to-video (diffusion): generate video clips from text or image prompts. Sora, Runway Gen-3, Kling.

4 Text-to-audio (autoregressive + diffusion): generate speech, music, and sound effects. ElevenLabs, Suno, Bark.

5 Vision-language models (autoregressive) process images alongside text. Upload a photo and ask the model to describe it, extract data from a receipt, or analyze a chart. GPT-5, Claude, and Gemini all support this.

Multimodality

Each of the output types above used to require its own dedicated model.

That’s changing; models like GPT-5 and Gemini now handle text, images, audio, and video within a single architecture. The question is how. How does a text prompt like “a photo of a cat” actually guide an image model to generate the right image?

Through cross-modal embeddings.

An embedding converts content into a list of numbers that captures its meaning. Cross-modal embeddings do this across different content types, text and images, placing them on the same map.

CLIP, developed by OpenAI, is the most well-known example. It places both “a photo of a cat” (text) and an actual cat photo (image) side by side on this shared map. That shared representation enables a text description to guide image generation.

Model Architecture Variants

Not all foundation models are built the same way…

Most are based on the Transformer (covered in the glossary above), but they use it differently. The two main jobs of a language model are understanding input and generating output.

Different architectures handle these jobs in different ways:

1 Decoder-only models focus entirely on generation: they take an input and produce text left-to-right, one token at a time. GPT, Claude, Llama, and Gemini are all decoder-only. This is the dominant architecture for text generation today.

2 Encoder-only models (like BERT, an early Transformer‑based language model) focus entirely on understanding: they read input and produce a numerical representation of its meaning, but don’t generate new text. Used for classification, sentiment analysis, and search ranking. For example, when a support system reads “I’m furious about my broken order” and classifies it as “negative sentiment, high urgency”, that’s an encoder-only model at work. In GenAI products, they play supporting roles, such as classifying user intent or detecting toxicity.

3 Encoder-decoder models (like Google’s T5 and Meta’s BART) do both: one part reads and understands the input, the other generates the output. Suited for translation, summarization, and rewriting.

4 Mixture of Experts (MoE) is a different kind of optimization.

Instead of a single large model where every parameter is used for every input, MoE splits the model into multiple specialized groups, called “experts.” For each input, a router determines which experts are relevant and activates only those. A question about Python code might activate code-specialized experts, while a question about French history might activate different experts. Models like Mistral’s Mixtral 8x7B use this approach, with 46.7 billion total parameters but only around 12.9 billion active per token. You get more capability per unit of compute, but you need more GPU memory to store all the experts.

The Model Landscape: Proprietary, Open-Weight, and Open-Source

When choosing a model, there are three categories with different trade-offs:

1 Proprietary models (GPT, Claude, Gemini) are accessed via API. They offer the highest capability and are the easiest to start with, but give users the least control. You can’t see the model’s internal parameters, modify its architecture, or run it on your own servers.

2 Open-weight models (Llama, Mistral, Gemma) release their trained parameters publicly. You can download them, run them on your own GPUs, and fine-tune them. Training data and methods are typically not shared. This gives you full control over the model, but it also means you have to manage the infrastructure yourself.

3 Open-source models go further by sharing not just weights but training code, data, and the full methodology behind how the model was built.

The gap between proprietary and open-source is narrowing…

By early 2025, models like Llama 4 Maverick and DeepSeek-V3 were competitive with proprietary models on many tasks. Most production systems use a mix: a proprietary model for complex tasks and an open-weight model for high-volume, simple ones.

Model Selection

Each model family offers a range of sizes (e.g., GPT-5 vs. GPT-5 mini, Claude Opus vs. Haiku, Llama 4 Maverick vs Scout).

Smaller models are faster and cheaper, but less capable. Beyond size, some models offer specialized variants. OpenAI’s GPT-5, for example, includes configurable reasoning effort, and Claude offers an extended thinking mode. These reasoning capabilities require more compute during inference to work through problems step by step, breaking complex questions into smaller parts before answering. This improves accuracy on tasks such as math, logic, and multi-step analysis, but it costs more per request and takes longer to respond.

This trade-off is another reason model routing matters: route simple questions to a faster model, and complex reasoning tasks to a model with these capabilities.

Public benchmarks help narrow down which specific model is best suited to a task. MMLU tests general knowledge across 57 subjects; HumanEval and MBPP focus on code generation; Chatbot Arena by LMSYS aggregates human preference scores from real conversations; and benchmarks like GPQA and ARC-AGI evaluate reasoning and problem-solving abilities.

But benchmarks have a well-known problem: contamination.

Models can be trained on benchmark test data, inflating scores. It’s like a student who memorized the answers to last year’s exam; they score high, but can’t solve newer problems. A model that tops MMLU may underperform on a specific task. Benchmarks are useful for narrowing the field, but final selection requires evaluating on your own data.

That covers the model landscape, what models exist, how they differ, and how to pick between them. All of these models, though, start with random parameters. What turns them into something useful is data.

Reminder: this is a teaser of the subscriber-only newsletter series, exclusive to my golden members.

When you upgrade, you’ll get:

• Simple breakdown of real-world architectures

• Frameworks you can plug into your work or business

• Proven systems behind ChatGPT, Perplexity, and Copilot

Unlock Full Access

Part 2: The Data Layer

A model’s capability comes entirely from the data it was trained on.

The quality of that data, how it’s collected, cleaned, and organized directly determines how well the model performs.

Training Data

Foundation models learn from massive datasets: trillions of tokens scraped from books, websites, code repositories, scientific papers, and forum posts. For diffusion models, training data consists of millions of images, each paired with a text description of its contents.

Raw data is never used directly.

It passes through a data pipeline that cleans, filters, deduplicates, and transforms the data before training begins. For example, the same news article might appear on hundreds of websites. Without deduplication, the model sees the same phrases hundreds of times and overlearns them.

The pipeline handles problems like near-duplicate web pages that inflate certain patterns, toxic or biased content the model would absorb, personally identifiable information (PII) that shouldn’t be memorized, and low-quality text that adds noise without a useful signal.

The quality of this pipeline directly determines the model's quality.

Synthetic data generation is increasingly used to fill data gaps. An existing model generates additional training examples for areas where real data is scarce, such as specialized medical terminology, underrepresented languages, or edge-case code patterns. The quality of synthetic data depends on the model that generates it and how it’s validated.

Bias detection starts here. If the training data overrepresents certain perspectives, demographics, or viewpoints, the model will reflect those imbalances. Data teams audit for representation gaps and offensive content before training begins, but detection is imperfect. Some biases only surface after the model is trained and tested.

NSFW and harmful content requires explicit handling during this stage. Some pipelines filter it entirely. Others retain it in a controlled form so the model can later learn to recognize and refuse it. The approach depends on the product’s intended use.

Data Storage and Retrieval

Training and inference have very different data needs.

During training, the dataset can be terabytes or more in size, containing text, images, and code. Teams typically store this in distributed file systems like HDFS (Hadoop Distributed File System), which splits large files across a cluster of machines so they can be read in parallel, or cloud object storage like S3 (Amazon’s Simple Storage Service), which stores files as objects in the cloud and is widely used because it’s cheap, scales without limits, and requires no infrastructure to manage. Training pipelines stream data from these systems to GPU clusters in batches, feeding the model continuously throughout training.

At inference time, the training data is no longer needed.

What matters is the model itself. The model’s parameters need to be loaded into GPU memory before it can serve any requests, and for large models, that means hundreds of gigabytes. Once loaded, the model takes in a prompt and generates tokens one at a time, streaming them back to the user.

The challenge is keeping sufficient GPU capacity available to handle many users simultaneously without long wait times.

Some GenAI systems also retrieve data at runtime, pulling in documents, search results, or database records when a user asks a question. This is how products like Perplexity stay current beyond the model’s training data.

In Part 6, we cover how this retrieval works, embeddings, vector databases, and structured data handling when we design the full product architecture.

But having clean, well-organized data is only the starting point.

The next question is what happens to that data, and how a pile of text becomes a model that can actually follow instructions…

Let’s keep going!

Read more

• Share this post & I'll send you some rewards for the referrals.

• Block diagrams created using Eraser.

Object storage isn’t just “upload a file, get a URL back.”

That’s true for a small side project. It stops being true when you’re storing 100 trillion objects like AWS S3 does today.

Amazon Simple Storage Service (S3) launched in 2006.

In 2013, Amazon reported storing 2 trillion objects. By 2021, that number crossed 100 trillion. The system handling all of that doesn’t look anything like a file server or a relational database. It’s a fundamentally different class of infrastructure, built around a different set of tradeoffs.

This problem shows up in senior- and staff-level interviews at companies building storage-heavy products…

It tests the exact skills that separate mid-level engineers from seniors: understanding why different storage types exist, designing for durability at a scale where hardware failures occur daily, and making smart trade-offs between consistency, cost, and performance.

We’re going to build this from scratch:

We’ll start with what object storage actually is and how it differs from other storage systems, then work through requirements, capacity estimation, the high-level architecture, disk-based data persistence, durability strategies, metadata design, object versioning1, large-file uploads, and garbage collection.

At each step, we’ll explain the why behind each decision…

Find out why 150K+ engineers read The Code twice a week (Partner)

Tech moves fast, but you’re still playing catch-up?

That’s exactly why 150K+ engineers working at Google, Meta, and Apple read The Code twice a week.

Here’s what you get:

• Curated tech news that shapes your career - Filtered from thousands of sources so you know what’s coming 6 months early.

• Practical resources you can use immediately - Real tutorials and tools that solve actual engineering problems.

• Research papers and insights decoded - We break down complex tech so you understand what matters.

All delivered twice a week in just 2 short emails.

Sign up and get access to the Ultimate Claude code guide to ship 5X faster.

Join 150K+ Engineers

(Thanks for partnering on this post and sharing the ultimate claude code guide.)

I want to reintroduce Hayk Simonyan as a guest author.

He’s a senior software engineer specializing in helping developers break through their career plateaus and secure senior roles.

If you want to master the essential system design skills and land senior developer roles, I highly recommend checking out Hayk’s YouTube channel.

His approach focuses on what top employers actually care about: system design expertise, advanced project experience, and elite-level interview performance.

What is Object Storage?

Before we design anything, we need to understand what object storage actually is…

Engineers often confuse the three main storage categories. Each one exists for a reason, and choosing the wrong one at design time is expensive to fix later.

Block Storage

Block storage is the oldest and lowest-level.

When you plug a hard drive or SSD into a server, the operating system (OS) sees it as a sequence of raw blocks, each typically 4KB in size. The OS decides how to format those blocks and build a file system on top of them. Some applications, such as databases and virtual machine engines, skip the file system entirely and manage blocks directly, which gives them maximum control and performance.

Block storage doesn’t have to be physically attached…

You can connect to block storage over a network using protocols like Fibre Channel or iSCSI2. The server still sees raw blocks, as if the drive were directly attached, but the data resides elsewhere on the network. This is how cloud providers like AWS offer Elastic Block Store (EBS): you attach a network drive to your VM, and it behaves like a local disk.

Block storage is fast and flexible, but it’s also expensive and doesn’t scale cheaply to petabytes.

File Storage

File storage is built on top of block storage.

It adds a layer that handles the complexity of managing blocks and gives you a familiar directory hierarchy: folders, subfolders, and files. You don’t deal with blocks at all. You just read and write files using paths like /documents/report.pdf.

File storage becomes especially useful when many servers need to share the same files.

Protocols like NFS and SMB/CIFS3 allow many machines to mount the same file share and read and write to it concurrently. This is how shared drives inside organizations typically work, and it’s how many legacy enterprise applications store data.

File storage is easier to use than block storage, but it still doesn’t scale to the level object storage does. Hierarchical directory structures become slow and complex when you have billions of files.

Object Storage

Object storage is the newest of the three and the most different…

It makes a deliberate tradeoff: give up performance and mutability in exchange for near-unlimited scalability, very high durability, and low cost.

There are no directories in object storage.

Everything lives in a flat namespace inside containers called buckets. Every object is accessed via a RESTful HTTP API using a unique key. You can’t partially update an object. If you want to change a file, you replace the entire object or create a new version. This immutability4 constraint sounds limiting, but it’s actually what makes object storage so cheap to operate at scale, because it simplifies replication and consistency considerably.

AWS S3, Google Cloud Storage, and Azure Blob Storage are all object storage systems. They’re the foundation of most modern cloud architectures: video files, backups, data lake storage, machine learning datasets, static website assets, and more.

Comparison Table

The key constraint to internalize: objects are immutable.

You cannot edit part of an object. You replace the whole thing, or you version it. This constraint is not an accident. It’s a deliberate design choice that enables the durability and scale properties that make object storage useful in the first place.

Key Terms

These are the concepts you need to know before we get into the design:

Bucket

A bucket is a logical container for objects.

Think of it like a top-level folder, except it’s not really a folder since there’s no hierarchy inside it. Bucket names must be globally unique across all system users, not just within your account.

You have to create a bucket before you can store anything in it.

Object

An object is an individual piece of data stored in a bucket.

It has two parts:

• Payload is the actual data bytes, which can be anything: a photo, a video, a CSV file, a binary blob.

• Metadata is a set of key-value pairs that describe an object, such as content type, creation timestamp, custom tags, and anything else the application needs to store alongside the data.

The metadata is stored separately from the payload and is much smaller.

Object Key

Every object is identified by a key, which is just a string.

In S3, that key looks like a file path: photos/2024/vacation.jpg. But there are no actual directories. The entire string, slashes and all, is just the key. S3 lets you use slashes as a convention to simulate folders, but under the hood, it’s still a flat namespace.

This distinction matters when we design the listing feature later.

Versioning

Versioning is a bucket-level feature that keeps all previous versions of an object instead of overwriting them.

When versioning is enabled, uploading an object with the same key as an existing object doesn’t replace it. Instead, it creates a new version alongside the old one. You can retrieve, restore, or delete any version at any time.

This protects against accidental overwrites and deletions.

Durability SLA

S3 Standard storage class is designed for 99.999999999% durability, also known as eleven nines.

In practical terms, if you store 10 million objects for 10,000 years, you’d expect to lose one. That’s not an accident. It comes from specific engineering decisions around replication and error correction, which we’ll get into in the durability section.

Clarifying Requirements

Before touching any design, you need to understand what you’re actually building and at what scale. Candidates who skip this step in interviews fail immediately, because they end up designing the wrong thing at the wrong scale…

Questions to Ask

• What are the core operations? Upload, download, delete, list?

• Do we need versioning?

• How much data do we need to store in year one?

• What durability and availability targets do we need?

• Do we need to support large file uploads, such as files over multiple gigabytes?

• Any access control requirements? Do different users own different buckets?

Our Assumptions

For this design, we’ll assume:

• Core operations: bucket creation, object upload and download, object versioning, and listing objects in a bucket by prefix

• 100 petabytes of total data

• Durability target: six nines, which is 99.9999%

• Availability target: four nines, which is 99.99%

• Must handle both small objects (tens of kilobytes) and large objects (several gigabytes)

Now, let’s figure out what these requirements actually mean for infrastructure…

Capacity Estimation

Math matters in system design because it tells you what kind of infrastructure you need. Vague statements like “we’ll need a distributed database” aren’t useful without concrete numbers.

Let’s work through the estimates:

Object Size Distribution

In practice, object storage systems see a mix of object sizes. A reasonable assumption for a general-purpose system:

• 20% of objects are small, under 1MB, with a median size of about 0.5MB. These might be thumbnails, config files, or short documents.

• 60% of objects are medium-sized, ranging from 1MB to 64MB, with a median of about 32MB. These might be images, audio files, or compressed datasets.

• 20% of objects are large (>64MB), with a median of about 200MB. These might be videos, database backups, or large archives.

Total Object Count

We’re targeting 100 petabytes of stored data.

In practice, storage systems don’t fill to capacity, so let’s assume 40% utilization, meaning we provision enough storage to hold 100PB when 40% full.

100 PB = 10^11 MB

Weighted average object size: (0.2 x 0.5MB) + (0.6 x 32MB) + (0.2 x 200MB)

= 0.1 + 19.2 + 40.0

= 59.3 MB per object (average)

Total objects at 40% utilization: (10^11 x 0.4) / 59.3

= approximately 680 million objects

680 million objects are a lot.

This tells us immediately that no single machine can store or index all of this. We need distributed storage and a distributed metadata index from day one.

Metadata Storage

Each object needs a metadata record. If we assume roughly 1KB per record (object name, bucket, timestamps, UUID, tags), then:

680 million objects x 1KB = ~680GB of metadata

680GB of metadata is manageable in a database, but the access patterns will require sharding as we scale. The metadata store is separate from the data store, which we’ll cover in the design.

IOPS Constraints

A standard SATA hard drive5 spinning at 7200 RPM can handle roughly 100-150 random seeks per second.

This is called IOPS6, or input/output operations per second. At 680 million objects spread across many disks, the IOPS constraint becomes a real bottleneck, especially for small object workloads where you’re doing many small reads and writes rather than a few large sequential ones.

This is one reason we’ll later choose to merge many small objects into a single larger file on disk, instead of storing each object as its own file.

Design Philosophy: Separating Metadata from Data

Before we look at the full architecture, there’s one core design principle that shapes everything else: metadata and data are stored separately, and for good reason.

This idea comes from how UNIX file systems work:

In UNIX, when you save a file, the filename and the actual data bytes are not stored together. The filename and other file information (size, permissions, timestamps, disk location) live in a data structure called an inode7. The actual data bytes live in separate disk blocks that the inode points to.

Object storage works the same way.

The metadata store is like the inode layer. It holds the object name, bucket, size, and a UUID that points to the location of the actual bytes. The data store is like the disk. It holds the raw bytes, and it only knows about UUIDs, not names or paths.

So why separate them?

Because they have completely different characteristics. The data is immutable. Once written, it never changes. The metadata is mutable. You can update tags, rename objects (in some implementations), or add versioning records. They need different consistency guarantees, different storage engines, and different scaling strategies.

Keeping them separate lets us optimize each independently.

High-Level Architecture

With that principle in mind, here’s how the full system is structured:

Components

Upload Flow

Let’s trace exactly what happens when a user uploads a file named report.pdf to a bucket called company-docs:

1. Client sends a HTTP PUT /company-docs/report.pdf request with the file bytes in the request body.

2. Request hits the load balancer and gets routed to one of the API service instances.

3. API service calls the IAM service8 to confirm the user has WRITE permission on the company-docs bucket. If not, the request is rejected immediately with a 403 Forbidden.

4. API service forwards the file bytes to the data store. The data store persists the bytes and returns a UUID, a unique identifier for this specific object.

5. API service then writes a metadata record to the metadata store. This record contains the object name (report.pdf), bucket ID, UUID returned from the data store, file size, creation timestamp, and any metadata tags the user provided.

6. A 200 OK response is returned to the client.

The metadata record now serves as the bridge between the human-readable path (company-docs/report.pdf) and the actual bytes stored under the UUID in the data store.

Download Flow

Now, let’s trace what happens when someone requests the same file:

1. Client sends a GET /company-docs/report.pdf request.

2. API service calls IAM to verify READ permission.

3. API service queries the metadata store: “What is the UUID for the object named report.pdf in the bucket company-docs?”

4. API service then uses that UUID to fetch the actual bytes from the data store.

5. The bytes get returned to the client.

Notice that the data store never knows the file was called report.pdf.

From its perspective, someone asked for the object with a specific UUID, and it returned the bytes. The translation from name to UUID always happens in the metadata store.

Reminder: this is a teaser of the subscriber-only newsletter series, exclusive to my golden members.

When you upgrade, you’ll get:

• High-level architecture of real-world systems.

• Deep dive into how popular real-world systems actually work.

• How real-world systems handle scale, reliability, and performance.

Subscribe now

Deep Dive: Data Store

The data store is where most of the interesting engineering happens…

Let’s break down its internal architecture:

Internal Components

Data Routing Service

The data routing service is the entry point into the data store…

It’s stateless, meaning it holds no state itself, so you can scale it horizontally by adding more instances. When a write comes in, the placement service determines which data node should receive the data, then sends the data there.

When a read comes in, it asks the placement service where the data lives, then fetches it.

Placement Service

The placement service9 is responsible for knowing the physical layout of the entire storage cluster…

It maintains a virtual cluster map10, which is essentially a registry of every data node, including the rack and availability zone it’s in, how many disks it has, and how much space is used on each disk.

Placement service continuously receives heartbeat messages from every data node.

A heartbeat is a small message a node sends every few seconds, saying, “I’m alive, here’s my current state.” If the placement service doesn’t hear from a node within a configurable grace period (typically 15 seconds), it marks that node as down and stops sending new data to it.

Because the placement service is so critical, you run it as a cluster of 5 or 7 nodes using a consensus algorithm11 like Raft or Paxos.

A consensus algorithm ensures the cluster agrees on a single consistent view of the world, even if some nodes fail. With a 7-node cluster, you can lose 3 nodes simultaneously, and the service keeps running. With a 5-node cluster, you can lose 2. You never run this as a single instance, because if it goes down, the entire storage cluster becomes unavailable for writes.

Data Nodes

Data nodes are where the actual bytes live…

Each data node manages one or more physical disks. Each node runs a daemon process that sends heartbeats to the placement service with information about disk count and available space. When the placement service receives a heartbeat from a new node it hasn’t seen before, it assigns that node an ID, adds it to the virtual cluster map, and tells the node where to replicate data.

Durability is the central promise of object storage and the hardest engineering problem in this design.

We'll get there.

But before we can talk about how data survives hardware failures and entire node outages, we need to understand exactly how a write moves through this system.

The decisions made in the next few steps determine whether durability can be guaranteed…

Read more

• Share this post & I'll send you some rewards for the referrals.

Since people are always asking me for more “practical” system design content:

Following is the free edition of a premium 3-part newsletter series... This covers how mobile apps work under the hood. And it’s explained in a simple, realistic, and useful way.

On with part 1 of the newsletter:

Trying to debug a request that touched 5 services? (Partner)

Distributed systems help you move faster…until you have to debug them.

This blog shows how to use Sentry tracing and logging to follow a request end-to-end. You don’t need prior microservices experience to follow these steps.

Read the blog and start fixing

(Thanks to Sentry for partnering on this post.)

I want to reintroduce Shefali Jangid as a guest author.

She’s a web developer, technical writer, and content creator with a love for frontend architecture and building things that scale.

Check out her work and socials:

• Shefali.dev

• GitHub

• Twitter

You’ll often find her writing about web development, sharing UI tips, and building tools that make developers’ lives easier.

Building “scalable mobile apps” is not just about writing frontend code and calling APIs…

It’s actually one of the toughest parts of software engineering!

Mobile apps don’t run in ideal environments. The network can go down, batteries run low, the operating system can close your app in the background, devices vary, and users still expect everything to work instantly, even offline.

When building mobile apps, you always have to think about limits. Every decision affects speed, stability, data safety, security, and user experience simultaneously. And making the right choices turns a simple app into a strong, production-ready package.

In this newsletter, we’ll focus on 19 concepts as the foundation:

1. Client-Server Architecture,

2. WebSockets & Persistent Connections,

3. Push Notifications,

4. Polling, Long Polling & SSE,

5. REST vs GraphQL vs gRPC,

6. Network Resilience,

7. Idempotency in APIs,

8. Request Batching & Payload Optimisation,

9. Resumable Uploads,

10. Handling Intermittent Connectivity,

11. On-Device Caching,

12. HTTP Caching (ETag, Cache-Control),

13. Offline-First Architecture,

14. CDN Strategy & Media Optimisation,

15. Cache Invalidation Strategies,

16. Local Database Design,

17. Schema Migration Strategy,

18. Pagination (Cursor vs Offset vs Page Number),

19. Data Modeling for Mobile Constraints.

(…and much more in parts 2 & 3!)

For each concept, I’ll cover:

• What it is and how it works

• Real-world example

• The tradeoffs

• Why it matters for mobile

But before we get into the concepts, let’s first understand what mobile system design really means…

What is Mobile System Design?

Mobile system design is about planning how a mobile app works in the background.

It includes how the app connects to servers, saves and reads data, stays fast, keeps user data safe, and handles errors.

It’s not just about designing screens. It’s about understanding how data flows within the app, what happens when the network goes down, how the app runs in the background, and how it works across different devices.

Now that we understand what mobile system design means, let’s start with one of the most important parts: how your app connects and communicates with the outside world.

Networking & Real-Time Communication

This section explains how a mobile app connects to servers and handles real-time data…

You’ll learn how apps send and receive data, choose the right APIs, and handle slow or unstable networks so the app stays fast and reliable.

1. Client-Server Architecture (Thick vs Thin Clients)

When building a mobile app, an important question to ask is: how much work should the phone handle, and how much should the server do?

There are two main approaches:

a. Thick Client

A thick client means the app does most of the work on the device.

It can:

• Save data on the phone

• Handle some logic by itself

• Sync data later in the background

• Work without network connectivity

Most modern apps use this approach because:

• The network is not always stable

• Phones are powerful now

• Users expect apps to work offline

b. Thin Client

A thin client means the app does very little on the device.

It:

• Sends requests to the server

• Waits for the server to process everything

• Shows whatever the server returns

If the network fails, the app mostly stops working.

Why It Matters

This decision affects your entire app design:

• Whether your app can work offline

• How difficult syncing will be

• How easily you can update features

• How much important logic stays on the server

Real-World Example

• Gmail is a “thick” client. You can read emails and write drafts even without network connectivity. It syncs later.

• A simple app that only loads a website is a thin client. It’s useless without the network.

Trade-offs

• Thick client: Faster and works offline, but syncing and updates can be harder.

• Thin client: Easy to update and simpler to build, but it doesn’t work well with a bad network.

Practical Advice

For most apps:

• Keep the main app logic thick (use local storage and background sync).

• Keep sensitive logic on the server (like payments, login, pricing, fraud detection).

This gives you the best balance between speed, reliability, and security.

Once you decide what the app and the server should handle, the next question to ask is: How they should communicate, especially when you need real-time updates?

2. WebSockets & Persistent Connections

Usually, mobile apps use HTTP requests to talk to a server.

That means:

1. App asks for data

2. Server sends a response

3. Connection closes

If the app needs new data, it has to ask again.

WebSocket works differently…

It creates a long-lasting connection between the app and the server.

Once connected:

• App can send data anytime

• Server can also send data anytime

• Connection stays open

This is called real-time communication.

Why It Matters

With normal HTTP, the app must keep asking:

• “Any new messages?”

• “Any updates?”

With WebSockets, the server sends updates immediately when something happens.

This is very important for:

• Chat apps

• Live sports scores

• Stock price updates

• Multiplayer games

• Collaborative tools

If updates are slow, users feel like the app is broken.

Real-World Example

Apps like Slack, WhatsApp, and Figma use WebSockets.

When someone sends a message, you see it instantly because the server pushes it directly to your app.

Important Things to Remember (For Mobile)

On mobile devices, keeping a connection open all the time can drain the battery, use more data, and cause the operating system to close it.

Instead:

• Send small ‘heartbeat’ signals to keep it alive

• Reconnect if the network drops

• Close the connection when the app goes to the background

Otherwise, the connection may appear active but stop working.

How It Works

1. App starts with a normal HTTP request.

2. Server upgrades it to a WebSocket connection.

3. After that, both sides can send messages at any time.

4. Connection stays open until one side closes it.

Trade-offs

Pros

• Very fast (low latency)

• Instant updates

• Two-way communication

Cons

• Harder to manage

• Needs reconnect logic

• Can affect the battery if not handled properly

In simple words, use WebSockets when your app needs instant updates. Just manage the connection carefully on mobile devices.

But not all updates require a constant open connection.

What if the app is completely closed?

That’s when push notifications become useful…

3. Push Notifications (APNs & FCM)

Push notifications let your app send updates, such as “Your order has been picked up”, “You got a new message”, “Your payment was successful”, to users even when the app is closed.

But here’s the important part: Your app does not stay connected to your server all the time.

Instead, special services handle this for you:

• APNs (Apple Push Notification Service) for iPhones

• FCM (Firebase Cloud Messaging) for Android

These services act like messengers between your server and the user’s phone.

How It Works

1. Something happens on the server (e.g., a new message).

2. Server sends the notification to APNs (iOS) or FCM (Android).

3. APNs/FCM deliver it to the user’s device.

4. The mobile phone’s operating system shows the notification.

5. If needed, the app wakes up in the background.

Your app does NOT keep a permanent connection.

The phone’s operating system manages a “single” shared connection for all apps to conserve battery.

Why It Matters

Without push notifications, your app would need to stay connected all the time and keep checking for new updates. Neither iOS nor Android allows this because it drains battery.

Push services solve this problem in a battery-efficient way.

Real-World Example

Imagine a food delivery app. When your order is picked up:

1. The backend sends a push notification using FCM or APNs.

2. Your phone receives it instantly.

3. Even if the app is fully closed, you still get the update.

Trade-offs

Pros

• Saves battery

• Works even if the app is closed

• Delivers updates quickly

• No need for a constant background connection

Cons

• Delivery is not 100% guaranteed

• Can fail on a poor network

• Limited message size

• Should not be used for very critical data

You should never assume the user received the notification. Always refresh or sync data when the app opens.

Pro Tip

You can send a silent push notification (background push).

This shows nothing to the user. It simply wakes the app in the background to fetch fresh data. So when the user opens the app, everything updates without a loading spinner.

Push notifications are useful for sending updates to users even when the app is closed. But in many cases, the app itself needs to fetch updates from the server while it’s running.

In such situations, use techniques such as polling, long polling, or Server-Sent Events.

4. Polling, Long Polling & SSE

Sometimes apps need server updates, but using WebSockets isn’t always necessary or practical. In those cases, use simpler methods such as polling, Long Polling, or Server-Sent Events (SSE) to receive updates from the server.

Each method works a little differently.

Why It Matters

Not every feature needs instant real-time updates:

• Metrics dashboard might refresh every 30 seconds.

• Notification count might update every few seconds.

Using WebSockets for these cases can be unnecessary and complex. Choosing the right approach can save server resources and keep the system simple.

a. Polling (Short Polling)

Polling is the simplest method.

The app requests updates from the server at fixed intervals:

1. Every 10 seconds, the app sends a request.

2. The server responds with new data (if any).

So the process looks like this:

Problem:

Most of the time, there's no new data, yet the app keeps asking the server. This wastes network and server resources.

b. Long Polling

Long polling improves on basic polling.

Instead of responding immediately, the server keeps the request open until new data is available.

How it works:

1. App sends a request.

2. Server waits until something changes.

3. When new data arrives, the server sends the response.

4. The app immediately sends another request.

This reduces unnecessary requests compared to short polling.

Yet it still requires managing many open connections on the server.

c. Server-Sent Events (SSE)

SSE allows the server to push updates to the client over a single open HTTP connection.

How it works:

1. Client sends a request.

2. Connection stays open.

3. Server sends updates whenever something happens.

The client keeps listening for these updates.

SSE also has automatic reconnection built into browsers through the EventSource API.

Real-World Example

• GitHub uses SSE to stream live action logs.

• Many analytics dashboards use polling every few seconds.

• Early versions of Twitter used long polling for real-time updates.

Trade-offs

• Short Polling: Simple to implement, but wastes resources if there are no updates.

• Long Polling: More efficient than polling, but harder to manage on the server.

• SSE: Efficient for real-time updates and simple compared to WebSockets, but communication is one-way (server → client only).

The bottom line:

• Polling: Client repeatedly asks for updates.

• Long Polling: Server waits until new data appears.

• SSE: Server pushes updates over a single open connection.

These approaches are useful when WebSockets are unnecessary or unavailable, which helps you keep the system simple and efficient.

Once you understand how apps receive updates from the server, the next step is deciding how the app requests data from the server.

This is where API design becomes important.

APIs define how the mobile app and server communicate, what data is requested, and how that data is returned.

5. REST vs GraphQL vs gRPC

There are different ways to design APIs.

Three common approaches are REST, GraphQL, and gRPC. Each one has its own advantages depending on the type of app you’re building.

REST

REST (Representational State Transfer) is the most common and widely used API style:

1. App calls specific endpoints like /users, /orders, /products.

2. Server returns predefined data.

It is simple and easy to understand, which is why many mobile apps use it.

But REST can sometimes return too much or too little data.

For example, a screen needs only the user’s name and profile photo, but the API returns email, address, preferences, and more. This is called over-fetching.

Sometimes the opposite happens: the app needs multiple API requests to load one screen. This is called under-fetching.

GraphQL

GraphQL was designed to address the problems of over-fetching and under-fetching.

With GraphQL:

1. Client asks for exactly the data it needs.

2. Server returns only that data.

Example request:

GetUser {
  name
  profilePhoto
}

This makes GraphQL very useful for complex mobile interfaces where different screens need different types of data. Yet GraphQL can be harder to cache and secure compared to REST.

gRPC

gRPC is another approach focused on high performance.

Instead of sending JSON data like REST or GraphQL, gRPC uses a binary format called Protocol Buffers, which is smaller and faster.

It runs over HTTP/2, which allows faster communication between services. Because of this, gRPC is often used for internal microservices, backend communication, and high-performance mobile systems.

But it requires special tooling and is less common for public APIs.

Why It Matters

The API style you choose affects network speed, battery usage, number of requests, and app performance.

If an API returns too much data, it wastes bandwidth and battery. If it returns too little, the app needs more requests, which increases latency. Choosing the right approach helps keep the app fast and efficient.

Real-World Example

• Facebook created GraphQL because its mobile apps needed many REST requests to load a single screen.

• GitHub moved its public API to GraphQL.

• Google uses gRPC internally for inter-service communication.

Trade-offs

• REST: Simple, widely supported, and easy to cache, but sometimes returns too much or too little data.

• GraphQL: Clients request exactly the data they need and great for complex UIs, but caching and security can be more complicated.

• gRPC: Very fast, efficient, and great for internal services, but requires HTTP/2 and special tooling.

Practical Advice

A common approach in real systems is:

• REST for simple CRUD APIs.

• GraphQL for complex UI data requirements.

• gRPC for internal communication between services where performance matters most.

Once you decide how the app will request data from the server, the next step is making sure those requests still work properly when the internet connection is unstable.

6. Network Resilience (Exponential Backoff & Retry Strategy)

Mobile networks can be slow, unstable, or unavailable.

As a result, requests may fail due to weak signals, timeouts, or temporary server issues. Instead of failing immediately, most apps retry the request after a short delay.

Yet retrying requests often can cause serious problems.

Why It Matters

Imagine thousands of mobile apps trying to reconnect at the same time after a server outage.

If every app retries instantly, the server suddenly receives a huge number of requests at once. This can overload the server again, causing another failure. This situation is called the thundering herd problem, when many clients try at the same time and overwhelm the system.

A good retry strategy spreads these retries over time so the server can recover…

Exponential Backoff (The Common Solution)

To avoid retrying too aggressively, apps use exponential backoff. This means the app waits longer between each retry:

• First retry → wait 1 second

• Second retry → wait 2 seconds

• Third retry → wait 4 seconds

• Fourth retry → wait 8 seconds

Each retry waits longer than the previous one. Thus reducing pressure on the server.

Adding Jitter

If every app waited the exact same time, they would still retry together.

To avoid that, add jitter, which is a small random delay. For example, retry after 4 seconds + random delay.

This spreads the requests over time instead of sending them all at once.

Real-World Example

Many popular services, such as AWS SDK, Google Cloud client libraries, and Stripe API clients, use this strategy by default. This approach is considered the standard for reliable network communication.

Trade-offs

Retry strategies must balance reliability and user experience.

If the app takes too long to retry, the user may wait too long for a response. If it retries too quickly, it might overload the server.

To handle this, many systems use a circuit breaker.

A circuit breaker stops retrying after several failures and shows a clear error message instead of retrying forever.

After adding retries to handle network failures, another challenge arises: what happens if the same request is sent many times?

Sometimes a request reaches the server successfully, but the response never reaches the app because the network drops.

If the app retries the request, it could perform the same action again…

7. Idempotency in APIs (Safe Retries)

In mobile apps, network requests can fail halfway through.

For example:

1. App sends a request to the server

2. Server successfully processes it

3. But the response never reaches the app because the internet connection drops

From the app’s perspective, the request appears to have failed, so it retries it. If the server processes the request again, the action may happen twice.

This is where idempotency becomes important.

Idempotency means sending the same request many times produces the same result, rather than repeating the action.

How Idempotency Works

To make retries safe, the client sends a unique identifier with each request, called an idempotency key. This key is usually a unique ID (UUID) generated by the app.

Example request header:

Idempotency-Key: 8f3c2c2e-92f1-4f9b-b8b0-1b0f1b1a3e3f

When the server receives a request:

1. It checks whether this key was used before.

2. If the key is new, the server processes the request normally.

3. If the same key appears again, the server does NOT repeat the action. Instead, it returns the previous response.

This prevents duplicate operations.

Why It Matters

Imagine a user taps the “Pay” button.

1. The payment request reaches the server.

2. Server processes the payment successfully.

3. But the response never reaches the phone because the network drops.

So the user taps “Pay” again!

Without idempotency, the user might be charged twice.

This is why idempotency is extremely important for operations that change data.

Real-World Example

Many large systems rely on idempotency:

• Stripe requires an Idempotency-Key for payment API calls.

• Uber uses idempotency keys for ride requests.

• Most financial systems use this approach to prevent duplicate transactions.

Trade-offs

To support idempotency, the server must store previously used keys and their responses for a period of time (typically 24 hours).

This requires some storage, but the cost is small compared to the problems caused by duplicate operations.

It’s also important to design the key carefully, usually combining:

• user ID

• operation type

• unique request ID

Practical Tip

The idempotency key should be generated on the client before the request is sent. The app should also save the key locally.

If the app crashes or restarts, it can retry the request using the same key, ensuring the action is not repeated.

After making retries safe, the next step is to improve the app's efficiency when communicating with the server. One important way to do this is by reducing the number of network requests the app sends.

8. Request Batching & Payload Optimisation

Every time a mobile app sends a request to the server, it adds latency and uses battery.

The total time for a request to reach the server and its response to return is called a network round-trip.

If an app sends many small requests separately, it can slow down the app and drain the device’s battery.

To solve this, use these two techniques:

• Request batching

• Payload optimisation

a. Request Batching

Request batching combines multiple requests into a single request.

Instead of doing this:

App → Request 1
App → Request 2
App → Request 3

The app sends one combined request:

App → Batch Request (Request 1 + Request 2 + Request 3)

This reduces the number of times the app needs to communicate with the server.

b. Payload Optimisation

Payload is the data sent between the app and the server.

Payload optimisation focuses on reducing the amount of data being transferred.

This can be achieved by:

• Compression (like gzip or Brotli)

• Sending only the needed fields

• Using efficient formats like binary data instead of large text formats

Reducing the app's payload size helps it load faster and use less network data.

Why It Matters

On mobile devices, sending a request wakes up the device’s network radio (the hardware responsible for communication).

This wake-up process takes time and consumes battery.

For example:

• Sending 10 separate requests means the radio wakes up 10 times.

• Sending 1 batched request wakes the radio only once.

So batching requests can significantly reduce both latency and battery usage.

Real-World Example

Many popular apps use batching to improve performance.

• Facebook batches many GraphQL queries into a single HTTP request.

• Instagram loads images gradually but fetches data for the next screen in a single batched request.

This makes the app feel faster and smoother.

Trade-offs

Batching improves efficiency but also introduces challenges.

• If the app waits too long to collect requests before batching them, it can add extra delay.

• Another challenge is handling errors. For example, if a batched request contains 10 operations and 3 fail, the system must handle those failures properly.

To balance this, many systems use a short batching window, usually around 50–100 milliseconds, to collect requests before sending them together.

After reducing the number of network requests, another challenge appears when apps need to upload large files, such as photos, videos, or documents.

Large uploads can easily fail on mobile networks because connections may drop or become unstable. To handle this, apps use a technique called resumable uploads.

9. Resumable Uploads (Chunked Uploads)

When a mobile app uploads a large file, such as a 100MB video, sending the entire file in a single request is risky.

If the network disconnects halfway through the upload, the whole upload may fail. The app would have to start again from the beginning.

To avoid this problem, use resumable uploads, also called chunked uploads.

How Chunked Uploads Work

Instead of uploading the entire file at once, it’s split into smaller parts, called chunks.

For example, a 100MB file might be divided into 10 chunks of 10MB each, and each chunk is uploaded separately.

The process usually works like this:

1. App starts the upload and creates an upload session.

2. Server returns a session ID or upload URL.

3. App uploads the file chunk by chunk.

4. Server keeps track of how much data it has already received.

5. If the network fails, the app asks the server where the upload stopped.

6. The upload continues from that point instead of starting from the beginning.

This makes large uploads much more reliable on mobile networks.

Why It Matters

Mobile networks are often unstable.

Imagine uploading a 200MB video that reaches 99% when the network suddenly disconnects, causing the upload to fail…

Without resumable uploads, the app would have to restart the entire upload from the beginning, which wastes time, data, and battery.

With resumable uploads, the app can resume uploading from where it left off instead of starting over.

Real-World Example

Many popular apps use resumable uploads:

• YouTube for video uploads

• Google Drive for file uploads

• Dropbox for syncing files

Any app that allows users to upload media usually uses this approach.

Even with reliable upload systems, mobile apps still face a major challenge: network connections can drop at any time.

Users may lose connectivity when entering a subway tunnel, switching to airplane mode, or moving to an area with poor network coverage. Because of this, mobile apps must be designed to handle unstable or missing network connections without breaking.

10. Handling Intermittent Connectivity

Mobile apps should continue working as smoothly as possible even when the internet connection is weak or temporarily unavailable. Instead of failing immediately, the app should adapt to the network situation and recover when the connection returns.

Use these techniques to handle this:

a. Local Request Queuing

When the device is offline, the app can store user actions locally rather than sending them to the server immediately.

For example:

1. A user sends a message.

2. The app saves the message locally.

3. When the internet connection returns, the app sends it to the server.

This is called request queuing. The app keeps a list of pending actions and processes them once the network is available again.

b. Optimistic UI

Another common technique is optimistic UI.

With optimistic UI, the app updates the interface immediately, assuming the request will succeed.

For example:

• A user likes a post.

• App shows the post as liked instantly.

• Request is sent to the server in the background.

If the request fails later, the app can correct the state.

This makes the app feel fast and responsive, even when the network is slow.

c. Network State Awareness

Apps can also monitor the device’s network status.

This allows the app to know when it is online, offline, or on a slow connection.

Based on this information, the app can adjust its behaviour. For example, it might delay data syncing while offline and resume syncing when connectivity returns.

Why It Matters

Users frequently lose internet access during normal usage.

For example, in subway tunnels, on airplanes, or in areas with weak network coverage. If an app crashes, freezes, or loses user data, it feels unreliable.

Properly handling connectivity problems makes the app feel stable and trustworthy.

Real-World Example

Many modern apps handle intermittent connectivity this way:

• Notion queues edits locally and syncs them later.

• Google Docs allows you to continue editing even when offline.

• Messaging apps show a “sending” or “pending” status until the message is delivered.

Practical Tip

Mobile operating systems provide built-in tools to monitor network connectivity:

• NWPathMonitor on iOS

• ConnectivityManager on Android

These tools notify the app when the network status changes.

This approach is event-driven and battery-efficient, unlike constantly checking the network in a loop.

Even with reliable networking strategies, mobile apps cannot always depend on an internet connection. To keep the app fast and usable even without internet access, developers use caching.

Caching & Offline

This section focuses on how mobile apps store and reuse data to reduce network requests and improve performance.

You’ll learn how caching strategies and offline-first approaches help apps stay fast and usable even when the internet is slow or unavailable.

11. On-Device Caching (Memory vs Disk)

Mobile apps usually store cached data in two places on the device:

• Memory (RAM)

• Disk (device storage)

Using both together creates a two-layer caching system.

a. Memory Cache (RAM)

Memory cache stores data in device RAM, which is extremely fast to access.

When data is stored in memory:

• App can load it almost instantly.

• It avoids making a network request.

Yet memory has a limitation:

• Data disappears when the app closes.

• Operating system may remove cached data if the device runs low on memory.

So memory caching is fast but temporary.

b. Disk Cache (Storage)

Disk cache stores data on the device’s permanent storage.

This means:

• Data remains available even after the app restarts.

• App can load previously downloaded data without contacting the server again.

But disk access is slower than memory access.

So disk caching is slower than RAM but more persistent.

Why It Matters

Fetching data from the server can take 50–500 milliseconds, depending on the network. But reading data from memory happens in nanoseconds, which is almost instantaneous.

So if the app can load data from the cache instead of the network, the user interface feels much faster. For example, profile pictures, app configurations, or recently viewed content.

These are perfect candidates for caching.

Real-World Example

Many image-loading libraries use this two-layer caching approach. For example, SDWebImage and Kingfisher.

When an image is requested:

1. App first checks the memory cache.

2. If it’s not there, it checks the disk cache.

3. Only if both are missing does it requests image from the network.

This makes images load almost instantly.

Trade-offs

Both types of caching have advantages and limitations:

• Memory Cache: Extremely fast, but cleared when the app closes.

• Disk Cache: Data remains available after restarting the app, but reading from disk is slower than memory.

Cache size also needs to be controlled carefully.

If too much memory is used for caching, the operating system may remove the cached data or close the app to free memory.

Practical Tip

Mobile platforms provide built-in caching tools that automatically manage memory. For example, NSCache on iOS and LruCache on Android.

These systems automatically remove older items from memory when the device is running low on resources. This helps prevent the app from using too much memory.

Using simple data structures like dictionaries or hash maps for caching is NOT ideal, because they do not automatically respond to memory pressure. As a result, they may keep too much data in memory, increasing the risk that the app will be terminated by the operating system.

Local caching stores data on the device, which helps apps load faster.

But the internet itself also has built-in caching mechanisms that can reduce unnecessary network requests. This is where HTTP caching becomes useful.

12. HTTP Caching (ETag, Cache-Control)

HTTP caching allows apps and browsers to reuse previously downloaded responses instead of requesting the same data again.

This is controlled using special HTTP headers, Cache-Control, and ETag. These headers tell the app two important things:

• How long can cached data be reused

• How to check if the data has changed

a. Cache-Control

This header tells the app how long it can reuse the cached data before asking the server again.

Cache-Control: max-age=300

This means the response can be reused for 300 seconds (5 minutes).

During those 5 minutes, the app can load the data directly from cache instead of making a new request to the server.

This improves speed and reduces network usage.

b. ETag

An ETag is a unique identifier that represents a specific version of the data.

When the server sends a response, it also includes an ETag value:

ETag: “abc123”

When the app requests the same data again, it sends that value back to the server:

If-None-Match: “abc123”

The server checks whether the data has changed:

• If the data has NOT changed, the server returns 304 Not Modified with no response body.

• If the data has changed, the server sends the new data.

This saves bandwidth because the server does NOT need to resend the entire response.

Why It Matters

HTTP caching reduces unnecessary network requests.

For example, if an API response is 200 KB, returning a 304 Not Modified response avoids downloading that 200 KB again.

This helps improve app performance, network efficiency, and battery usage. The best part is that HTTP caching works automatically when the correct headers are set.

Real-World Example

A news app might load a list of articles using an API. The server might return a header like this:

Cache-Control: max-age=60, stale-while-revalidate=300

This means cached response can be used for 60 seconds. After that, the app can still show the cached content while checking for updates in the background.

As a result, the articles appear instantly without showing a loading spinner.

Trade-offs

Different caching strategies have different advantages:

• Strong caching (long max-age): Faster loading and fewer network requests, but the data may become outdated.

• Weak caching (ETag validation): Always checks if data is fresh, but still requires a small network request.

Because of this, cache durations should match the frequency of data changes.

Caching helps apps load data faster, but it still depends on the network at some point. When the network connection disappears completely, caching alone cannot solve the problem.

To keep apps usable even without a network connection, many apps use a design approach called offline-first architecture.

13. Offline-First Architecture

Offline-first architecture means the app is designed to work normally even without a network connection.

Instead of always asking the server for data, the app mainly works with data stored locally on the device. The app shows and updates local data first, and then synchronises changes with the server when the network becomes available.

This approach helps the app stay fast and responsive, even when the network is slow or unavailable.

How It Works

In an offline-first system, the app reads and writes data from local storage first.

The process usually works like this:

1. The app reads data from the local database instead of requesting it from the server.

2. When the user makes a change, the app updates the local database immediately.

3. A background process then sends those changes to the server when a connection is available.

4. If the server has new updates, the app downloads them and updates the local database.

5. When the local data changes, the app automatically updates the user interface.

Because the UI always reads from local data, the app feels fast and responsive.

Why It Matters

Apps that depend entirely on the network often feel slow.

Every action requires a request to the server, which can take 100–300 milliseconds or more. This delay often appears as loading spinners.

Offline-first apps avoid this problem.

Since the UI reads from local data, interactions feel instant, while the network syncing happens quietly in the background.

Real-World Example

Many popular apps use an offline-first architecture:

• Spotify downloads playlists so they can be played without internet.

• Notion saves edits locally and syncs them later.

• Google Maps stores map tiles for offline navigation.

Apps dealing with media, productivity, or travel benefit from this approach.

Trade-offs

Offline-first systems provide a better user experience, but they are complex to build.

They require:

• Local database to store data

• Sync system to send and receive updates

• A way to handle conflicts when local and server data differ

• Plan for updating the data structure over time

Because of this complexity, an offline-first design usually needs to be planned early in app development.

Caching helps apps reuse data locally, but another challenge is delivering large files like images and videos efficiently. These files often make up most of the data in app downloads.

To improve speed and reduce data usage, use techniques like CDNs and media optimisation.

14. CDN Strategy & Media Optimisation

Images and videos usually make up a large part of the data used by mobile apps.

To deliver this content quickly and efficiently, apps often rely on Content Delivery Networks (CDNs) and various optimisation techniques. A CDN is a network of servers distributed around the world. Instead of downloading content from a central server, the app downloads it from the nearest server, reducing load time.

While media files can be optimised, the app only downloads what it actually needs.

What a CDN Does

When a user opens an app and requests an image/video:

1. The request is sent to the closest CDN server.

2. CDN returns the file from a nearby location instead of a distant server.

3. The content loads faster because the distance and network delay are reduced.

This speeds up and improves the reliability of content delivery, especially for users in different parts of the world.

Media Optimisation

Another important improvement is optimising the size and format of images and videos.

Some common techniques include:

• Image resizing: Instead of sending a large image (e.g., 4K), the server sends a smaller version that fits the device screen, such as 200×200 pixels.

• Modern image formats: Newer formats like WebP or AVIF store images more efficiently than older formats like JPEG or PNG. They provide similar image quality with smaller file sizes.

• Lazy loading: Images are loaded only when they are about to appear on the screen. This avoids downloading content that the user might never see.

Together, these techniques significantly reduce data usage and improve loading speed.

Why It Matters

Large media files can slow down apps and consume a lot of mobile data.

For example, loading a 4MB image on a small phone screen wastes most of its pixels because the device only displays a small portion of the image’s resolution.

Serving images that match the device size can reduce total data usage by up to 80%, making the app faster and more efficient.

Real-World Example

Many popular apps rely on CDN and media optimisation:

• Cloudinary and Imgix dynamically resize images based on URL parameters.

• Instagram serves images that match the exact resolution of the user’s device.

• On modern Android devices, Instagram also uses the AVIF format to reduce image size.

Trade-offs

Using a CDN improves performance but introduces additional cost, as CDN providers charge based on data transfer.

As a result, CDN assets are usually cached for long periods. Apps often use versioned or hashed URLs to update assets. This process ensures that the new version gets downloaded when needed.

Practical Tip

To make images appear faster, apps often show a temporary low-quality preview while the full image loads.

Techniques like BlurHash or LQIP (Low Quality Image Placeholder) display a blurred or low-resolution version first. This makes the page feel fast, even before the high-quality image finishes loading.

Even with strong caching and efficient content delivery, one important challenge remains: keeping cached data accurate and up to date.

15. Cache Invalidation Strategies

Caching helps apps load data faster by storing it locally.

But cached data can become outdated when the original data on the server changes. Because of this, apps need a way to update or remove old cached data. This process is called cache invalidation.

In simple terms, cache invalidation means deciding when cached data should no longer be used and needs to be refreshed.

Why It Matters

If outdated data stays in the cache for too long, the app may show incorrect information.

For example, a user might see an old product price, a deleted message, or an outdated address. In these situations, showing stale cached data can be worse than not using a cache at all.

That’s why managing cached data correctly is very important!

Common Cache Invalidation Strategies

There are several common ways to control when cached data should expire or update:

a. TTL (Time-To-Live)

TTL means cached data is only valid for a certain amount of time.

For example, if a cache has a TTL of 5 minutes, the app will reuse the cached data for five minutes. After that, it must fetch fresh data from the server.

This approach is simple but may briefly show outdated data.

b. Event-Driven Invalidation

In this approach, the server actively tells the app when cached data is no longer valid.

For example, a product price changes, and the server sends an event to clear or update the cache.

This ensures the app always shows the latest data, but it requires more complex systems to send these updates.

c. Stale-While-Revalidate

This strategy allows the app to show cached data immediately, even if it is slightly outdated.

The app quietly requests fresh data in the background. When the new data arrives, the cache is updated.

This approach improves the user experience by loading the app instantly while keeping data up to date.

d. Versioned URLs

Sometimes cached assets like images or style sheets are stored for a very long time.

To update them safely, the filename or URL includes a version or hash.

Example:

style.abc123.css

When the file changes, the version in the URL changes. Because the URL is different, the cache automatically downloads the new file.

Real-World Example

An e-commerce app might use different caching strategies depending on the data.

For example:

• Product listings may use a TTL of 5 minutes, since a short delay in updates is acceptable.

• Inventory or pricing data may use event-driven invalidation because it must always be accurate.

• Static assets, such as images or stylesheets, can use versioned URLs, allowing them to be cached for a long time.

Trade-offs

Each caching strategy has its advantages and limitations:

• TTL-based caching: Simple to implement, but cached data may briefly become outdated.

• Event-driven invalidation: Keeps data accurate but requires more complex infrastructure.

• Stale-while-revalidate: Makes the app feel fast, but temporarily shows slightly outdated data.

Important Tip

Caches should never have infinite expiration times for data that changes frequently.

If cached data never expires, the app may continue showing outdated information, making it difficult to detect and debug.

Once data is cached or fetched, the app also needs a reliable way to store and organise that data locally, which leads to the next part: storage and database design.

Storage & Data

Mobile apps need efficient ways to store and manage data on the device.

This section explains how local databases, schema design, and pagination help apps handle large datasets while maintaining smooth performance on resource-limited devices.

16. Local Database Design (Schema Modeling)

Mobile apps often store data locally on the device so they can load information quickly and continue working even when the network is slow or unavailable.

To do this, apps use local databases such as SQLite, which is commonly used on both Android and iOS. A database schema simply describes how data is organised in the database. It defines tables, columns, and how different pieces of data are connected.

Good schema design is important because it helps the app load and display data quickly.

How Data Is Often Structured on Mobile

On servers, databases are usually designed to avoid duplicating data across many places. This approach is called normalisation.

But mobile apps often use a slightly different approach called denormalisation.

Denormalisation involves storing related data in the same table, enabling quick reads without complex queries.

This helps mobile apps load data faster because the app does NOT need to combine multiple tables every time it displays information.

Why It Matters

Mobile apps need to display data quickly.

For example, imagine an app showing a list of 50,000 messages. If the database is poorly designed, finding the right messages might take much longer than expected.

A missing index (a structure that helps the database locate data quickly) can cause the database to scan every row in a table. This can turn a 1-millisecond query into a 500-millisecond query, causing visible delays or lag in the user interface.

So the way a database is designed directly affects how smooth the app feels.

Real-World Example

Consider a messaging app.

Instead of storing only a user ID with each message, the app may also store the sender’s name and profile image directly inside the message record.

This means the app can display messages immediately without looking up additional data from another table.

This approach speeds up loading when thousands of messages need to be shown quickly.

Trade-offs

Each database design approach has advantages and disadvantages:

• Denormalised structure: Faster data reading and works well for displaying lists in the UI, but updating repeated data can be harder. For example, if a user changes their name, multiple records may need to be updated.

• Normalised structure: Cleaner, more organised, and easier to maintain consistent data, but requires more complex queries, which may slow down UI rendering.

Because mobile apps mostly read data to display it on the screen, they often prefer denormalised structures for faster reads.

Practical Tip

When writing many records to the database, it’s usually faster to group them into a single transaction.

Instead of saving each item one by one, the app saves many items together in a single operation. This reduces overhead and makes database writes much more efficient.

As apps grow and evolve over time, the database structure may need to change as well. Because of this, apps must handle database schema updates carefully, which leads to the next topic: schema migration strategies.

17. Schema Migration Strategy

Mobile apps often store data in a local database on the device.

Over time, the app may change how this data is structured. For example, a new version of the app might add a new column, change a table, or introduce new data fields.

When this happens, the database structure must be updated safely. This process is called schema migration.

A schema migration ensures that the existing database on a user’s device is updated to match the new version of the app.

Why It Matters

Unlike a web server, mobile databases exist on millions of individual devices that developers cannot directly control.

Some users may update the app immediately, while others may skip several versions before updating. For example, a user installs version 1 of an app and later updates directly to version 5.

The database must correctly apply every change that happened between those versions. So the migrations must run step-by-step:

1 → 2 → 3 → 4 → 5

If a migration fails, the app may crash on launch. Because the database exists on the user’s device, fixing it later can be difficult…

Real-World Example

Many mobile frameworks provide tools to manage database migrations.

For example, Room on Android requires every database version to define how the schema should be updated.

This helps prevent mistakes where the app changes the database structure without providing a safe upgrade path.

Trade-offs

There are two main ways to handle schema changes:

a. Destructive Migration

In this approach, the app deletes the existing database and creates a new one.

This is simple and safe for the app structure, but it removes all locally stored data.

b. Additive Migration

In this approach, the database is updated without deleting existing data.

For example, adding new columns, adding new tables, and keeping old data intact.

This method is usually preferred because it preserves user data.

Practical Tip

Sometimes migrations may fail due to issues like a corrupted database.

In these situations, it is better for the app to reset the database and re-download the data from the server rather than crash. Losing cached data can be recovered, but an app that keeps crashing when it starts creates a much worse experience.

As apps handle larger amounts of stored data, another important challenge appears: loading and displaying large datasets efficiently, which is where pagination becomes useful.

18. Pagination (Cursor vs Offset vs Page Number)

Mobile apps often display long lists of data, such as messages, posts, products, or search results.

Sometimes these lists can contain thousands of items.

Loading all of that data at once would be slow and would use a lot of memory. Because of this, apps usually load data in smaller chunks called pages. This technique is called pagination.

Pagination helps apps load content gradually as users scroll, keeping the app fast and responsive.

How It Works

Instead of loading everything at once, the app requests a small set of items at a time.

For example:

1. App requests page 1 of data.

2. The server returns the first group of items.

3. When the user scrolls further, the app requests the next page.

This process continues until the user reaches the end of the list…

Common Pagination Methods

There are three common ways to implement pagination:

a. Page Number Pagination

This is the simplest method. The client requests data using page numbers.

For example:

?page=1
?page=2
?page=3

Each page contains a fixed number of items.

This approach is easy to understand and works well for things like search results or product listings.

b. Offset Pagination

Offset pagination retrieves data starting from a specific position.

For example:

?limit=20&offset=40

This means skip the first 40 items and return the next 20 items.

This method works well for static data, but dynamically adding new items while the user is browsing can cause problems.

For example, if new posts appear at the top of a feed, the list may show duplicate items or skip items.

c. Cursor Pagination

Cursor pagination uses a cursor token that represents the last item the user has seen. Instead of requesting a page number, the app requests the next items after a specific position.

For example:

next_cursor = “abc123”

Then the next request might look like:

?after=abc123

The cursor usually represents a stable value, such as the ID or timestamp of the last item. This approach works well for data that constantly changes, such as social media feeds.

Why It Matters

Imagine a social feed with 10,000 posts. If the app tried to load everything at once:

• The request would take a long time

• Rendering the UI would be slow

• Device might run out of memory

Pagination solves this by loading only a small portion of the data at a time, which keeps scrolling smooth.

Real-World Example

Many popular apps, such as Twitter, Instagram, and TikTok, rely on cursor-based pagination.

These apps load new posts as the user scrolls, using a cursor that points to the last item that was displayed.

This ensures the feed remains stable even when new posts appear…

Trade-offs

Each pagination method has advantages and limitations:

• Page number pagination: Simple to understand and works well for search results, but less efficient for large datasets.

• Offset pagination: Easy to implement and allows jumping to any position, but it can break when new data is inserted.

• Cursor pagination: Stable and efficient for live feeds and prevents duplicates and skipped items, but does not easily support jumping directly to a specific page.

Because mobile devices have limited memory and storage, loading and storing data must be done carefully! Therefore, it’s important to design efficient data models that work well within these constraints.

19. Data Modeling for Mobile Constraints

Mobile apps often store data locally on the device so they can load information quickly and continue working even when the network is slow.

Yet mobile devices have limited storage, CPU power, and battery, so the way data is structured must be carefully designed.

Because of these limits, mobile apps usually store and organise data to help the UI load quickly, even if it means repeating some information.

Things Mobile Data Models Must Consider

When designing data models for mobile apps, several important constraints should be considered:

a. Limited storage

Mobile devices cannot store unlimited data.

Apps should avoid caching everything and instead remove old or unused data after a period of time. For example, an app might store cached data for a few hours or days, then automatically delete it.

b. CPU performance

Complex database queries can slow down the app.

Queries that join many tables may take longer to process. To keep the app fast, store related data together so the app can read it quickly without performing complicated queries.

c. Battery usage

Every time an app writes data to disk, it uses battery power.

Frequent database writes can drain the battery over time. Because of this, apps try to minimise unnecessary writes and batch operations when possible.

Why It Matters

Data models used on backend servers are usually designed to save storage space and keep data perfectly structured.

But mobile apps have a different goal: fast rendering on the screen.

This means the data model on mobile may look different from the backend data model, because it’s optimised for speed and user experience rather than strict structure.

Real-World Example

Imagine a contacts app that shows a list of people.

Instead of calculating everything each time the list loads, the app may store some information in advance, such as display name, profile picture URL, and the first letter used for alphabetical sections.

Because this information is already stored, the app can render the list instantly without performing extra calculations.

Trade-offs

Storing extra or pre-computed data improves speed, but it also has some downsides.

Pre-computed data uses more storage space and must be updated whenever the original data changes.

Yet this trade-off is usually acceptable because storage is relatively cheap, while slow or laggy interfaces are immediately noticeable to users.

Final Words

None of these concepts works on its own.

The choices you make in one area affect everything else. Networking affects caching, caching affects sync, and sync affects how conflicts are handled.

It’s all connected.

The best engineers aren’t the ones who know the most concepts. They’re the ones who understand the tradeoffs and know when to use what.

So remember this:

Start simple. Add complexity only when you really need it. And when that time comes, you’ll know what to use.

👋 I’d like to thank Shefali for writing this newsletter!

Plus, don’t forget to check out her work and socials:

• Shefali.dev

• GitHub

• Twitter

You’ll often find her writing about web development, sharing UI tips, and building tools that make developers’ lives easier.

Get the full premium newsletter series and max your system design career leverage:

Subscribe now

Plus, there are group discounts, gift options, and referral rewards available.

Want to reach 210K+ tech professionals at scale? 📰

If your company wants to reach 210K+ tech professionals, advertise with me.

Thank you for supporting this newsletter.

You are now 210,001+ readers strong, very close to 210k. Let’s try to get 211k readers by 27 March. Consider sharing this post with your friends and get rewards.

Y’all are the best.

Share

• Block diagrams created using Eraser.

• Share this post & I'll send you some rewards for the referrals.

You’ve probably been there: You’re sitting across from two engineers (or staring at them through a webcam), and they’ve just asked you to design Instagram’s feed. Or a ride-sharing app. Or some chat system with offline support and end-to-end encryption.

The clock is ticking…

You have 45 minutes to turn a vague prompt into something that looks like you know what you’re doing.

Your mind races: Where do I even start? Do I draw boxes first? Ask about the user numbers? Talk about databases? What if I forget something critical?

Most mobile engineers don’t realize this about system design interviews: They’re NOT testing whether you can build Instagram. They’re testing whether you can take something fuzzy and turn it into something concrete.

This is the same skill you use every day when your PM says, “We need push notifications,” or your designer drops a Figma file in Slack with the message “Thoughts?” in your day job.

12% OFF MiniMax M2.7 - The SOTA Cowork Agent Model That Just Outranked Opus and Gemini 3.1. (Partner)

MiniMax M2.7 is here, and it’s unlike any agent model released before it.

Built on their proprietary SOTA Cowork Agent Model architecture, M2.7 shows early echoes of self-evolution: the ability to follow complex, layered instructions across long workflows without losing the thread.

At 5 tools, every model looks sharp.

At 40, almost all fall apart silently.

Wrong tool invoked. Step missed. No error thrown. You find out 3 days post-deploy.

M2.7 was tested across 40 complex skills, each exceeding 2,000 tokens.
Skill adherence rate: 97%.

Here’s what that looks like in production:

• Alert fires → metrics correlated → root cause traced across systems

• DB queried to verify → fix submitted

• Recovery time: under 3 minutes. Manual process: 4+ hours.

Most models are built to write code.

M2.7 is built to understand your production systems.

And it ships inside something new: the world’s first subscription plan for an all-modality-capable model. One API key covers M2.7, Speech, Image, Video, and Music generation; the full stack, no provider switching.

MiniMax Token Plan 12% OFF

I want to introduce Tjeerd in ‘t Veen as the guest author.

He recently released the Mobile System Design book bundle based on years of interviewing mobile engineers and building large-scale mobile apps. This newsletter distills the core process (framework) from the books into something you can use to prep for interviews or plan features at work.

If you want to go deeper, the full book bundle has more examples, detailed architecture patterns, and code-level implementation strategies.

What this newsletter covers

There are plenty of great general system design guides. This one is about mobile constraints and how they shape the architecture.

This newsletter walks through the actual process:

• How to take a vague prompt and figure out what the interviewer actually wants

• How to uncover the hidden requirements nobody mentioned (because they’re testing if you’ll think of them)

• How to structure your thinking so you don’t forget the mobile-specific stuff (offline state, battery drain, app lifecycle, memory constraints)

• How to turn abstract architecture talk into concrete implementation details

This works whether you’re in an interview or planning a real feature at work.

The process is the same.

Who this is for

If you’re a mobile engineer (iOS, Android, Flutter, React Native, etc.) and you need to:

• Prepare for system design interviews

• Stop feeling like you’re fumbling when asked to “design something.”

• Learn how to think through mobile architecture in a structured way

• Get good at turning vague requirements into actual plans

Then this is for you…

I’m assuming you already know how to develop mobile apps. You understand the basics of networking, data persistence, and why you shouldn’t do heavy work on the main thread. You’ve shipped something to production and dealt with the consequences.

What you might not have is a structured way to think about system design. That’s what we’re covering here…

Interview Structure

Let’s talk about what actually happens in a mobile system design interview:

The format can vary wildly between companies. Some give you 45 minutes, others give you 60. Some want you on a whiteboard, others use tools like CodeSignal or HackerRank. Some companies even ask backend system design questions in mobile interviews, which is a separate discussion entirely.

But most mobile-focused interviews follow a similar arc.

Here’s what you can typically expect…

Typical flow

Most interviews break down roughly like this:

At a high level, the flow is:

• Requirements and scope (5-10 minutes): clarify what you’re designing and the constraints

• API and data needs (5-10 minutes): define what the app needs from the server

• Architecture (10-15 minutes): outline client components and data flow

• Deep dive (15-20 minutes): walk one critical flow or adjust for a new constraint

• Wrap-up (1-5 minutes): summarize decisions and tradeoffs

I’ll go deep on how to execute each phase in the next section…

What’s expected at different levels

Interviewers don’t judge all mobile engineers the same way.

Here’s what interviewers are actually looking for based on your level:

Junior to mid-level engineers

You’re expected to:

• Ask clarifying questions (even if they seem obvious)

• Draw a reasonable architecture diagram without major gaps

• Explain common mobile patterns (caching, background tasks, handling lifecycle)

• Discuss basic tradeoffs - ”This approach is simpler but uses more memory.”

You’re not expected to:

• Know every edge case off the top of your head

• Have opinions on monorepo versus multi-repo

• Design for 100 million users on day one

If you get stuck on something complex (like offline conflict resolution), just say so. “I haven’t built a system that merges offline edits before, but I’d start with last-write-wins and ask about requirements,” is fine.

Honesty beats bullshitting.

Senior engineers

Everything from junior/mid, plus:

• Proactively uncover hidden requirements without prompting - ”What about users on limited data plans?”

• Propose architecture with clear reasoning for your choices - ”I’m using a repository pattern here because we need to support offline-first”

• Demonstrate deep knowledge of mobile-specific challenges (app lifecycle, memory pressure, battery drain, offline sync)

• Adapt when requirements change mid-interview - ”Okay, if we need to support 50MB video uploads, we’ll need background upload with chunking…”

You should drive the conversation.

The interviewer shouldn’t have to pull answers out of you.

Staff and principal engineers

Everything from senior, plus:

• Frame the problem: “Before we design this, let’s talk about whether we even need offline support for this use case.”

• Consider team structure: “If we structure this as a separate module, we can have one team own it independently.”

• Think long-term: “This works for v1, but if we add feature X later, we’ll need to refactor. Let’s design for that now.”

• Make architectural tradeoffs at the system level: modularization strategy, build times, deployment, backward compatibility

You’re expected to think like a tech lead or architect, not just an IC implementing a feature. The interviewer wants to see if you can make decisions that affect the entire team or org.

Now that you know what the interview structure looks like, let’s talk about the actual process you’ll use during those 45 minutes.

Core Process

The interview structure gives you the timeline. But what do you actually do in each phase?

This is the process you’ll follow, roughly in order, during the first 30-40 minutes of the interview. Think of it as a checklist for making sure you forget nothing critical.

You won’t always do these steps in exactly this order. Sometimes you’ll jump back and forth. Sometimes the interviewer will steer you in a different direction. That’s fine.

But this is the general flow that works:

A reusable pattern (applies to every step)

Most steps look different, but the rhythm is the same:

• Ask 2-3 high-impact questions before designing.

• Restate what you heard and get a clear “yes.”

• Write down decisions: constraints, must-haves versus nice-to-haves, and open questions.

• Call out red flags early instead of waiting.

• You don’t have to solve everything on the spot. Show you see the issue, pick a reasonable tradeoff, and move on.

Capturing the briefing

Here’s how most system design interviews start:

The interviewer says something like, “Design a feed for a social media app,” or “Build a way for users to share photos with friends.”

And your brain immediately starts racing: Should I ask about the backend? Should I start drawing? What if I forget to mention caching?

Before you do any of that, pause.

Your first job is not to design anything. It’s understanding what you’re being asked to design.

This sounds obvious, but most candidates blow past this step in 60 seconds and then spend 40 minutes designing the wrong thing…

What you’re actually trying to do

In the first 5-10 minutes, you’re trying to turn a vague prompt into something concrete enough that you know what problem you’re solving.

You’re not trying to get every detail. You’re trying to get enough clarity that you’re not completely guessing.

Think of it like this: If someone asks you to “design a messaging feature,” do they want:

• WhatsApp (end-to-end encrypted, real-time, requires phone number)

• Slack (threaded, searchable, integrates with other tools)

• Twitter/X DMs (simple, text-only, tied to social graph)

All of those are “messaging.” They’re also completely different architectures.

So your job is to ask questions that reveal which one they actually want.

Questions that actually matter

Here’s what you should ask:

About success criteria:

• “What does success look like for this feature?” (Are we optimizing for speed? Reliability? Simplicity?)

• “What’s the most important thing this needs to do well?” (If we could only do one thing right, what would it be?)

About scope:

• “Is this the whole app, or are we focusing on one part?” (Don’t design all of Instagram if they just want the feed)

• “What platforms are we targeting?” (iOS only? Android? Cross-platform?)

About constraints:

• “What network conditions should we plan for?” (Always online? Intermittent? Fully offline?)

• “Are there any technical constraints I should know about?” (Existing systems we need to integrate with? Legacy code?)

About users:

• “Who are the primary users?” (Consumers? Enterprise? Internal tools?)

• “What devices are they on?” (Flagship phones? Low-end Android? Tablets?)

Don’t ask everything.

Pick the questions that will most change your design. You have limited time.

A bad and a good example

Bad example:

Interviewer: “Design a feed for a social media app.”

You: “Okay, so I’ll use MVVM for the architecture. We’ll have a ViewModel that fetches data from a Repository, which talks to both a local database and a REST API. UI will be built with—”

Interviewer (internally): They didn’t ask a single question. Do they even know what they’re designing?

You just spent 3 minutes designing an architecture without knowing:

• Whether it needs to work offline

• Whether it’s a text-based feed or heavy on images and video

• Whether it requires pre-caching for a smoother scrolling experience

That’s a red flag. Interviewers are explicitly looking for whether you ask questions.

Good example:

Interviewer: “Design a feed for a social media app.”

You: “Okay, let me make sure I understand what we’re designing. Is this feed mostly images and video, or mostly text? That affects caching and pre-fetching on mobile.”

Interviewer: “Mostly images, some video.”

You: “Got it. For mobile, I’d plan on disk caching images (and maybe short videos) and pre-fetching upcoming content so the feed loads fast and works offline. Is that aligned with what you want?”

Interviewer: “Yes, basic offline support for previously loaded content is important.”

You: “Makes sense. One more thing: do we need live updates, or is pull-to-refresh enough for v1?”

Interviewer: “Pull-to-refresh is fine.”

You: “Perfect. So, to summarize: we’re designing a media-heavy feed with basic offline support. We’ll cache images (and maybe short videos), pre-fetch upcoming content for fast loads, and start with pull-to-refresh for updates. Does that sound right?”

Interviewer: “Yep, that’s it.”

Reminder: this is a teaser of the subscriber-only newsletter series, exclusive to my golden members.

When you upgrade, you’ll get:

• High-level architecture of real-world systems.

• Deep dive into how popular real-world systems actually work.

• How real-world systems handle scale, reliability, and performance.

Subscribe now

Briefing wrap-up (red flags + next steps)

Sometimes the briefing is incomplete or contradictory. Call it out early instead of guessing…

Common red flags:

• Requirements that are vague or overly broad - ”make it fast.”

• Contradictory statements - ”we want it simple, but also highly customizable.”

• Missing critical info (no mention of offline, auth, or data sources)

When you spot these, call them out:

“You mentioned we want it to be ‘fast.’ Can you be more specific? Are we talking about load time under 1 second, smooth scrolling, or something else?”

“I’m noticing we haven’t talked about where the data comes from. Should I assume we’re pulling from a REST API, or is there a different data source?”

Don’t rush ahead, hoping it’ll become clear later. It won’t. You’ll end up designing something that doesn’t match what they wanted…

Next steps once the briefing is clear:

• Write down key constraints in the shared doc or on the whiteboard (”Offline support required”, “100k users”, “Existing auth system”)

• Mentally categorize what is a must-have versus a nice-to-have

• Identify the biggest unknowns you’ll need to address

Lesson: Don’t start designing until you know what you’re designing.

This can feel slow, but it’s faster than designing the wrong thing…

Defining scope and requirements

Okay, so you’ve asked your questions, and you have a basic understanding of what you’re designing. Now comes the hard part: figuring out what you’re actually going to design.

Here’s the problem: If someone asks you to “design a messaging feature,” there are about 47 different things they could mean. Do they want read receipts? Typing indicators? Group chats? Voice messages? End-to-end encryption? Reactions? Threads?

You can’t fit all of that into v1. You probably can’t even fit half of it.

So your job is to separate what’s essential from what’s nice-to-have.

This is where most people mess up. They either:

1. Try to include everything at once and run out of time

2. Design the wrong things and miss what actually matters

3. Don’t ask at all and just guess

All three are bad. Let’s talk about how to do this right:

What you’re trying to figure out

You need to answer these questions:

Read more

• Share this post & I'll send you some rewards for the referrals.

Every large language model1 (LLM) you use has lied to you with confidence, fluency, and frequency…

Ask any model about something that happened last week. It doesn’t know. It can’t know. Its knowledge was frozen months ago. They might try, and if they do, they will hallucinate.

This isn’t a bug.

It’s a fundamental architectural limitation. LLMs keep knowledge in their parameters2. These are billions of numerical weights learned during training. Once training ends, the knowledge is locked. The model doesn’t know what it doesn’t know, so it fills the gaps with confident fabrication. Studies show that hallucination3 rates can be as low as 1% for simple summarization tasks. However, they can exceed 58% for complex work.

This is exactly what Retrieval-Augmented Generation4 (RAG) solves.

RAG doesn’t bake knowledge into the model. Instead, it pulls in relevant context when you ask a question. The model stays smart, your data stays current, and every answer is traceable back to its source.

This is the architectural pattern that makes AI actually useful.

Here’s how it works, why it works, and how to build it:

Still juggling 10 different tools? Learn AI workflows and replace 80% of them (Partner)

Perplexity’s new computer thinks, designs, codes, and manages projects — all without you lifting a finger to switch tools.

This is where AI is headed: one place to get everything done. Smarter, faster, and built to make you 10x more productive.

Now is your chance to enter into your most productive era!

That’s why we recommend joining Outskill- the world’s first AI learning platform where over 10+ Million Learners have learnt from top industry leaders like Microsoft, NVIDIA, and Google.

They are hosting a 2-day LIVE AI Mastermind where you’ll build automations, create personalized agents, and learn to turn AI into your ultimate competitive edge.

START LEARNING NOW

You will also unlock exclusive bonuses for free when you show up: A Prompt Bible, AI monetization roadmap, and a personalized toolkit builder - for which you would have to pay $1000+ outside.

🧠 Happening LIVE- Saturday and Sunday
🕜 10 AM EST to 7PM EST

Register here before they run out of seats. (free for the next 72 hours only!)

I want to introduce Eric Roby as a guest author.

He’s a senior backend and AI engineer focused on building real-world systems and teaching developers how to do the same. He runs the YouTube channel @codingwithroby, where he focuses on backend engineering, and created the platform The Backend OS.

Through his content and courses, he helps engineers go beyond tutorials, think in systems, and develop the skills that actually matter for real backend roles.

Check out The Backend OS, built to close the knowledge gaps you don’t even know you have.

The Knowledge Problem

We need to understand why LLMs struggle with real-world knowledge before we fix them. The problem has three layers, and none of the obvious solutions work…

LLM knowledge is frozen at training time

Every LLM has a cutoff date for its knowledge.

Everything the model knows was scraped, processed, and compressed into parameters during training. After that date, the model is blind. It isn’t aware of your new product launch, yesterday’s security issue, or this morning’s HR policy change.

This isn’t a minor inconvenience.

In business, accuracy is crucial. This includes areas like customer support, legal analysis, internal search, and compliance. A model that can’t access current information is a liability. It doesn’t help the business at all.

Context Windows Aren’t the Answer

The brute-force approach is tempting: dump everything into the prompt.

Context windows5 have increased a lot. Some models can now handle over a million tokens6. But this approach has three fatal problems.

First, it’s expensive.

You pay for each token. Sending your entire knowledge base with every query will wipe out your budget.

Second, there are hard limits.

A million-token window can’t fit all of a large enterprise’s documents or its databases.

Third, and this is the one most people miss, models get worse with more context.

Research from Stanford and UC Berkeley shows LLM performance follows a U-shaped curve. Models do best when key information is at the start or end, but accuracy drops sharply when important facts are buried in the middle. In Liu et al.’s multi-document QA experiments, accuracy for some models dropped to roughly 25% when key information was placed in the middle of a 20-document context.

It’s clear: adding more context to the prompt doesn’t guarantee the model will use it.

Fine-tuning isn’t the answer either

Fine-tuning7 is when you take a pre-trained model and continue training it on your own specific data, so it learns new knowledge or behavior.

Fine-tuning changes the model’s weights to incorporate new knowledge. Think of it like sending someone back to school for a specialized course. In theory, this lets you teach the model about your domain.

In practice, it creates more problems than it solves…

It requires GPU compute, machine learning expertise, and carefully prepared training data. It takes days or weeks to complete. The result is a snapshot. Once your underlying data changes, your fine-tuned model becomes stale.

What’s Actually Needed

The key is to provide the AI with the right information at the right time for each query. And it should be done without changing the model itself.

That’s RAG.

Now let’s investigate RAG deeper:

What RAG actually is

Retrieval-Augmented Generation is an architectural pattern, not a product. The concept is straightforward, and the best analogy is an open-book exam.

A student taking a closed-book exam relies on what they have memorized.

That’s a standard LLM. It’s smart and can reason, but it’s limited to what’s in its parameters. A student with an open-book exam has the same reasoning skills. They can check relevant pages before answering.

That’s RAG.

The formal definition comes from a 2020 paper by Patrick Lewis and his team at Facebook AI Research and University College London. RAG models combine parametric memory8 with a pre-trained language model and non-parametric memory9. This non-parametric memory uses an external knowledge index. It’s accessed by a neural retriever10.

You can still update the model’s knowledge by swapping the retrieval index. No one needs to retrain.

In practice, RAG follows a three-step loop:

1. Query comes in: a user asks a question.

2. Retrieve: System looks in an external knowledge base for the best information chunks.

3. Question and context go to the LLM. It then creates an answer based on the retrieved documents.

You’re not changing the model. You’re changing what it sees. That distinction makes RAG so powerful and so practical.

The open-book analogy makes sense at a high level. But understanding why it works so well means diving deeper into the topic…

How RAG Works Under the Hood

The three-step loop sounds simple. The engineering that makes it work is where things get interesting.

A RAG system has two main parts:

• An offline ingestion pipeline that gets your data ready.

• An online retrieval pipeline that answers queries.

Let’s walk through each one…

Embeddings: Core Mechanism

Before anything else, you need to understand embeddings11. This is the concept that enables semantic search12.

Text embeddings turn text into dense numerical vectors13. These are arrays of floating-point numbers, usually with 1,536 or 3,072 dimensions. They capture the text meaning. The magic is in what “capture meaning” means. Words and sentences with similar intent are close together in vector space. This happens even with different words.

Consider this: “How do I reset my password?” and “I can’t log into my account” use completely different words.

But when converted to embeddings, they produce nearly identical vectors. The distance between them is tiny because their meanings are similar. This is measured using cosine similarity14, dot product, or Euclidean distance.

The key insight for RAG is clear.

System finds relevant content. It does this even if the user’s question doesn’t use the exact words from the source document. It searches by meaning, not by keywords.

Data Ingestion Pipeline (Offline Phase)

First, process your knowledge base. Then, index it. Only then can your RAG system answer questions.

This happens in five steps:

What triggers the offline phase?

In traditional workflows, this pipeline runs when new data is available. This can happen when documents are added or updated. It can also occur when a database changes or on a regular schedule, like nightly or weekly. Some teams trigger re-ingestion if retrieval quality drops.

They also do this when a new data source connects:

1. Load documents from anywhere: PDFs, databases, APIs, wikis, Slack channels, and Confluence pages. Frameworks like LangChain15 and LlamaIndex16 offer ready-made connectors for many common sources.

2. Chunking17: Split documents into semantically meaningful pieces. This is the single highest-leverage step to get right, and we will cover it in depth later.

3. Embedding: Convert each chunk into a vector using an embedding model.

4. Storage: Store the vectors in a vector database18 optimized for similarity search.

5. Metadata tagging: Tag each chunk with source, timestamp, category, and access control information. This metadata becomes critical for filtering, attribution, and security later.

Retrieval Pipeline (Online Phase)

When a user asks a question, the retrieval pipeline kicks in:

1. Query embedding: User’s question turns into a vector. This uses the same embedding model from ingestion. This is critical: the query and the documents must live in the same vector space.

2. Similarity search: The system finds the top-K19 similar chunks. It compares the query vector to each vector in the database.

3. Retrieval strategy: This is where the real engineering decisions happen. Three primary approaches exist:

   1. Sparse retrieval20 uses a statistical method that matches exact keywords. It weighs these matches using term frequency and inverse document frequency.

   2. Dense retrieval (embeddings): Semantic search via vectors. Finds conceptually relevant content even when the wording differs.

   3. Hybrid search (combining both): Use both sparse and dense retrieval. Then, merge the results with Reciprocal Rank Fusion21 (RRF). This boosts documents that rank high in both systems.

4. Re-ranking: After retrieving results, a cross-encoder model rescans them. It processes the query and each document as one input. This captures fine-grained relevance that bi-encoders miss.

Generation Phase

With the most relevant chunks in hand, the system assembles the final prompt:

1. Prompt Construction:

   1. System Prompt: Combine the system prompt (the instructions that tell the LLM how to behave, like “You are a helpful customer support agent”) with the user’s request.

   2. Retrieved Context Chunks: Integrate the relevant pieces of text pulled from your knowledge base. For instance, if someone asks, “What’s your refund policy?” You might pull two paragraphs from your company’s policy document.

   3. User’s Original Query: Include the user’s actual question or task — exactly as they typed it.

2. LLM Call: The model generates an answer grounded in the retrieved documents. The key facts are in the prompt. The model uses these facts to reason instead of relying on its trained memory. Think of it like giving someone an open-book exam instead of asking them to answer from memory.

3. Citation and Attribution: The system shows which source documents were used and provides verifiable citations. For example, the response might say, “Based on Section 3.2 of the Employee Handbook...” so the user knows exactly where the answer came from. This is one of RAG’s biggest advantages over fine-tuning: transparency.

Now that you know how RAG works mechanically, the real question is whether it’s the right tool. That depends on what you’re comparing it to…

Why RAG over alternatives?

RAG isn’t the only approach to giving AI access to knowledge.

You need to know when to use it and when not to. This means comparing it to other options.

RAG vs. Fine-Tuning

This is the comparison most teams face first. Here’s how they stack up:

The key nuance: fine-tuning and RAG are complementary, not competing.

Fine-tuning is the best option when you want to adjust the model’s behavior, style, or output format, not just its knowledge. The best production systems do two things: they fine-tune style and behavior, and then they use RAG for knowledge. You can adjust a model to produce structured JSON in a certain format.

Then, use RAG to fill it with up-to-date data.

RAG vs. Long Context Windows

Context windows keep growing. Why not dump everything in the prompt?

Cost and precision. RAG retrieves only the 5-20 most relevant chunks. Long context stuffs everything in and hopes the model finds the needle. RAG is cheaper per query. It’s also more precise in what it finds. Plus, it can search millions of documents, unlike others that have a token limit.

They work well together: first, use RAG to get the right 20 chunks.

Then, use long context to process them all at once. The “lost in the middle” research shows that retrieving small, relevant chunks works better than using large context windows.

Reminder: this is a teaser of the subscriber-only newsletter series, exclusive to my golden members.

When you upgrade, you’ll get:

• High-level architecture of real-world systems.

• Deep dive into how popular real-world systems actually work.

• How real-world systems handle scale, reliability, and performance.

Subscribe now

Traditional RAG vs. Agentic RAG

Traditional RAG follows the straightforward three-step loop described above: retrieve, then generate. It works well for simple question-answering but has limitations. What happens when the first retrieval doesn’t return good results? Traditional RAG just pushes forward with whatever it finds.

Agentic RAG adds a reasoning layer on top.

An AI agent decides how to handle each query. It can change the search, link several retrievals, pick a knowledge base, or skip retrieval completely. Think of Traditional RAG as a student who looks up one page and writes their answer. Agentic RAG is like a student. They check several sources. They re-read sections that seem off and cross-reference information before writing.

Multi-Step RAG Pipeline

Production RAG systems often go beyond the simple retrieve-and-generate loop. A multi-step pipeline adds intelligence before and after retrieval:

1. Query Intent Parsing: Before searching, the system analyzes what the user actually wants. Is it a factual question? A comparison? A request for a summary? Understanding intent helps the system choose the right retrieval strategy and knowledge base.

2. Query Reformulation: The system may rewrite the user’s question to improve retrieval quality. For example, “Why is my app slow?” might become “application performance bottleneck causes and solutions.”

3. Retrieval: System searches for relevant chunks (as described above).

4. Live Web Search: If internal documents aren’t enough, some RAG systems can do live web searches. This helps them get current information from the internet.

5. Reranking22 and Filtering: Results are scored, filtered, and reranked for relevance.

6. Generation: LLM produces an answer grounded in all the gathered context.

This multi-step approach is what separates demo-quality RAG from production-quality RAG.

RAG Limitations

RAG is powerful, but it’s not a silver bullet…

Understanding its limitations helps you decide when to use it and when to look elsewhere:

Read more

• Share this post & I'll send you some rewards for the referrals.

Everyone is talking about AI agents, but only very few people are actually building them.

Plugging a chatbot into your website is just a UI update. The biggest opportunity is in agentic AI, and most people are missing it…

Onward.

The best way to build any app (Partner)

The real tax on vibe builders right now isn’t effort, it’s cost.

Every tool wants you to pay for its model usage and hosting forever. Orchids.app lets you bring your own and use tools/SDKs that already exist elsewhere.

Plug in your ChatGPT, Claude Code, Gemini, Copilot, GLM, or any API key you already pay for.

You keep control of the bill. And the best part is: you don’t get price locked because you shipped with Orchids. Deploy your code straight to Vercel with one click.

Get Started Today

(Use this discount code to get a onetime 15% off during checkout: MARCH15)

I want to introduce Fran Soto as a guest author.

Fran is a software engineer at Amazon. He’s responsible for most of the “Add to Cart” and other buttons you see in the search results of the Amazon marketplace, and now works at Ring.

Fran helps software engineers grow their careers. He teaches them productivity and the AI automation skills they need to grow to the next level.

To celebrate this collaboration, Fran is giving away his premium System Design Template completely free to all new subscribers. This template is the perfect tool to structure your architectural decisions before you write any code. You can practice filling it with the case studies and deep dives from this newsletter!

For the next few days, you can also claim a 25% discount on the StrategizeYourCareer annual subscription from the links in this newsletter. Whether you grab the annual discount or just want the free template, join today to become a productive engineer.

Check out his newsletter: StrategizeYourCareer.

Imagine you have just bought some furniture and now you have to assemble it…

The fixed set of instructions that comes in the box is a workflow. It is a linear path that works perfectly as long as everything goes right. It’s the equivalent of scripts and programming languages. But if you make a mistake and break a wooden part, the instructions cannot help you.

There is no recovery from that.

An LLM would be like having a customer support line available for you to call.

You can describe the broken piece, and they will give you a brilliant, empathetic answer. They might even explain why the wood snapped. But at the end of the call, you are still sitting on the floor with a broken piece. No matter how nice the other person is, they don’t have hands to help you with your broken piece.

You are the agent in this scenario.

You are the one with the tools, like a hammer, nails, glue, and a drill. You are the one with the brain to make decisions on how to best use those tools and how to recover from errors that were never in the initial instructions.

In this newsletter, I want to show you how to build AI agents, providing the tools and knowledge they need to do work on your behalf.

I will use an incident response agent as an example, but the principles of giving an AI both a brain and a set of hands apply to almost any process.

In this newsletter, you’ll learn:

• The differences between LLMs, workflows, and agents.

• Why the non-deterministic nature of AI is a good feature.

• A 10-step framework to evolve from manual work to autonomous agents.

• How and when to keep the human in the loop.

Let’s get started:

Why the lack of determinism in AI is good for reliability and self-healing

I see a lot of skepticism about AI because it is NOT as reliable as traditional code.

We are used to code that does exactly the same thing every time. However, I believe this lack of determinism is actually a good thing for certain tasks.

Humans are not as reliable as a machine, yet we are the ones who fix machines when they break. We prefer humans using machines because it brings the best of both worlds. Machines give us the speed and precision of determinism, while humans make things not go as expected.

An agent mimics this human ability to adapt to new situations. When a system fails in a way you did not anticipate, an agent can review the logs and adjust its approach.

It’s self-healing.

However, to get the most out of agents, instead of using the out-of-the-box generic tool, you need to create your own agents tailored to your use case.

How to build your first AI agent

AI companies provide us with generic agents that have planning capabilities and built-in tools. While these are useful, I do not think we want a generic assistant for specialized tasks.

They also offer integration points to our own memories and tools.

This is where the opportunity is. The same way a company would hire a software engineer rather than a “generic human”, we want that same level of expertise in our agents.

Building an agent is not a one-step process where you just flip a switch and everything works perfectly. It is an iterative process in which you gradually offload parts of your cognitive load to the machine while keeping a close eye on its performance.

I’ve followed this 10-step framework that starts with pure manual work and ends with a fleet of autonomous agents working together…

(This process starts with the way we have always worked: Solving it manually.)

Step 1. Do the work manually

The first step is always to do the work yourself…

I recommend outlining the steps as if you were documenting the runbook for someone else to follow:

1. You open the monitoring dashboard and check any metric outside the usual pattern.

2. You check the runbooks for troubleshooting info for that specific metric.

3. You query the error logs for the last 15 minutes, looking for requests that resulted in an HTTP 500 error.

Current state: At this stage, you are the bottleneck in the entire process. Every step requires your manual intervention and your attention. This is the baseline that we want to improve.

Step 2. Start using an LLM

Now that you have a solid manual process, you can use it as a guide for an LLM.

This moves us from manual labor to having an assistant who can analyze the data for you. You can prompt the AI by copy-pasting the logs from your terminal or the metrics from your dashboard.

I suggest paying close attention to the prompts you usually use.

• “Review these logs to find what error is happening: <copy-paste of logs>”

At this point, it’s important to notice the areas where the AI does not do what you want, and how you steer it back to the right path when it goes off track.

Current state: The AI can start analyzing patterns, but you still have to orchestrate everything and move data around manually.

Step 3. Add MCP Tools

At this stage, you have an AI partner helping with the analysis, but you are still stuck in the middle, acting as a bridge.

The next step is to stop copy-pasting data. You can use the Model Context Protocol (MCP1) to allow the AI to take actions for you. MCP is like HTTP for AI agents. It’s an API that enables the model to execute operations in other systems.

Once you connect to the right MCP servers and enable the MCP tools, your prompts will look like this:

• “Obtain the alarms that fired in the last 15 minutes. You must give me only the alarm name.”

• “Check the logs for requests ending with an HTTP 500 error. You must give me the message of the log, the error message, and the stack trace...”

The AI can get the incident details on its own!

Current state: The human can drive the entire process through the AI interface without switching between tabs and tools.

Step 4. Add Agent Skills

Up to this point, you still have to be an expert in incident response…

AI is useful for doing everything from one place. Now we’ll start looking at the notes we have been taking on how to improve the process and where AI usually goes wrong.

We have to onboard the AI to incident response the same way we’d onboard a junior engineer recently hired:

• Teach exactly how to use each tool.

• Teach how you want the work done.

We can combine guides and Standard Operating Procedures.

These provide step-by-step instructions for routine tasks. We’d write them in human language, using small scripts, to address a problem with determinism.

For example, if the MCP tools require a query as input, we can create the Skill of building a query. We’ll combine a markdown guide containing the fields from the logs with a few scripts to build queries from a template. This is much better than having the AI write complex queries every time. Plus, it reduces the hallucinations of fields that don’t exist in your logs.

We call these reusable pieces of logic Agent Skills.

Agent Skills are great because they’re loaded into the model ONLY when needed. The agent may have skills for metrics and logs. If we only ask about fired alarms, the agent ignores the log data. It just doesn’t need it.

This saves memory.

With agent skills, our interaction would be like this (Slash is the syntax used in IDEs like Cursor2 to force the agent to load a particular skill):

• /Metrics-and-alarm-management Find the alarms fired

• <LLM response with metrics information>

• /My-service-logs-management Find the logs with errors in these endpoints

• <LLM response with the logs information>

• /Code-expert Where in code is this log emitted?

• <LLM response with the code information>

Current state: We have now taught AI how we want certain tasks done, such as metrics and log information. The human is orchestrating the process to do things in order.

Step 5. Add memory

We have given our agent tools and domain expertise, but every time you start a new chat, it’s like the agent has amnesia. It knows how to solve an incident, but it doesn’t remember the one you fixed yesterday.

An agent needs context to be effective…

You can add simple short-term memory and long-term memory files to your system.

The short-term memory is built by design into the agent. This is essentially the AI’s ability to remember the previous messages in your current conversation. It helps it stay on track without you having to repeat yourself. When making an API call to get the model’s response, it sends the previous messages along with the new ones.

For an incident response agent, we also want some long-term memory.

We have added some memory thanks to the skills, but we can add more context by creating files that the AI can access.

Some time ago, each company had its own format, but now they have agreed on a standard called Agents.md3. This means the information inside Agents.md file will be read by the AI regardless of which AI it is.

Also, each implementation allows you to point to which folders/files to use for the agent’s context.

Current state: We still prompt AI the same way. It uses data from skills and logs. It also reads files to understand the system it is debugging. This helps the AI find past errors and the solutions that worked.

Step 6. Orchestrate with Agent SOPs

Up to this point, we have given our agent a brain, a set of hands through MCP tools, and a memory.

But you are still the one directing every move. While the agent is now more capable, it still requires you to decide manually what to use and when. This is the bottleneck that we solve with Agent SOPs.

Instead of prompting the AI to use a series of skills one by one, you can create a step-by-step process for the AI to follow. This is what’s called a Standard Operating Procedure (SOP) for your agent.

An Agent SOP is a file written in accordance with RFC 21194. This RFC defines key capitalized words (MUST, SHOULD, MAY, etc.) to specify requirement levels. Besides, it contains the steps in order.

I learned about this concept at work, as Amazon uses it in its Kiro5 and Strands6 agents. However, I’m also using Agent SOPs with Cursor and Claude Code7 easily. You just have to write the markdown file as an Agent Skill.

Now, instead of prompting step by step, we just execute our agent with that SOP. For example, if we wrap our SOP in a skill:

• /Incident-response-agent-sop Investigate the incident reported at JIRA-1234

Current state: This makes the agent much more autonomous. The human just invokes the right SOP, and the agent handles the rest. It’s like a high-level manager finding the right engineer to loop into a call.

Step 7. Define an agent

At this point, we have many MCP tools, skills agent SOPs that need to be configured and made available for our agent to work.

This is easy if this is your first agent, but when you have dozens of them, each using their own set of artifacts, things get hard…

We solved this problem in traditional software with dependencies. We declare which packages our software depends on, so it can run in different environments.

Now it’s time to declare those dependencies with a new artifact called an agent. This way, we’d have our incident response agent, and we can define other agents for other purposes.

This guarantees the AI has all the tools it needs, so you do not need to provide them manually. Currently, there isn’t a proper package manager to install all of these, but it’s just a matter of time…

Current state: We now have our agent, and we can delegate the entire incident response investigation to it. We just invoke it and wait for the results.

Step 8. Execute your agent periodically

Up to this point, we have a fully defined agent that knows exactly what to do, but it only moves when you tell it to.

You have built a great tool, but you are still the one holding the power button…

Now that we can delegate most work to AI, you may ask: Why am I still getting paged at 3 am just to invoke the agent and wait?

It’s time to set the AI agent to run autonomously…

I recommend using CLI tools to run your AI agent independently. We can start with a simple script that checks for work and runs the AI agent when found; otherwise, sleep for N minutes.

Current state: Now the human isn’t involved in triggering the agent, but it will find the alarm and the agent’s output together.

Step 9. Improve state management

The agent is now running on its own, but it’s still isolated on your local machine using basic loops. To make it a true part of your team, it needs to be integrated into your production environment…

The goal is to define an end-to-end workflow that triggers AI agents at the right time without human involvement. For example, a webhook from PagerDuty could automatically trigger the AI agent in Lambda.

For periodic reports, you may want to set up a cron8 job that triggers the AI agent at regular intervals.

The AI agent performs the investigation and then posts a summary of the root cause or a periodic report to a Slack channel. This means your agent is working while you sleep, even if it only does one specific thing well.

Current state: Now our agent is like a real engineer involved when an incident happens, or an engineer who has scheduled work in their calendar.

Step 10. Scale to multiple agents

At this stage, you have a specialized agent working 24/7.

But as your system grows, one agent can become overloaded with too many responsibilities. Just as you wouldn’t ask one engineer to handle every single microservice, we shouldn’t ask one agent to do it all.

The final step is to have multiple agents collaborating on the same incident. Each agent can have a different function based on its expertise:

• One agent gathers the raw logs while another checks for known infrastructure issues.

• Another agent could review the recent code deployments for errors.

There are different orchestration paradigms you can use, like sequential, routing, or swarms.

Even for the same function, there’s no limit to how much you specialize. If an incident response agent has too many details, you can create agents to check different parts of the system.

You need a bit of abstraction to manage your AI workforce, just as a manager coordinates a team of specialists. You just do the same: multiple specialist agents, and a manager that invokes each subagent.

You can use different triggers, such as schedules or events, to make agents collaborate via the file system. No need to start with complex agent-communication protocols; instead:

1. Start with an orchestrator agent triggered by events, schedule, or polling.

2. The orchestrator invokes other subagents that you have also defined. Most AI companies already implement subagents.

3. All agents write to the filesystem, so they see each other’s work.

I suggest running agents on your own computer and letting them work together by sharing files.

Current state: At this point, the AI performs complex work with multiple steps without your involvement. Humans are no longer the bottleneck.

We did it!

Now we have our autonomous AI agent for incident response.

I know it looks easier in diagrams than in practice, and the AI doesn’t work as expected. Let me also share some lessons from building AI agents.

Lessons from the trenches

I’ve spent a lot of time building these agents, and I want to share some practical advice that will save you hours of frustration:

• Make sure you have enough examples to build a robust agent. You wouldn’t want to automate a task that isn't frequent, so you're likely to have multiple instances to iterate on. I asked my team to handle all intakes of a service for a month. I’m sure your manager will be happy you're taking on more work.

• Iterate multiple times per step. Don’t move to the next step of the automation until you have solved several tickets or tasks to find the real patterns and failure modes.

• Don’t switch tabs while AI works. Watch the agent work live to catch its mistakes instead of letting it run in the background.

• Use tmux9 for persistent loops. Run your agents from your computer in a tmux session so they keep working even if you lose your terminal connection.

• Log everything for observability. Pipe all the agent’s output into a dedicated log file to give you the same level of visibility when you move to executing it 24/7.

• Refine prompts based on observations. Even after you consider the 10 steps done, new edge cases will arise. Make AI keep learning, either by updating it yourself or by letting the AI update its memory files between iterations.

Defining the role of the human in the loop

I want to be clear that this is not about all-human or all-AI systems…

I’ve focused the incident response agent examples on read-only tasks, such as checking alarms, metrics, and logs.

Each of us has to define where and when to involve humans for the highest leverage. You can have the agent triage the incident and suggest a fix, then wait for your approval and execution.

Maybe someone else wants to plug the AI into the codebase to create a PR with the fix and wait for a human to approve it and deploy the hotfix.

I still think we don’t want AI making changes in production without human oversight. I have read postmortems where AI ran unauthorized production operations in an AWS account without any human review.

If we think about the incident response process and a code change, there are many steps before we change something in the system…

My current stance on agents is to push the human to the right as much as possible. This is all about making the human do less work, but still overseeing. This is important because no one will ask the AI for ownership or accountability.

They’ll ask the humans for it!

TLDR: The Building Blocks of AI Agents

• Brain (LLM): The Large Language Model processes information, makes decisions, and determines which steps to take.

• Hands (Tools & MCP): They are the interfaces that allow the AI to interact with the real world. Using the MCP, an agent can execute code, query databases, or fetch logs instead of just “talking” back to you.

• Skills (Agent Skills & SOPs): They teach the model how to perform tasks. This is like a junior engineer following a runbook. It reduces errors and hallucinations.

• Memory (Context & Agents.md): This allows the agent to remember past actions. Short-term memory handles the current conversation. Long-term memory provides historical context through shared files. This prevents the agent from repeating mistakes.

• Manager (Orchestrator): Use an orchestrator to manage multiple specialized agents as you scale. One agent might find the error, while another suggests a fix, all working together toward a final result.

• Alarm clock (Cron job): It lets the agent run on a schedule without you having to manually start it.

• Workspace (Local filesystem): A common area where agents read and write. This allows them to collaborate on complex problems.

Closing Thoughts: The future of agentic work

It is often said that AI will not take your job, but someone using AI will.

I truly believe this.

But “using AI” does not just mean installing a new extension in your IDE. I have seen many people get frustrated when the AI does not do exactly what they want on the first try, thinking the AI is just dumb.

The people who succeed are the ones who build an army of AI agents to do the heavy lifting for them. All those AI companies are giving us default tools, but we are the ones who must adapt them to our specific needs.

I hope this guide was helpful and that you start building your own agents today.

👋 I’d like to thank Fran for writing this newsletter!

• Also subscribe to his newsletter, StrategizeYourCareer.

It will help you master productivity and the AI automation skills needed to thrive in your career, just like we learned in this post.

Remember to claim your free System Design Template by signing up today at StrategizeYourCareer. You can also take advantage of his special discount before it expires!

If you find this newsletter valuable, share it with a friend, and subscribe if you haven’t already. There are group discounts, gift options, and referral rewards available.

Subscribe now

Thank you for supporting this newsletter.

You are now 210,001+ readers strong, very close to 210k. Let’s try to get 211k readers by 21 March. Consider sharing this post with your friends and get rewards.

Y’all are the best.

Share

• Share this post & I'll send you some rewards for the referrals.

Reinforcement learning (RL) is making a comeback and becoming mainstream.

From humanoid robots and game-playing AIs that beat world champions to the LLMs you chat with every day, all are trained using RL, which lets them learn from experience and get better through feedback.

Contrary to many tutorials that can make it look so hard, RL is actually pretty intuitive, and you really don’t need a PhD to understand how it works.

In this newsletter, we will go through 21 key RL concepts to build a solid foundation from the ground up. These terms are all explained in plain language without using any fancy math equations.

Let’s start from the very beginning…

Want to reduce AI coding bugs before they hit production? (Partner)

AI is writing more production code than ever, but it also introduces more risk.

CodeRabbit’s research found that AI-generated code contains 1.7x more issues, including 75% more logic errors.

That’s exactly why you should grab a copy of the Ultimate Prompting Cheat Sheet.

Here’s what you get:

• Proven prompting techniques to reduce AI coding errors.

• Actionable workflows for AI-assisted development.

• Clear frameworks for safer AI-generated code.

• Practical examples you can apply immediately.

And many more!

These techniques reduce defects without slowing velocity, regardless of which AI coding tool you use.

Sign up and get access to the ultimate prompting cheat sheet right now:

GET INSTANT ACCESS

(Thanks to CodeRabbit for partnering on this post & sharing the ultimate prompting cheat sheet.)

I want to reintroduce Ashish Bamania as a guest author.

He’s a self-taught software engineer and an emergency physician. He is also the editor and primary author of the newsletters Into AI and Into Quantum.

The images used in this newsletter come from his books:

• LLMs In 100 Images

• Reinforcement Learning in 100 Images (upcoming)

Grab the book at 20% discount today with the discount code NEO20, pre-applied to the link above!

1. What is Reinforcement learning?

Let’s start with the definition of Reinforcement Learning (RL).

RL is a type of machine learning (besides Supervised and Unsupervised learning) that deals with an entity called an ‘Agent’, trying to learn to perform a task better in its ‘Environment’ through trial and error.

For example, a deer (Agent) foraging in a forest (Environment) to survive, avoiding being eaten by its predators.

We will soon move to more AI-related examples, I promise, but first, let’s better understand what the terms ‘Agent’ and ‘Environment’ mean.

2. Agent

An agent is the central entity in RL.

It is one that studies and interacts with its environment, makes decisions, takes actions, and learns from its outcomes.

In our previous case, we used a deer as an agent example, but from now on, we will discuss concepts assuming an LLM as the agent as well.

3. Environment

The environment is everything outside the agent with which it interacts.

The function of an environment is to:

• Get affected by an agent’s actions

• Change its state (or maybe hold its state constant) depending on the agent’s actions

• Give the agent a reward or punishment based on its actions, so that the agent can modify its intent the next time it takes an action

In the first case, the deer was functioning in a forest as its environment. Based on the deer’s actions (moving around in different directions), either the forest (environment) lets it progress towards food (reward) or it exposes it to a cheetah (punishment).

In the case of an LLM as an agent, everything outside it, such as the following, is part of its environment:

• User inputs

• System prompt

• Tools / APIs it can call

• System responses (tool results, API outputs, error messages)

• Context (documents, conversation history, files)

4. State

A State is a snapshot of an environment at a time. It is everything an agent sees at a given moment that it can use to make its next decision.

For a deer agent, it could be its current location in the forest, nearby predators, and the time of day.

For an LLM agent, it is all the context it has access to at a moment, which it can use to decide its next action.

But what’s action?

5. Action

Read more

• Share this post & I'll send you some rewards for the referrals.

• Block diagrams created using Eraser.

No time for small talk today:

Following is the third of a premium 3-part newsletter series... if you want to be successful with system design in 2026, then this newsletter is for YOU.

On with part 3 of the newsletter:

===

Some of these are foundational, and some are quite advanced. ALL of them are super useful to engineers building production-ready systems.

Curious to know how many were new to you:

77. Webhooks,

78. WebRTC,

79. CQRS,

80. Event Sourcing,

81. Service Discovery,

82. Circuit Breaker Pattern,

83. Bulkhead Pattern,

84. Strangler Fig Pattern,

85. Backend for Frontend.

86. Sidecar Pattern,

87. Service Mesh,

88. Observability,

89. Logging,

90. Metrics,

91. Distributed Tracing,

92. Correlation IDs,

93. Full-Text Search Engine,

94. Time Series Database,

95. Vector Database,

96. Materialized Views,

97. Query Optimization,

98. Connection Pooling,

99. Cache Stampede,

100. Cache Warming,

101. PACELC Theorem,

102. Security Secrets Management,

103. Role-Based Access Control,

104. Single Sign-On,

105. Checksums,

106. Bloom Filter,

107. B-trees and B+ trees,

108. LSM Tree,

109. Merkle Tree,

110. HyperLogLog,

111. Batch vs Stream Processing,

112. ETL Pipelines,

113. MapReduce,

114. Erasure Coding.

For each, I’ll share:

• What it is & how it works--in simple words

• A real-world analogy

• Tradeoffs

• Why it matters

Let’s go.

The best way to build any app (Partner)

Most “AI app builders” aren’t actually app builders. They are infrastructure middlemen. You live inside their box, choose from their secret stack, and force you to start over the moment you want to use a different database, payment processor, or tool.

With Orchids:

• You’re not locked-in to Supabase or Stripe.

• Not forced to spend credits, bring your own AI subscriptions with you.

• One-click deployment straight to Vercel

Build anything. From web, mobile, internal tools, browser extensions, scripts, and bots. Orchids.app is capable of building anything that you can put your mind to.

Get Started Today

Use this discount code to get a onetime 15% off during checkout: MARCH15

77. Webhooks

Webhooks are HTTP callbacks that allow one system to notify another system when an event occurs by sending a POST request to a configured URL.

Instead of constantly polling for updates, the source system pushes data to your endpoint when something happens.

You register a URL with the provider, and they call it when events trigger. Webhooks enable real-time, event-driven integrations between systems.

Analogy

A doorbell that alerts you when someone arrives instead of you checking the door every few minutes…

When a package arrives, the delivery person rings the bell, and you respond immediately. You don’t waste time repeatedly checking if anyone’s there.

Tradeoff

Webhooks eliminate polling overhead and provide near-instant notifications.

Yet they require exposing a public endpoint, handling retries when your server is down, securing the endpoint against unauthorized requests, and handling duplicate or out-of-order events.

The sender controls retry logic, and you can’t always guarantee delivery.

Why it matters

Integrating with payment providers like Stripe, receiving GitHub repository events, getting Slack notifications, building event-driven workflows, or any system needing real-time notifications from external services.

Essential for modern API integrations. Combine with message queues for reliable processing and idempotency for handling duplicates.

78. WebRTC

WebRTC (Web Real-Time Communication) is a technology that enables peer-to-peer1 audio, video, and data sharing directly between browsers without requiring servers to relay media.

After initial signaling through a server to establish the connection, media flows directly between peers.

WebRTC automatically handles:

• Getting through NAT and firewalls

• Encrypting the connection

• Adjusting video quality based on network speed (adaptive bitrate streaming)

It’s built into modern browsers, so users don’t need to install plugins.

Analogy

It is like two people being introduced by a mutual friend:

The friend shares their phone number, but once they start talking, the friend is no longer on the call.

Tradeoff

WebRTC offers very low latency, thus making it ideal for live conversations. It can also reduce server costs because media often bypasses backend servers.

But:

• It’s difficult to implement correctly

• You usually need a TURN server as a backup when a direct connection cannot be made

• Debugging network problems can be hard

• Some corporate networks block peer-to-peer connections

Why it matters

WebRTC powers:

• Video calls

• Screen sharing

• Voice chat in games

• Live collaboration tools

It’s used in Zoom, Google Meet, and Discord.

If you’re building a real-time app, you can use WebRTC directly. But consider managed services like Twilio or Agora so you do not have to run your own signaling and TURN servers.

79. CQRS

CQRS (Command Query Responsibility Segregation) means separating how your system writes data from how it reads data.

• Commands are used to change data. They use a write model optimized to handle updates.

• Queries are used to read data. They go to a separate read model designed for fast retrieval.

The read model is often eventually consistent, updated asynchronously from the write model using events.

Analogy

Imagine a restaurant with two separate counters:

One counter takes orders... it focuses on taking them correctly. The other counter only handles pickups. It focuses on quickly delivering food to customers.

Both handle the same orders, but each serves a different purpose.

Tradeoff

Benefits:

• Scale reads & writes separately

• Design different data models for reading and writing

• Create specialized read models from the same write model

Drawbacks:

• Adds architectural complexity

• Read side may not reflect changes immediately

• Requires event handling or synchronization between models

• Makes the system harder to understand and maintain

It is usually too complex for simple CRUD applications.

Why it matters

CQRS is useful when:

• Your system has very different read & write workloads

• Reads significantly outnumber writes

• You need many views of the same data

• Domain & business logic is complex

It’s often used together with Event Sourcing.

But a single model for both reads and writes is usually enough for simple systems.

80. Event Sourcing

Event Sourcing is a way of storing data in which you record every change that happens in the system, rather than only saving the latest state.

Each change gets captured as an immutable event.

You can rebuild the current state by replaying events from the beginning or from a snapshot. You can also rebuild the state at any point in time and audit the complete history of changes.

Analogy

Think of a bank that records every transaction in a ledger:

Instead of only storing your current balance, bank keeps a list of all deposits and withdrawals. Your balance can always be calculated by adding up all those transactions.

Tradeoff

Benefits:

• You get a history of all changes

• Easy to audit what happened and when

• You can rebuild the system state at any time

• You can create new views or reports from past events

Drawbacks:

• Harder to design & implement

• Events are immutable, so fixing mistakes requires new corrective events

• Getting the current state often requires projections or snapshots

• Event formats may change over time, which makes schema evolution difficult

• Event storage grows unbounded without snapshots

Why it matters

• Audit-heavy domains such as financial systems

• Systems that need historical queries

• Systems that need many views built from the same data

It’s often used together with CQRS.

81. Service Discovery

Service discovery allows services within a system to automatically find and communicate with each other, eliminating the need for fixed IP addresses.

When a service starts, it registers itself in a service registry. The registry tracks where each service instance is running. When another service wants to call it, it asks the registry for the instances and connects to one of them.

Common tools: Consul, etcd, and Eureka.

Analogy

Think of it like a phone directory:

Instead of remembering everyone’s phone number, you look them up in the directory. If someone changes their number, the directory gets updated so people can still reach them.

Tradeoff

• Requires health checks

• Adds infrastructure complexity

• It can become a critical dependency

• Service lookups can add a small amount of latency

• Cached results might sometimes point to instances that are no longer healthy

Why it matters

Service discovery is important in environments where services move or scale frequently:

• Microservices architecture

• Container platforms like Kubernetes

• Cloud systems with auto-scaling

It allows services to locate and communicate with each other even when their IP addresses or instances change dynamically.

82. Circuit Breaker Pattern

Circuit breaker pattern protects a system from repeatedly calling a failing service.

A circuit breaker sits between a service and the dependency it calls (for example, another service, API, or database). It monitors failures such as errors or timeouts.

If failures pass a threshold, the circuit breaker “opens” and stops sending requests to the failing service. Instead, it immediately returns an error or a fallback response.

After a set time, the circuit breaker moves to a “half-open” state. It allows a small number of test requests.

• If those succeed, the circuit “closes” and normal traffic resumes.

• If they fail, the circuit “opens” again.

Analogy

It works like an electrical circuit breaker in a house:

If too much current flows, the breaker cuts power to prevent damage. After the problem gets fixed, you reset the breaker and power flows again.

Tradeoff

Benefits:

• Prevents cascading failures in distributed systems

• Reduces load on failing services so they can recover

• Fails fast instead of waiting for long timeouts

• Protects system resources

Drawbacks:

• Adds complexity to service calls

• Requires tuning thresholds & timeout values

• Can open unnecessarily if thresholds are set incorrectly

• Needs monitoring to detect when circuits are open

Why it matters

It’s useful when systems make network calls that can fail, such as:

• Microservices calling other services

• Services calling external APIs

• Systems that depend on remote databases or infrastructure

It helps prevent one failing dependency from causing failures across the entire system.

Common libraries: Resilience4j & Polly.

Reminder: this is a teaser of the subscriber-only post, exclusive to my golden members.

When you upgrade, you’ll get:

• Full access to system design case studies

• FREE access to (coming) Design, Build, Scale newsletter series

• FREE access to (coming) popular interview question breakdowns

And more!

Get 10x the results you currently get with 1/10th the time, energy & effort.

Get Instant Access

Read more

• Share this post & I'll send you some rewards for the referrals.

In 2021, I applied to Amazon and failed the system design interview.

I knew something was off during the interview, but I couldn’t figure out what. At that time, I was a Software Engineer at Trainline on the Search/Checkout Web Services team, building software for over 100 million customers across Europe. I could code, I understood distributed systems, and I had shipped real features.

On paper, everything looked fine.

Looking back, I can see exactly what I missed…

I jumped straight into designing without clarifying requirements, never validated my assumptions, and never asked what actually mattered. I just started drawing boxes and arrows. I felt great in the moment and got smiles from the interviewer, but ultimately I failed.

Fast forward today…

I’m an L5 Software Engineer at AWS, who has contributed to CloudWatch, EC2, and Regional Services. I conduct system design interviews for candidates. That journey from failure to interviewer taught me something important: most engineers who fail these interviews aren’t bad engineers. They just don’t have a framework for turning vague problems into conversations.

This newsletter is everything I wish I knew before my 2021 interview.

I’m going to share what you actually need to pass: the framework, the pacing, the collaboration checkpoints, and the mindset that separates candidates who pass from those who don’t.

Before we get into frameworks and strategies, let’s explore why these interviews feel so different from everything else you do as an engineer.

Onward.

Need public web data, without scraper headaches? (Partner)

SerpApi turns search results into predictable JSON with built-in scale, location options, and protection from blocks.

That’s why engineers use it to ship:

• AI applications

• Product research

• Price tracking

• SEO insights

All without maintaining scrapers or infrastructure.

Try For Free

(I’d like to thank SerpApi for partnering on this post.)

I want to introduce Abdirahman Jama as the guest author.

He’s a Software Engineer at AWS, where he’s contributed to production services like CloudWatch and EC2. He also conducts system design interviews for engineering candidates.

Before AWS, he worked at Trainline and BT, building software for over 100 million customers across Europe. He was an AWS Community Builder and served as a coding instructor at Code First Girls in collaboration with Skyscanner.

He regularly shares free software engineering tips, system design breakdowns, and career advice for engineers breaking into big tech.

You can connect with Abdirahman here:

• LinkedIn

• Instagram

• Newsletter

Why System Design Interviews Feel Hard

In coding interviews, the problem is clear: “Write a function that does X.”

You can write the function, add some tests, and confirm whether it works. The feedback loop is immediate and obvious.

When you’re building systems at work, you have context.

You know your users, your constraints, and the internal components you can integrate. Also, you have teammates to bounce ideas off and challenge assumptions. Plus, you have time to assess different options, weigh pros/cons, and make informed decisions. And most importantly, you can use Google.

System design interviews strip all of that away:

You’re given a vague problem like “Design ChatGPT” and roughly 30-40 minutes to propose a solution. You can’t search for anything. Yet somehow you’re expected to show enough depth and breadth to cover important concepts.

Now that you understand the challenge, let’s look at what’s actually being evaluated. Knowing the scoring criteria changes how you prepare.

What Interviewers Look For (& Mental Model You Need)

Here’s what I learned as an interviewer: we’re not looking for the “right answer.”

There isn’t one! We’re evaluating whether you can think “systematically” about ambiguous problems…

Here are five dimensions most companies will assess you on:

1. Communication and Collaboration

• Do you ask clarifying questions?

• Do you validate assumptions throughout the interview?

• Do you treat the interviewer as a partner, or do you just talk at them for 40 minutes straight?

This is crucial as it shows if you’d be a good teammate.

2. Structured Thinking

• Can you break down a vague problem systematically?

• Do you tackle things logically or jump around between different database choices, caching strategies, and API design?

Strong candidates show how they can organise their thoughts even when the problem is ambiguous.

3. Technical Judgment

• When you pick a technology, a database, a cache, or a message queue, can you explain why?

• Can you walk through the trade-offs? For example, why DynamoDB over PostgreSQL for this specific use case?

This shows you choose tools based on requirements, not habit.

4. Practical Depth

• Do you think about failure modes, scale, and operations?

• Or do you only design the happy path?

This is where we spot engineers who’ve shipped production systems and those who haven’t. Real systems fail. Databases go down. Networks partition. Strong candidates anticipate and design for this.

5. Time Management

• Can you pace yourself to cover breadth and depth in 40 minutes?

• Do you spend a long period covering a specific area?

It’s always worth working with your interviewer to understand if there’s an area of interest for them, or you should propose what you’ll focus on. Get alignment.

Between my failed 2021 interview and the ones I passed, the biggest shift was adopting a mental model, a framework that works for any system design problem. This framework keeps people organised and shows how to solve ambiguous problems.

Let’s walk through each step…

Framework: 5 Simple Steps to System Design Interviews

Step 1: Clarify the problem

• Goal: Understand the problem before you design anything.

• Estimated time: 5 minutes

Start by understanding what’s in scope and what isn’t:

• Are you focusing on Feature A or Feature B, or both?

• Are you targeting a specific platform?

• How many users are you building for?

• Where are they located?

These aren’t just theoretical questions…

The answers can fundamentally change your design.

Functional Requirements (What Are We Building?)

Be specific about what your system needs to do:

• What are the main use cases?

• Who are the users?

• What core features are you building?

• What are you explicitly leaving out of scope?

Non-Functional Requirements (What Are the Constraints?)

This is where numbers matter:

• How many daily and monthly users do you have?

• How many requests per second?

• What’s the read-to-write ratio?

• What are the latency requirements? Is 200ms acceptable, or do we need sub-100ms?

• What’s an acceptable downtime?

These constraints will drive every technical decision you make…

A system for 1000 users looks completely different from one serving 200 million daily active users.

TIP: Check with your interviewer and ask if the scope and requirements make sense.

Step 2: Define Core Data & APIs

• Goal: Model what you’re storing and how clients interact with it.

• Estimated time: 10 minutes

Data Model

Identify the core entities and their attributes:

• What are the main “things” in your system? Think about how they relate to each other.

• Define your primary keys, foreign keys, and the indexes you’ll need for common queries.

Data Characteristics

Understand the shape of your data:

• How big is each record?

• Is the data append-only or mutable?

• What’s the growth rate?

• How will the data be queried?

  • Are you doing point lookups by ID or complex joins?

  • Which data needs strong consistency, and what can be eventually consistent?

  • What absolutely cannot be lost?

API Contracts

Define your main endpoints with clear request and response formats. Think about pagination strategies, error handling, rate limiting, and versioning.

Database Choice

Pick a technology and explain why.

Don’t just say: “Use PostgreSQL.” Justify your choice based on the requirements, discuss alternatives, and explain why you chose one over another.

TIP: Check again with your interviewer: “Does this data model and API structure make sense for the requirements?”

Step 3: High-Level Architecture

• Goal: Design system components and show how data flows through them.

• Estimated time: 15 minutes

You should spend most of your interview time here.

Start by identifying your core components: load balancers, API servers, a cache layer, and queues. Keep this list focused and include only what your design needs.

Draw Data Flow

Walk through how a request moves through your system.

From the client, through the load balancer, to the API server, into the cache or database, and back. Do this for both reads and writes.

Discuss Asynchronous Workflows Where Appropriate

When appropriate, explain why you’d decouple work into background processing. Also discuss trade-offs: faster responses, but eventual consistency.

Caching Strategy

Be specific about what you’re caching, for how long, and how you handle invalidation. Plus, explain the trade-offs between freshness and performance.

CAP Theorem Considerations

For different parts of your system, choose between consistency, availability, and partition tolerance. Explain why each part of the system may need a different approach.

Step 4: Bottlenecks, Scale, and Reliability

• Goal: Show you understand what breaks at scale and how to handle failures.

• Estimated time: 5 minutes

Don’t just design the happy path; show you think about failure and scale.

Identify Bottlenecks

Where will your system struggle under load?

Name the component, quantify the limit, and propose a mitigation.

Single Points of Failure

What happens if a component goes down? Identify the risks and explain the mitigation.

Failure Modes and Graceful Degradation

• Cache failure: fallback to a slower but functional path

• Message queue backed up: serve stale data instead of failing requests

• Network partition: degrade gracefully rather than returning errors

Redundancy

• Database: multi-AZ replicas for high availability

• Application servers: auto-scaling across availability zones

• Cache: distributed cache with replication

Monitoring, Logging, and Alerting

This is my favourite part of the interview. It shows you’re thinking about operations and observability--one of the clearest signals separating junior from senior engineers.

You should cover:

• API latency at p50, p95, and p99

• Error rates and thresholds

• Queue depth and backlog alerts

• Structured logs with request IDs for distributed tracing

TIP: Ask your interviewer: “Are there specific failure scenarios you’d like me to explore further?”

Step 5: Tradeoffs and Extensions

• Goal: Summarise key decisions and discuss what you’d build next.

• Estimated time: 5 minutes

Wrap up by summarising the key trade-offs you made. What did you prioritise, and what did you sacrifice? Be explicit about the why.

Then discuss extensions. What would you add if you had more time? This shows you’re thinking beyond the immediate problem. That’s the framework.

Next let’s see it in action with a real example:

Connect with Abdirahman here:

• LinkedIn

• Instagram

• Newsletter

He regularly shares free software engineering tips, system design breakdowns, and career advice for engineers breaking into big tech.

Reminder: this is a teaser of the subscriber-only post, exclusive to my golden members.

When you upgrade, you’ll get:

• Full access to system design case studies

• FREE access to (coming) Design, Build, Scale newsletter series

• FREE access to (coming) popular interview question breakdowns

And more!

Subscribe now

Access the interview example and quick reference framework 👇

Read more

• Share this post & I’ll send you some rewards for the referrals.

You’ve spent three days crafting the perfect prompt.

You’ve added role-playing instructions, sprinkled in a few-shot1 examples, and even included that magic phrase everyone swears by, “Let’s think step by step”.

Your AI travel assistant is ready for production.

Then your first real user types: “Book me a hotel in Paris for the DevOps conference.”

Your agent cheerfully books the Best Western in Paris, Kentucky.

This isn’t a prompting failure. It’s an architecture failure. And if you’ve been building AI agents in production, you’ve probably hit this wall where no amount of clever phrasing can make up for missing information.

The solution isn’t to write better prompts. It’s engineering a better context!

Onward.

Unblocked: The context layer your AI tools are missing (Partner)

Give your agents the understanding they need to generate reliable code, reviews, and answers. Unblocked builds context from your team’s code, PR history, conversations, documentation, planning tools, and runtime signals. It surfaces the insights that matter so AI outputs reflect how your system actually works.

See How It Works

(Thanks, Unblocked, for partnering on this post.)

I want to introduce Priyanka Vergadia (“Cloud Girl”) as the guest author.

She is a cloud and AI expert who has worked at two of the hyperscalers: Google and Microsoft. Check out her newsletter and social media:

• Cloud Girl Bytes (newsletter)

• LinkedIn

• Twitter

• YouTube

• Website

• Instagram

It’ll help you quickly master cloud and AI.

This newsletter will walk you through the shift from Prompt Engineering (optimizing what you write) to Context Engineering (architecting what the model receives).

By the end, you’ll understand why this distinction matters and how to implement it in your own systems. But first, let’s make sure we’re speaking the same language…

Key Concepts: A Quick Primer

Whether you’re just getting started with AI development or looking for a refresher, here are the foundational terms we’ll use throughout this post:

LLM (Large Language Model)

Think of this as the “brain” of your AI application.

Models like GPT-4, Claude, or Llama2 are trained on massive amounts of text to understand and generate human-like language. They don’t “know” things the way you do—they “predict” what text should come next based on patterns they’ve learned.

Prompt

The instructions or text you send to an LLM.

If the LLM is an actor, the prompt is the script you hand them. A prompt might be as simple as “Summarize this email” or as complex as a multi-paragraph system message with examples and constraints.

Context

All the information the LLM has access to when generating a response.

This includes your prompt, as well as any additional data you provide: conversation history, retrieved documents, user profiles, tool outputs, and more. Context is everything in the model’s “working memory” at the moment of inference.

Context Window

The maximum amount of text (measured in “tokens”—roughly 3/4 of a word) that an LLM can process at once.

Think of it as the model’s short-term memory capacity. GPT-4 Turbo has a 128K token window; Claude 3 offers up to 200K. Anything beyond this limit simply won’t be seen by the model.

Inference

The moment when the model actually generates a response.

Everything you send—prompts, context, conversation history—gets processed at inference time. It’s the “go” moment.

RAG (Retrieval-Augmented Generation)

A technique where you fetch relevant documents or data from an external source (like a database or search engine) and inject them into the context before inference. This gives the LLM access to information it wasn’t trained on—like your company’s travel policy or today’s flight prices.

Agent

An AI system that can take actions, not just generate text.

Agents typically operate in loops: they reason about a task, call tools3 (such as APIs or databases), observe the results, and iterate until the task is complete. Your travel booking assistant is an agent.

Hallucination

When an LLM confidently generates information that isn’t true.

Without proper context, models fill in gaps with plausible-sounding but incorrect details—like assuming “Paris” means Kentucky when the user clearly meant France.

With these concepts in hand, let’s explore why traditional prompt engineering often falls short—and what to do instead.

1. The Micro-Optimization: Prompt Engineering

The “Static Script”.

If you’ve been working with LLMs, you already know Prompt Engineering. It’s the craft of writing effective instructions for a model, and for the past two years, it’s been the primary way developers have tried to improve AI output.

Think of an LLM as a talented improv actor:

Prompt Engineering is the script you hand them before the curtain rises. It defines their character, the scene's tone, and the rules of engagement. A good script can make a significant difference in performance.

The standard developer toolkit for prompt engineering includes:

• Role Assignment – Setting the persona with instructions like “You are an expert travel agent.” This primes the model to use vocabulary and behaviors associated with that role.

• Few-Shot Examples – Providing input/output pairs demonstrating the exact format you want. For example, showing the model three examples of properly formatted JSON responses before asking it to generate one.

• Chain of Thought (CoT) – Instructions like “Let’s think step by step” that encourage the model to show its reasoning. This dramatically reduces errors in logic and math problems by forcing the model to “work through” the problem rather than jumping to an answer.

• Constraint Setting – Hard limits on output, like “Limit your response to 50 words,” or “Respond only with valid JSON.”

These techniques are valuable and worth mastering.

But they share a fundamental limitation that becomes apparent when you move from demos to production.

The “Paris, Kentucky” Failure

The limitation of Prompt Engineering is that it is static.

Let’s return to our travel agent. The user says: “Book me a hotel in Paris for the DevOps conference.”

An agent relying only on prompt engineering sees the words “Paris” and “DevOps.” It has no access to external data, so it does what LLMs do when they lack information: it makes a probabilistic guess. There’s a Paris in France, sure, but there’s also a Paris in Kentucky, Texas, Tennessee, and several other states. Without additional context, the model has no way of knowing which one you mean.

The result?

Your user gets booked into the Best Western in Paris, Kentucky.

No amount of clever prompt phrasing can fix this. You could write “Always book hotels in major international cities”—but what about the user who actually wants Paris, Texas? The problem isn’t the prompt. The problem is that the model lacks three critical pieces of information:

1. User lives in London (suggesting international travel is more likely)

2. DevOps conference this year is actually in Paris, France

3. The user’s company policy requires booking Marriott properties under €200/night

This is where the paradigm shifts.

We need to move from optimizing static scripts to dynamically assembling the right information at runtime. We need Context Engineering.

2. The Macro-Architecture: Context Engineering

The “Dynamic Set Design”.

If Prompt Engineering is handing a script to your improv actor, Context Engineering is building the entire stage, providing the props, dressing the set, and handing them a dossier about their scene partners—all before they speak a single line.

Context Engineering is the programmatic assembly of the “Context Package”—the exact bundle of tokens sent to the LLM at inference4 time. It’s not about writing a better sentence; it’s about building an Orchestrator Loop that fetches the right information from multiple sources before the model ever generates a response.

This is a systems engineering challenge, involving data pipelines5, vector databases6, state machines, and API integrations.

The traditional flow looks like this:

User Input → LLM

The context-engineered flow looks like this:

User Input → [Memory + RAG + State + Tools] → Assembled Context → LLM

Let’s break down each component of this architecture.

The Four Pillars of Context Engineering

1. Memory Management (The Brain)

Here’s a naive approach that doesn’t scale: dump the entire conversation history into the context window every time.

With a 128K token context window and $10/million tokens, this gets expensive fast—and it significantly slows response time (a metric called Time to First Token, or TTFT).

Smart memory management means being strategic about what you include:

Short-term memory

A rolling window of the last N conversations is used to maintain immediate context. The user said, “Actually, make it a window seat”. The model needs to remember they were just discussing flight preferences.

Long-term memory

Persistent facts about the user, retrieved semantically. Instead of storing every conversation verbatim, you extract and index key facts: “User location: London”, “Seating preference: Aisle.” At inference time, you query this memory store to retrieve only what’s relevant to the current task.

Travel Agent Example:

When the user asks about booking a hotel, the system retrieves “User: London-based” and “Preference: Aisle seat” from long-term memory, while keeping the last 3 conversation turns in short-term memory.

2. RAG (Retrieval Augmented Generation)

RAG is your connection to “ground truth”—information that exists outside the model’s training data. This might be company policies, product documentation, real-time pricing, or any other external data source.

A robust RAG implementation typically combines:

Hybrid Search

Combining traditional keyword search (BM25)7 with semantic vector search8. Keyword search catches exact matches (“policy ID 998”); semantic search catches conceptual matches (“rules about hotel spending”).

Smart Filtering

You don’t want to retrieve the entire company handbook—that would blow your token budget. The context engine must extract only the relevant chunks, ranked by relevance to the current query.

Travel Agent Example:

When the user mentions booking a hotel, the retrieval system queries the corporate travel policy database and pulls the specific snippet: “Max Hotel Spend: €200/night, Approved Chains: Marriott, Hilton.” This grounds the agent’s booking decisions in the company's actual rules.

3. State Management (The Workflow)

Unlike simple chatbots, agents are stateful—they operate within multi-step workflows where sequence matters.

You can’t book a hotel before you’ve searched for options. You can’t confirm a reservation until the user approves the price.

The LLM needs to know where it is in this business process. State management includes:

Workflow Tracking

What phase are we in? Discovery, Search, Selection, Booking, Confirmation?

Progress Indicators

What constraints have already been satisfied? What’s still needed?

Travel Agent Example:

The system injects a state object like:

{
  "Phase": "Booking",
  "Status": "Searching",
  "Constraints_Met": ["dates", "location"],
  "Constraints_Pending": ["user_approval", "payment"]
}

This prevents the agent from accidentally jumping ahead to book before search results are reviewed.

4. Tool Access (The Hands)

The LLM is a reasoning engine, not a database.

It can’t check flight prices, look up conference locations, or execute bookings on its own. We must give it “hands”—well-defined interfaces (APIs) it can call.

Tool integration involves:

Function Schemas

JSON definitions of available tools.

For example:

search_conference_details(name: string),
search_hotels(location: string, check_in: date, max_price: number),
execute_booking(hotel_id: string)

The model learns what tools exist and what parameters they accept.

Orchestrator Execution

When the model decides to call a tool, it outputs a structured request.

The Orchestrator intercepts this request, executes the actual code, and returns the result to the context for the next inference step.

Handling Failures

Tools don’t always work perfectly. A robust context engine must handle failure scenarios gracefully:

Timeouts:

If a tool call takes too long (e.g., an API doesn’t respond), the system might retry with exponential backoff, try an alternative tool, or inform the user of the delay.

Errors:

If a tool returns an error (e.g., “No hotels found matching criteria”), the agent should recognize this and either adjust parameters or ask the user for clarification.

Ambiguous Results:

If a tool returns unclear or partial data, the agent might call additional tools to verify, request human confirmation, or explicitly note the uncertainty to the user before proceeding.

Travel Agent Example:

The agent calls:

search_conference_details(”DevOps Conference”)

…and receives:

{ “Location”: “Paris, France”, “Dates”: “Oct 12-15” }

Now it knows exactly which Paris the user means. If this API times out, the system retries twice, then asks the user: “I’m having trouble looking up the conference details. Can you confirm the location?”

When Context Exceeds Capacity: Prioritization

Even with large context windows, you’ll often have more relevant information than you can fit. When the assembled context exceeds your token budget, the system must prioritize.

Here’s a typical priority hierarchy:

1. System Instructions (always included) – The core behavioral rules and role definition never get cut.

2. Current State (highest priority) – Where are we in the workflow? This prevents the agent from taking nonsensical actions.

3. Active Tool Outputs (high priority) – Results from the most recent tool calls are essential for the current reasoning step.

4. Retrieved Policies/Documents (high priority) – Compliance rules and ground truth data that constrain valid actions.

5. User Profile/Long-term Memory (medium priority) – Persistent facts about the user that improve personalization.

6. Conversation History (lower priority) – Earlier turns can be summarized or dropped when space is tight. Keep the most recent turns intact.

7. Few-shot Examples (lowest priority) – These can often be reduced or removed once the model is performing well on the task.

Think of it this way: if your agent forgets a few-shot example, the output format might drift slightly. If it forgets a compliance policy, it might book an unauthorized hotel. If it forgets its current state, it might try to book a hotel that doesn’t exist.

Prioritize accordingly.

3. The Solution: “Runtime Prompt”

When you move from Prompt Engineering to Context Engineering, the actual text sent to the model (Runtime Prompt) is dynamically generated.

Here is what the “Real” System Architecture sends to the LLM for Travel AI Agent Example:

*** SYSTEM INSTRUCTIONS ***

You are an Expert Travel Agent.
Review the memory, policy, and state below before answering.


*** STATE ***

Phase: Booking
Status: Searching


*** MEMORY & USER PROFILE ***

- User Location: London, UK
- Preference: Aisle seats
- History: Prefers Europe for conferences


*** RAG / RETRIEVED POLICY ***

- Corporate Policy ID: 998
- Max Hotel Spend: €200
- Approved Chains: Marriott, Hilton


*** TOOL OUTPUTS ***

> search_conference_details(”DevOps Conference”)
  Result: “Location: Paris, France. Dates: Oct 12-15”


*** USER INPUT ***

“Book me a hotel in Paris for the DevOps conference.”

The Result

Because the context was engineered before inference, the model now has everything it needs to make a correct decision:

• Disambiguation: The tool output explicitly clarifies “Paris, France.” No ambiguity remains.

• Compliance: The RAG retrieval enforces the €200/night limit and approved hotel chains. The agent won’t even consider the €400/night boutique hotel.

• Reasoning: With all constraints visible, the LLM selects the Paris Marriott Champs-Élysées (€190/night) instead of making a blind guess.

This is the power of Context Engineering: transforming the LLM from a probabilistic text generator into a reliable reasoning engine constrained by real-world data.

Understanding the Tradeoffs

More context isn’t free. As you implement these systems, be mindful of the costs:

Token Costs: Each additional token you send to the model costs money. At $10 per million input tokens, a 10K token context package processed 100,000 times costs $10,000. The prioritization hierarchy discussed earlier becomes essential at scale.

Latency: Larger context windows increase Time to First Token (TTFT). Users waiting 3+ seconds for a response may disengage. You’ll need to balance comprehensiveness with responsiveness.

System Complexity: Memory stores, RAG pipelines, state machines, and tool integrations all require maintenance, monitoring, and debugging. Every component is a potential point of failure.

Retrieval Quality: Bad retrieval is worse than no retrieval. If your RAG system surfaces irrelevant documents, you’re paying token costs to confuse the model. Invest in retrieval quality metrics and continuous improvement.

The right approach is iterative: start with the minimum viable context, measure what’s missing when failures occur, and expand strategically.

Closing Thoughts

Context Engineering represents a fundamental shift in how we build AI applications.

It moves us from treating LLMs as chatbots that need cleverer instructions to treating them as reasoning engines that need better inputs.

The investment is real, but the payoff is production-grade reliability that prompt tweaking alone can never achieve.

👋 I’d like to thank Priyanka for writing this newsletter!

Also subscribe to her newsletter, Cloud Girl Bytes, and follow her on LinkedIn, Twitter, YouTube, or the social network of your choice. It’ll help you quickly master cloud and AI.

I launched Design, Build, Scale (newsletter series exclusive to PAID subscribers).

When you upgrade, you’ll get:

• High-level architecture of real-world systems.

• Deep dive into how popular real-world systems actually work.

• How real-world systems handle scale, reliability, and performance.

• 10x the results you currently get with 1/10th of your time, energy, and effort.

👉 CLICK HERE TO ACCESS DESIGN, BUILD, SCALE!

If you find this newsletter valuable, share it with a friend, and subscribe if you haven’t already. There are group discounts, gift options, and referral rewards available.

Subscribe now

Thank you for supporting this newsletter.

You are now 210,001+ readers strong, very close to 210k. Let’s try to get 211k readers by 7 March. Consider sharing this post with your friends and get rewards.

Y’all are the best.

Share

• Block diagrams created using Eraser.

• Share this post & I'll send you some rewards for the referrals.

• Block diagrams created using Eraser.

Why should we care about uptime? And what exactly is high availability?

Downtime is expensive…

A payment system that goes down during peak shopping hours bleeds revenue. A hospital record system that crashes mid-shift puts patients at risk. An app going offline for ten minutes triggers trending hashtags from angry users.

Downtime is never abstract…

It translates into lost money, lost trust, and sometimes even legal penalties. Some of it can never be recovered. High availability started long before the cloud. In the 1960s, defense and finance systems had to run nonstop. Those engineers designed machines that could keep working even when parts failed. When the internet arrived, that same discipline moved online. Banks, retailers, and payment networks learned that a brief outage can erase months of profit.

With the widespread use of technology, the expectation of “always on” has never been greater. Now, uptime is not a luxury but a baseline.

The goal never changed: build systems that keep running when the world shakes.

So failures will happen. No matter how hard we try, we can’t avoid them. Hardware burns out, networks drop packets, software engineers create bugs. High availability (HA) is about absorbing those failures behind the scenes…it’s about the service being available regardless of failures.

People don’t think about their car tires until one goes flat.

High availability is having a spare one in the trunk. After we replace them, we can drive again. We can’t always control why the tire got flat, but we can carry a spare one. This is the core idea of high availability.

Now let’s put some numbers to it.

Onward.

The best way to build any app (Partner)

Treat your app like an Orchid: A beautiful flower that needs sunlight and a bit of water 🌸

Most “AI builders” make you grow your app in their pot. Same stack. Same limits. Same rules. And on their databases.

Orchids is different:

• It’s your build space, set up your way.

• Build anything, Web app, mobile app, Slack bot, Chrome extension, Python script, whatever.

• Bring your own AI subscriptions so you’re not paying twice.

• Plug in the database you already use and trust.

• Use any payment infra you want.

Try Orchids.app and build it the way you were meant to:

Get Started Today

(Thanks, Orchids, for partnering on this post.)

Use this discount code to get a one time 15% off during checkout: MARCH15

Fundamentals

HA means keeping a system running even when parts of it fail.

The higher the availability, the less impact each individual failure has. To manage HA, engineers use SLAs, SLOs, and SLIs. These turn vague ideas like “keep it up and running” into numbers we can measure:

• Service Level Agreement (SLA): Contract with customers about service performance.

  • This contract keeps a record of what the service provider promises to deliver

  • There are penalties or costs if the contract is not respected

  • In HA, this is an agreement about how much downtime is acceptable

  • For example, “our app will be online 99.9% of the time. If not, we’ll give your money back.”

• Service Level Objective (SLO): Specific internal goal for the service performance.

  • This is the target that internal teams are trying to hit with a desired metric

  • Your SLA is the minimum you promise customers; your SLO is the better performance you target internally

  • For example, if the SLA is 95% uptime, the SLO might be 98% to leave a 3% safety margin

• Service Level Indicator (SLI): Metric used to measure service performance.

  • Without measuring service performance, we can’t know if we’re hitting the targets

  • These metrics should reflect the SLO and SLA

  • For example, “Percentage of failed requests.”

Let’s illustrate it with an example:

• A restaurant promises customers that their food will be delivered within 20 minutes of ordering (SLA).

• The kitchen aims to finish orders in 15 minutes to stay ahead (SLO).

• They track the average order completion time (SLI).

These targets get expressed in “nines of availability”.

One nine means 90% uptime; two nines mean 99%; and so on…

Each extra nine sounds small, but cuts downtime by a ton. For example, 99% uptime allows over 3 days of outage a year, while 99.9% (“three nines”) allows only about 8 hours. Every nine added costs more. It needs better hardware, more redundancy, and more monitoring.

The closer you aim for perfect uptime, the more effort and money it takes to maintain it.

When failures occur, recovery metrics help us measure how quickly and effectively we can recover. Here are the most important ones:

• Recovery Time Objective (RTO)

  • How fast should the system recover from failure? How long can it be down for?

  • Larger RTO means more downtime is acceptable; smaller RTO means less downtime

  • For example, an RTO of 10 minutes means that the system should be able to recover within 10 minutes of failing

• Recovery Point Objective (RPO)

  • To what point in time does the system recover? How much data loss is acceptable?

  • Larger RPO means more data loss, smaller RPO means less

  • For example, an RPO of 5 minutes means 5 minutes of data gets lost

• MTTD (Mean Time to Detect)

  • This is the mean time needed to notice a failure

  • How long does it usually take for the system or team to detect that something is wrong?

  • Smaller MTTD means faster detection; larger MTTD means slower detection

  • For example, an MTTD of 30 seconds means issues get found half a minute after they occur on average

• MTTR (Mean Time to Repair)

  • This is the mean time needed to fix a failure. How long does the system usually take to recover?

  • Larger MTTR means more time to recover, smaller MTTR means less

  • For example, an MTTR of 5 minutes means the failure will take 5 minutes to recover on average

• MTBF (Mean Time Between Failures)

  • This is the mean time between two failures. How often does the system usually fail?

  • Larger MTBF means failures happen less often & vice-versa

  • For example, an MTBF of 1h means failures usually happen every hour

• MTTF (Mean Time to Failure)

  • This metric is designed for non-recoverable components. How long is the lifespan of this component?

  • This metric differs from MTBF because it lacks a recovery component. The component is alive, and then it crashes without recovery. MTTF is the time between those two points.

  • A larger MTTF means a component has a longer lifespan, and a smaller MTTF means a shorter one

  • For example, an MTTF of 3 years means a component usually lasts for 3 years before becoming unusable

Availability Formula

Availability links uptime & downtime in one line:

Availability = MTBF / (MTBF + MTTR)

If the system runs for 1000 hours before a 1-hour fix, uptime is 99.9%.

Every extra nine costs more to achieve. Past “three nines,” you buy less outage and pay more in redundancy, automation, and testing.

Takeaway:

HA is measurable. Metrics turn abstract goals into clear engineering targets.

What gets measured gets managed.

Ready for the best part?

Reminder: this is a teaser of the subscriber-only post, exclusive to my golden members.

When you upgrade, you’ll get:

• Full access to System Design Case Studies

• FREE access to (coming) Interview Academy

• FREE access to (coming) Design, Build, Scale newsletter series

And more!

Get 10x the results you currently get with 1/10th the time, energy & effort.

Unlock Your Next Level

Read more

• Share this post & I'll send you some rewards for the referrals.

• Block diagrams created using Eraser.

Most engineers think designing an AI chat system is just “calling OpenAI API and saving messages to a database”.

That approach doesn’t work when you need to handle ChatGPT’s scale:

• 5.8 billion visits per month

• 1 billion weekly active users sending 2.5+ billion prompts per day

Your database can’t store 500TB per year. API costs will hit $145 million per month. And you’ll need 11.4 million concurrent connections that no single server can handle.

This problem shows up in senior and staff-level interviews at companies building AI products because it tests the exact skills that separate mid-level engineers from seniors:

• handling stateful connections at scale,

• managing expensive external dependencies,

• keeping costs under control while maintaining good UX,

• and designing for failures when parts of your system go down.

We’re going to build this from scratch at ChatGPT’s real scale, starting with clarifying requirements, then moving through frontend and backend design, database sharding1, the complete user flow, GPU infrastructure, and finally explaining how to scale even further to 1 billion users.

Before we design anything, we need to understand exactly what we’re building and at what scale…

Unblocked: The context layer your AI tools are missing (Partner)

Give your agents the understanding they need to generate reliable code, reviews, and answers. Unblocked builds context from your team’s code, PR history, conversations, documentation, planning tools, and runtime signals. It surfaces the insights that matter so AI outputs reflect how your system actually works.

Try Unblocked Now

(Thank you, Unblocked, for partnering on this post.)

I want to reintroduce Hayk Simonyan as a guest author.

He’s a senior software engineer specializing in helping developers break through their career plateaus and secure senior roles.

If you want to master the essential system design skills and land senior developer roles, I highly recommend checking out Hayk’s YouTube channel.

His approach focuses on what top employers actually care about: system design expertise, advanced project experience, and elite-level interview performance.

Clarifying Requirements

In interviews, candidates who skip this step fail immediately.

Questions to ask:

• Are we building the AI model itself or integrating with existing ones?

• Do we need streaming responses (word by word) or complete responses?

• What’s our expected scale? Thousands or millions of users?

• Do users need accounts and conversation history?

• Are conversations private or shareable?

• Do we support multiple languages?

• How do we handle rate limiting2? Free vs paid tiers?

• Text only, or images and files too?

• Do we need content moderation?

For this design, let’s assume:

• Integrating with existing LLM (not building the model ourselves)

• Streaming responses required for good UX

• As of January 2026, ChatGPT has 800-900 million weekly active users, with an average of 2.5+ billion prompts per day

• Users need accounts, private conversations with history

• Text only, English first

• Rate limiting on both requests and tokens

• Basic content moderation required

Now that we’ve defined our requirements, let’s start with what users actually see and interact with.

Frontend Interface Design

Let’s visualize what we’re building. This clarifies requirements and influences backend decisions.

Main Interface:

Critical frontend decision: Streaming

You have two options:

Option 1: Wait for complete response

• Simple HTTP POST, and get a full response back

• User waits 10-30 seconds staring at the loading spinner

• Poor UX, feels slow

Option 2: Stream word by word

• Requires a persistent connection

• User sees progress immediately

• Much better UX

ChatGPT, Claude, and Gemini all use streaming.

Let me verify the exact implementation.

How streaming actually works:

All major AI chat products use Server-Sent Events3 (SSE), not WebSockets.

Let me explain why:

SSE (what ChatGPT uses):

• Works over standard HTTP

• Browser has a built-in EventSource API

• Auto-reconnection handled automatically

• One-way communication (server to client)

• Simpler to implement and debug

• Lower overhead than WebSockets

WebSockets (what ChatGPT doesn’t use):

• Requires protocol upgrade from HTTP

• Bidirectional communication

• More complex to implement

• Useful for collaborative features, typing indicators

• Overkill for just streaming AI responses

Example of SSE streaming:

User types: “Explain React hooks”.

Each word appears in the UI as it arrives, creating the typing effect.

Frontend tech stack:

• React for UI components

• EventSource API for SSE connection

• State management (Zustand or Redux) for conversations

• Optimistic updates (show user message immediately)

• Error handling for network failures, rate limits

The frontend sets user expectations for real-time streaming.

Now, let’s formalize what our system must do.

Functional Requirements

• User registration and authentication

• Create and manage conversations

• Send messages and receive streaming AI responses via SSE

• Save all conversations with full history

• Retrieve past conversations

• Stream responses token4 by token

• Rate limiting (requests + tokens)

• Content moderation for harmful requests

We’ve defined what the system does. But how well must it perform?

These requirements will drive our architectural decisions.

Non-Functional Requirements

How well it must perform (we’ll connect these to design decisions later):

• Availability: 99.9%

  • Max 43 minutes of downtime per month

• Latency: First token < 2 seconds

  • Users wait no more than 2 seconds

• Scalability: 200M DAU, 20M concurrent conversations

  • Assuming 10% concurrency

• Consistency: Zero message loss

  • Every message must be saved

• Cost efficiency

  • LLM APIs are expensive and need optimization

• Security: End-to-end encryption

  • Private conversations

We’ll show how our design achieves each of these.

We’ve set our performance targets. Now let’s run the numbers to see what infrastructure we actually need.

Capacity and Storage Estimation

Math matters because it drives infrastructure decisions.

Let’s use ChatGPT’s actual scale of growth projections:

Traffic:

• 1 billion weekly active users (WAU)

  • The current sources show 800-900 million weekly active users.

  • To accommodate growth and simplify our calculations, let’s design for 1 billion weekly active users.

• ~200 million daily active users (DAU)

• 2.5 billion prompts/messages per day

  • ChatGPT’s actual processing volume

• 12.5 messages per user per day average

• Peak hours (20% of daily traffic in 2 hours)

  • 69,444 messages/second

• Concurrent conversations at peak: ~20 million

  • Assuming 10% of DAU are chatting simultaneously

Storage per conversation:

• User message: 100 characters = 100 bytes

• AI response: 500 characters = 500 bytes

• Per exchange: 600 bytes

• 12.5 exchanges/day/user: 7.5 KB/user/day

• 200M DAU × 7.5KB = 1.5TB/day

• Annual: 548 TB/year

Metadata storage:

• User profiles (1KB each): 1B × 1KB = 1TB

• Conversation metadata (500 bytes each, 10 per user average): 10B conversations × 500B = 5TB

Total Year 1: ~554TB (need distributed database, single PostgreSQL won’t cut it)

Connection overhead:

• 20M concurrent SSE connections at peak

• Each connection: ~10-30 seconds average

• Need infrastructure that can handle millions of long-lived connections

Bandwidth:

• Average response: 750 bytes streamed over 20 seconds

• 20M concurrent: 20M × 750B / 20s = 750 MB/s = 6 Gbps

• Peak outbound bandwidth requirement

Key insight: At ChatGPT’s scale, single-server solutions don’t work. We need distributed systems, sharding, and multi-region deployment from day one.

Now that we know the scale we’re dealing with, let’s design the actual system that can handle it.

High-Level System Design

Before diving into costs and implementation details, let’s map out the complete system architecture.

OpenAI uses a multi-cloud setup to secure enough compute power and avoid relying on a single vendor:

• Microsoft Azure: Still their leading provider. They have a massive, long-term partnership in which Microsoft builds specialized supercomputers for training OpenAI’s models.

• AWS: OpenAI signed a $38 billion deal with Amazon in late 2025. They now run significant workloads on AWS’s GPU clusters and specialized AI hardware.

• Google Cloud (GCP): As of mid-2025, OpenAI uses GCP to power parts of ChatGPT Enterprise and the Team tiers, as well as their API.

• Oracle: They have a multi-billion dollar agreement to use Oracle’s data center capacity, specifically for the massive “Stargate” infrastructure project.

• CoreWeave: They also use this specialized “neocloud” provider for dedicated access to high-performance NVIDIA GPUs.

Components Overview

Our system has three main layers:

1. Client Layer

   • React web application

   • Mobile apps (iOS/Android)

   • SSE connections for real-time streaming