The System Design Newsletter

Reserve your seat today for just $99

Ready for the best part?

Skills and commands: reusable workflows

mobile system design was HARD until I learned these 53 concepts

Shefali Jangid — Mon, 22 Jun 2026 11:04:05 GMT

Share this post & I’ll send you some rewards for the referrals.

The following is the third of a premium 3-part1 newsletter series… where everything comes together for mobile system design: architecture, how the platform behaves, and what happens after you ship.

On with part 3 of the newsletter:

===

In this newsletter, we’ll learn 14 concepts that shape how production apps are built and operated:

Modular Architecture/Feature-Based Modules,
Backend for Frontend (BFF) for Mobile,
App Versioning & Backward Compatibility,
Feature Flags & Remote Configuration,
Deep Linking for Navigation & Growth,
Navigation & Screen Flow Architecture,
Permissions Model,
Accessibility (a11y),
Localisation & Internationalisation (i18n),
Graceful Degradation,
Geo Location Usage,
Crash Reporting & Performance Monitoring,
Crash-Free Rate as an SLI,
Staged Rollouts & Safe Releases.

For each concept, you’ll learn:

What it is and how it works
Real-world example
Tradeoffs
Why it matters for mobile

Onward.

Do you want to become an AI Engineer in 2026?

AI Engineer is LinkedIn’s #1 fastest-growing job, with many roles paying $180K–$350K.

The challenge isn’t learning AI tools.

It’s learning how to design, build, and deploy AI systems that work in production.

That’s why Saras AI Institute created the Advanced AI Engineer Certificate.

Co-designed with AI leaders from Microsoft (Eduardo Kassner, Chief Data & AI Officer) and Google (Tomas Pfister, Head of AI Research @ Google Cloud).

In 5 months, you’ll build a portfolio of 4 real-world AI projects, including agentic workflows, RAG systems, LLM applications, and production-ready AI solutions.

Turn your AI product idea into a production-ready solution with hands-on mentorship from industry experts.
Weekly live masterclasses with AI practitioners
100% online, designed for working engineers

Engineers from Microsoft, Apple, AWS, and other leading technology companies have already joined.

By graduation, you’ll have a GitHub portfolio that demonstrates your ability to design, build, and lead AI initiatives—not just talk about them.

Cohort starts August 1.

(Thanks to Saras AI Institute for partnering on this post.)

I want to reintroduce Shefali Jangid as a guest author.

She’s a web developer, technical writer, and content creator with a love for frontend architecture and building things that scale.

Check out her work and socials:

You’ll often find her writing about web development, sharing UI tips, and building tools that make developers’ lives easier.

Architecture

This section explains different architectural patterns and design approaches that help build apps that are modular, easier to maintain, and can grow without becoming difficult to update.

Let’s dive in!

40. Modular Architecture/Feature-Based Modules

As mobile apps grow, the codebase can become very large and difficult to manage.

If everything is written in one BIG project, it becomes harder to understand the code, make changes, and test new features.

To solve this problem, you can use modular architecture.

In modular architecture, app is divided into separate modules, where each module represents a specific feature of the app.

An app could have modules like:

Auth module: Handles login and signup.
Feed module: Shows posts or content.
Checkout module: Handles payments and orders.

Each module contains its own code and exposes only the parts other modules need.

This keeps features separate and prevents different parts of the app from interfering with each other.

This structure makes the app easier to understand and maintain.

Why It Matters

In a very large app, building the entire project can take a long time…

If a single codebase contains hundreds of thousands of lines of code, compiling the app could take several minutes.

With modular architecture, only the module that changed needs to be rebuilt. This makes builds much faster and helps you work more efficiently.

Real-World Example

Airbnb, Uber, and Spotify use modular architecture in their mobile apps.

They adopted this approach to accelerate development, improve code organization, and allow different teams to work on separate features simultaneously.

Trade-offs

Modular architecture also introduces some challenges.

You need to carefully design the boundaries between modules and define clear interfaces for how modules communicate with each other. Some shared functionality, such as analytics, navigation, or design systems, may need to live in a common core module that all features can use.

It can also be difficult to convert an existing large app into a modular structure after it has already grown.

Practical Tip

It’s useful to visualize how modules depend on each other.

Tools like dependency graph visualizers can help teams see how modules are connected and identify problems such as circular dependencies, where two modules depend on each other. This usually means the architecture needs improvement.

While modular architecture improves how the app code is organized, mobile apps also depend heavily on backend systems designed specifically for them.

41. Backend for Frontend (BFF) for Mobile

Mobile apps often need data from “many” different backend services.

A single screen requires data from a user service, a recommendation service, and a content service.

If the mobile app directly calls all these services, it would need to make several network requests and combine the results on the device. This can SLOW down the app and increase battery usage.

To solve this, use a technique called Backend for Frontend (BFF).

A BFF2 is a special backend layer designed specifically for a specific type of client, such as a mobile app.

Instead of the mobile app calling many services, it sends one request to the BFF. The BFF then communicates with the required services, gathers the data, combines it, and sends back a single optimized response.

This makes the mobile app simpler and reduces the number of network requests.

Why It Matters

Every network request adds delay & consumes battery.

If a mobile screen requires data from five services, the app will make five separate API calls. This increases latency and slows down the user experience.

With a BFF, the mobile app only makes one request, and the server handles the rest. This improves performance and keeps the app more efficient.

Real-World Example

Netflix uses the BFF approach to serve different types of clients.

They maintain separate backend layers for iOS apps, Android apps, TVs, and web browsers.

Each BFF returns data formatted specifically for that device.

A mobile BFF sends smaller responses optimized for mobile screens, while the TV version may include higher-resolution metadata.

Trade-offs

Using a BFF also introduces an additional service that needs to be maintained.

It is important BFF does not become overly complex. Its main responsibility should be to aggregate and transform data for the client. Core business logic should remain in the backend services that actually own the data.

As backend APIs evolve, another challenge appears: users often run older versions of the app...

42. App Versioning & Backward Compatibility

When mobile apps are released, users do NOT always update them immediately.

Some people update right away, but others may continue using an older version of the app for weeks or even months.

So the backend system must support many versions of the app simultaneously.

i.e., server should continue to work with older app versions while also supporting the latest version.

To manage this, use versioning:

Versioning assigns each API, or app release, a version number so the server knows how to communicate with different app versions.

Check these API endpoints:

/v1/users
/v2/users

Each version can return data in a format that matches what that specific app version expects.

Why It Matters

If the backend changes something in the API without considering older app versions, older apps will suddenly stop working…

If the server removes a field that an older app depends on, the app could crash or show broken screens.

Unlike websites, mobile apps cannot be updated instantly for every user. This is why systems must remain backward compatible.

Real-World Example

Stripe uses API versioning to maintain compatibility.

When a client makes an API request, it specifies which API version it expects. Even if Stripe changes the API later, older clients still receive responses in the format they were built for.

This allows the platform to evolve without breaking existing apps.

Trade-offs

Supporting many versions of an API requires extra maintenance.

You need to keep older fields or response formats in the system for some time.

A common strategy used to manage this safely is called the Expand–Contract pattern:

First, add new fields or features without removing the old ones.
Gradually update apps to use the new fields.
Once all users have updated, remove the old fields.

This approach prevents breaking older versions while still allowing the system to evolve.

Even when APIs remain backward compatible, you still need safe ways to release new features…

43. Feature Flags & Remote Configuration

When you add a new feature to a mobile app, you must release a new version of the app and wait for users to update it.

This process can take time because apps must go through the app store review, and not all users update their apps immediately.

Feature flags solve this problem:

A feature flag is a switch that lets you remotely enable or disable a feature without releasing a new version of the app.

The feature’s code is already included in the app, but it stays disabled by default. You can later enable it from a dashboard for specific users or for everyone.

This allows you to control when features are released.

Remote configuration extends this idea even further. It allows you to change certain app behaviors remotely, such as:

UI text or labels
Feature limits
Layouts or design elements
Experiment settings

Tools such as Firebase Remote Config are commonly used for this.

Why It Matters

Submitting an app update to the app store can take up to 3 days for approval.

If a new feature has a bug and has already been released to millions of users, fixing it through another app update can take time.

With feature flags, you can disable the feature instantly from a dashboard without waiting for another release.

This makes apps safer and easier to manage in production.

Real-World Example

Many companies use feature flags to run A/B testing3 experiments.

Airbnb tests two different versions of a feature:

One group of users sees the original version.
Another group sees a new version.

Feature flags control which users see which version, allowing the company to measure which design performs better.

Trade-offs

While feature flags are powerful, they can also create complexity.

If too many flags are added and never removed, the codebase can become cluttered with outdated flags. To avoid this, you should define a lifecycle policy where every feature flag has a purpose and an expiry date. Once the feature becomes permanent, the flag should be removed from the code.

Feature flags help control when features are released, but navigation systems determine how users actually reach those features inside the app.

Reminder: this is a teaser of the subscriber-only newsletter, exclusive to my golden members.

When you upgrade, you’ll get:

High-level architecture of real-world systems.
Deep dive into how popular real-world systems work.
How real-world systems handle scale, reliability, and performance.

And much more!

AI writes more code than ever. Reviewing it shouldn’t mean scrolling forty files in alphabetical order.

Ready for the next technique?

44. Deep Linking for Navigation & Growth

The Secret Architecture Behind AI Data Centers

Neo Kim — Fri, 19 Jun 2026 09:11:26 GMT

Share this post & I’ll send you some rewards for the referrals.

When people talk about Artificial Intelligence (AI), they often refer to models such as Generative Pre-trained Transformers (GPTs), Large Language Models (LLMs), and neural networks.

It sounds like magic, software that thinks…

But here’s the uncomfortable truth:

The biggest breakthroughs in AI are not coming from better models; they’re coming from better infrastructure.

If AI were only about the models, there would be no need for huge, warehouse-sized data centers packed with specialized hardware and powered by massive amounts of electricity.

So what’s actually going on inside those buildings?

To make sense of everything that follows, we need a simple mental model.

Let’s find out one layer at a time…

Onward.

Review the actual change, not the file list (Partner)

Cohorts group related files and chunks so you review one idea at a time. Layers order them so foundational changes - data shapes, contracts - come before the code that depends on them. Code Peek lets you click any variable, function, class, or type to see its definition and usages without leaving the tab, while Semantic Diff view cuts past formatting noise to show what actually changed.

In the early access, available for free to everyone.

From the team that pioneered AI code reviews. 2M reviews every week. 6M repos. 15K customers.The review interface built for the way modern PRs are actually written.

>>> Review your next PR with CodeRabbit Review Today

(Thanks to CodeRabbit for partnering on this post.)

I’d like to introduce Ashish Prajapati.

He is an AI infrastructure specialist and technology educator who helps engineers understand the technologies powering modern AI.

With expertise spanning data centers, cloud computing, GPU infrastructure, Generative AI, and Agentic AI, he focuses on turning complex concepts into practical, real-world knowledge.

His training programs have reached over 39,000 learners worldwide, and through his free mentoring initiative, Become a Solutions Architect (BeSA), he has helped over 23,000 professionals accelerate their careers.

As AI adoption continues to grow, understanding the infrastructure behind AI systems is becoming a valuable skill for architects, engineers, and IT professionals.

For a limited time, this newsletter's readers can use code SYSTEM-DESIGN to get 25% off Ashish’s live NCA-AIIO bootcamp.

This intensive one-day workshop covers the fundamentals of NVIDIA AI infrastructure, GPU networking, accelerated computing, and AI operations, and prepares you for the NVIDIA-Certified Associate: AI Infrastructure and Operations (NCA-AIIO) certification.

Learn, Prepare, and Pass NCA-AIIO in a Day.

Here’s what you’ll find inside this newsletter:

The hidden architecture behind AI. Why models are only the visible tip of a much larger infrastructure stack.
The physics that limit AI growth. How power, cooling, and space shape every AI data center.
Why GPUs became the backbone of AI. The architectural advantages that made modern AI training possible.
The real bottlenecks in modern AI. Why memory, networking, and coordination determine performance at scale.
How thousands of GPUs work together as one system. The software, networking, and orchestration layers that enable large-scale AI.

Think of an AI Data Center as a Factory

Before we go deep, build this mental model; it will explain almost everything that follows:

AI Data Center = Factory
Data = Raw material
Graphics Processing Units (GPUs) = Machines on the factory floor
Network = Internal transport system
Storage = Warehouse holding raw materials
Software = Assembly line logic
Output = Trained intelligence

Now imagine a car factory.

It doesn’t just need skilled workers; instead, it needs conveyors, warehouses, and power. Remove any of those, and production stops.

An AI data center works the same way…

AI infrastructure processes vast amounts of data using mathematical operations, similar to how a factory transforms raw materials into finished goods.

Remove any one ingredient, and the entire system stalls…

Once you see AI as a factory, the next question becomes obvious: what limits how fast this factory can run?

First Problem: Physics

Before AI is a software problem, it’s a physics problem…

Here are three hard physical constraints that shape every design decision in an AI data center:

1. Power

Modern AI workloads rely on high-density GPUs that consume enormous amounts of electricity.

A single server rack of GPUs can draw 80-120 kilowatts. i.e., roughly equivalent to 40 homes running at full load simultaneously. Yet you can’t just add more GPUs forever.

You’re limited by how much electricity your facility can physically receive and distribute.

2. Cooling

More power means more heat, and heat is the enemy…

Here’s what happens if you don’t remove it fast enough:

Hardware fails prematurely,
Performance throttles back to avoid damage,
In extreme cases, systems shut down entirely.

To manage the intense heat generated by AI workloads, modern data centers are increasingly adopting liquid cooling, which is far more effective than conventional air cooling.

Coolant is circulated through cold plates attached directly to GPUs and CPUs, absorbing heat much more efficiently than air. The warmed liquid is then pumped to heat exchangers, where the heat is removed before the coolant is recirculated.

This approach enables higher rack densities, lower cooling power consumption, and more consistent performance for large-scale AI training and inference1 workloads.

3. Space

Even if you had unlimited chips and unlimited power, you still need racks to mount them and floor space to put those racks.

Scaling AI is not just software--it’s real estate, energy infrastructure, and thermodynamics combined.

Takeaway

A data center is, at its core, a power plant combined with a massive, precisely controlled air-conditioning system. Every engineering decision flows from managing this reality.

After dealing with the physical challenges, the next question is: how we perform the computation?

Compute: Why GPUs Took Over

To understand why GPUs dominate AI, picture a very simple task…

Imagine you have 1,000 fence posts to paint:

Option A: One expert painter, working post by post. Slow, but the painter can handle any unexpected situation--a broken post, a curved section.
Option B: 1,000 workers, each painting exactly one post simultaneously. Extremely fast, but each worker knows only one simple task.

That’s the difference between a CPU and a GPU.

CPU (Central Processing Unit)

A few very powerful cores
Handles complex, varied, sequential tasks
Acts as the “manager” making decisions and coordinating work

GPU (Graphics Processing Unit)

Thousands of smaller, simpler cores
Handles same simple operation repeated millions of times, all at once
Acts as the “factory workers” doing mass production

AI training repeatedly performs matrix multiplications billions of times, making minor adjustments after each step. GPUs are specifically designed for this type of work.

DPU (Data Processing Unit)

As AI systems grow, a third processor becomes important.
DPU handles networking, storage I/O, and security operations--tasks that would otherwise steal cycles from the GPU and CPU.

Here’s the system view:

CPU = manager (decision-making)
GPU = factory workers (mass production)
DPU = logistics and operations (keeping materials moving and systems secure)

But even with powerful GPUs, computation alone isn’t the limiting factor; the real challenge is keeping them fed with data…

Hidden Bottleneck: Memory and Storage

Here’s something that surprises most people:

GPUs don’t fail because they’re slow. They fail because they’re hungry—constantly starved for data.

Memory Hierarchy

Data doesn’t live in one place. It moves through a layered system, trading speed for capacity:

GPU Memory (HBM2): Blazingly fast but tiny; usually tens of gigabytes. This is the GPU’s working memory.
CPU RAM: Larger (hundreds of gigabytes), but slower than HBM.
SSD/NVMe3 Storage: Enormous capacity at a lower cost, but still slower.
Network/Object Storage: Nearly unlimited, but the slowest link in the chain.

Every time the GPU needs data that isn’t already in its fast local memory, it must wait... And a GPU sitting idle costs just as much money as one doing useful work.

Takeaway

AI performance is not determined solely by how fast the GPU computes. Instead, by how fast data can be fed into the GPU.

A faster chip with a slow data pipeline is like a faster car stuck in traffic.

Tiered storage in practice

In real AI data centers, storage is organized into tiers that roughly map to hot, warm, and cold data:

Hot: NVMe SSDs holding active training datasets and model checkpoints.
Warm: Parallel file systems4, such as Lustre, General Parallel File System (GPFS), provide high-speed shared access across many GPUs.
Cold: Object storage, such as Amazon Simple Storage Service (Amazon S3), for storing raw datasets and older checkpoints.

As models grow larger, a single GPU is no longer enough…

Techniques such as model sharding5 and distributed training6 distribute the model across multiple GPUs and memory layers.

And once data starts moving across machines, a new bottleneck appears: how those machines communicate with each other.

Reminder: this is a teaser of the subscriber-only newsletter series, exclusive to my golden members.

When you upgrade, you’ll get:

High-level architecture of real-world systems.
Deep dive into how popular real-world systems actually work.
How real-world systems handle scale, reliability, and performance.

Networking: Bottleneck Nobody Talks About

When training a large AI model, you typically use thousands of GPUs working together rather than a single GPU…

And those thousands of GPUs must constantly communicate with each other, sharing what they’ve learned at every step…

Every second a GPU waits for data or synchronization is a second of lost productivity and wasted investment.

Cluster topology and fabrics

Modern AI clusters typically use leaf-spine network architectures7: a design built for high throughput and low latency at scale.

Every GPU server can reach every other GPU server in at most ‘two’ hops.

Within a node, GPUs talk over NVLink (intra-node fabric). Between nodes, traffic flows over InfiniBand or RDMA over Converged Ethernet (RoCE), the inter-node fabric8.

There’s also a clear speed hierarchy:

NVLink9: fastest path, a direct, private highway connecting GPUs within the same server.
InfiniBand/RoCE10: very high-speed interconnect between separate servers. Imagine it as the “bullet train” of the data center.
Ethernet: general-purpose, flexible, and ubiquitous, but not designed for AI’s most extreme bandwidth and latency demands.

Amdahl’s Law: Why More GPUs Aren’t Always Better

There’s a fundamental principle in distributed computing called Amdahl’s Law.

In plain English: if parts of your system must wait for other parts, adding more workers eventually stops helping.

In AI workloads, adding more GPUs increases computing power but also increases communication overhead. As GPUs spend more time sharing and synchronizing data, the bottleneck gradually shifts from computation to communication.

Smarter data movement

Modern AI systems address this with several techniques:

Remote Direct Memory Access (RDMA): Provides low-latency, high-throughput communication by enabling direct memory-to-memory data transfers between servers.
RDMA kernel11 bypass: Lets the network adapter access application memory directly from user space12, eliminating unnecessary operating system overhead.
GPUDirect: GPUs can read from and write to storage or network cards directly, bypassing the CPU entirely.
Zero-copy pipelines: Data takes the shortest path from source to GPU, with no redundant copying along the way.

The goal of AI networking is to make thousands of GPUs behave like one giant, unified processor, as if the data never had to travel at all.

Once communication is in place, the next challenge is dividing the work efficiently across all those GPUs…

30 Core Agentic Engineering Concepts, Explained Simply

Neo Kim — Wed, 17 Jun 2026 09:35:54 GMT

Share this post & I'll send you some rewards for the referrals.

How do you keep up with agentic engineering when a new tool ships every week?

You do not…

You learn the ideas behind the tools, and let the tools come and go.

The pace will not slow down. New models, new frameworks, a new “this changes everything” launch every few days. Chase all of it, and you spend more time switching tools than using them.

But underneath the noise, it is the same few ideas every time.

One calls it a skill, the next calls it a rule; it's the same job underneath.

Learn the idea, and it stops mattering which tool you pick.

By the end of this newsletter, you can read any agent post or news drop and see which of these ideas it actually covers, instead of feeling behind every time something new ships.

Onward.

”How to adopt externalized authorization: step-by-step roadmap” eBook by Cerbos (Sponsor)

Hardcoded authorization logic doesn’t scale. As systems grow and requirements change, those tightly coupled permissions end up slowing down development, creating security risks, and making compliance a nightmare.

This 80-page eBook breaks down how hundreds of engineering teams have tackled this problem. It walks through the practical steps for moving from hardcoded permissions to externalized authorization.

What’s in it:

Step-by-step adoption strategy from planning to PoC rollout
Frameworks, policy examples, code samples
Lessons from teams who’ve made the transition

If you’re dealing with authorization complexity, this might save you some of the trial and error.

Download free eBook

(Thanks to Cerbos for partnering on this post.)

I want to introduce Paul Hoekstra as the guest author.

Paul is a data engineer at Digital Power, a data and AI consultancy, and writes Paul’s Pipeline, a free newsletter on AI, data, and side projects.

This newsletter pulls out the ideas that keep mattering, whichever tool you pick. If it leaves you wanting more, he has a four-part series on the layers covered in this newsletter: configuration, capability, orchestration, and guardrails.

Here’s what you’ll find inside this newsletter:

Foundations. What an agent is, how it runs, where its state lives, and the patterns you build once you have more than one.
Configuration. How you shape the agent’s behavior before it writes a line of code: config files, reusable workflows, prompt caching, and the context rot that limits all of it.
Capability. What the agent can reach for once it is running: live documentation, AI-native search, long-term memory, and knowledge it queries but never owns.
Orchestration. How you coordinate multiple agents without letting them step on each other.
Guardrails. What stops the agents from doing damage: sandboxing, permissions, hooks, and the commit gates that catch the rest.
Observability. How you see what an agent actually did after it stopped typing: tracing, logging, replay, and metrics.

Try Apodex for free (early access)

Foundations

Let’s dive in!

1. What is an agent?

“Agent” gets used for almost anything these days.

Let us pin it down:

An agent is a large language model (LLM) that runs in a loop, has access to tools1, and decides what to do next. Instead of a single answer to a prompt, the model produces a chain of actions. And each action builds on the result of the last.

Coding is the most obvious use case so far.

The loop fits naturally there. Small building blocks. Fast feedback from tests and errors. Files the agent can edit directly. The same shape works for research agents, ops automation, customer support, and content workflows.

Use an agent when:

Task is open-ended (”debug this failing test”, not “format this date”)
The next step depends on the result of the last one
You cannot script the path in advance

But skip the agent when a single prompt or a plain script will do.

Loops are not free.

Every time round the loop costs time and money, and they add up fast. The agent gets less predictable. And it gets harder to debug: the runs are longer, and the agent does not always make the same choices twice.

You do not have to pick one mode for everything.

Use a single prompt or a plain script for easy problems. Reserve the agent for tasks where its flexibility earns the cost.

2. Execution model

An agent loop has a recognizable shape.

Three steps, repeated until something says we are done:

First, the model thinks. It reads the current state of the conversation and decides what to do next.
The model then acts. It emits a tool call. The harness2 around the model receives the call, validates it, and runs it: file read, shell command, MCP request, whatever fits.
Finally, the model observes. The tool result joins the conversation as new context. The next iteration starts with that result in scope.

This pattern goes by a few names.

ReAct (from the original paper), Think-act-observe, agent loop.

The name changes, but the loop does not.

A single model call has to predict the entire path up front and commit to it. The loop replaces that prediction with feedback. The model takes one step, observes what actually happened, and decides the next step based on the actual result rather than a guess.

This is why an agent can recover from its own mistakes.

A failing test returns a stack trace. A blocked command returns an error. Each one is just the next observation, and the next iteration plans against it. The loop catches a wrong first step on the next pass, instead of letting it poison everything after it.

Two variations sit on top of this basic loop:

Parallelism. Modern agents can fire several tool calls inside a single act step. That cuts wall-clock time on read-heavy work. It also causes conflicts when two parallel calls try to edit the same thing. We will come back to ways of handling that when we get to subagents (concept 17).

Blocking vs non-blocking. Most agents run synchronously: conversation waits while a tool runs, then the result comes back as the observation. Some run asynchronously and fire off long-running jobs in the background. Async unlocks fan-out, at the cost of more complexity.

3. Agent state

“State” gets used in two senses.

One is workflow progress: which step the agent is on and what is left. Agent loops carry that sense across iterations (concept 17). This concept is the other sense: everything the agent knows at any moment.

It breaks into two halves with very different properties:

Context window: everything the model can see right now. Your latest message, the system prompt, the agent’s previous tool calls, and the results of those calls. It has a hard ceiling (the token cap) and a soft one (concept 9, context rot). When a session ends, the context window is gone.

Everything outside it: what the model cannot see until it fetches it. Files on disk, database entries. Memory that survives across sessions (concept 14). Knowledge the agent only queries and never owns (concept 15). The model cannot reason over any of it directly, but the agent can pull pieces into context on demand.

So, where should your state actually live?

Files are usually the right default. They are diffable, versionable, and easy for both you and the agent to edit. Memory works well for facts that survive across sessions but do not need git history. Databases earn their place when the state needs to be queried by structure or shared across processes.

Multiple agents make the state harder.

Two agents reading the same file is fine. Two agents writing to it is the canonical race condition. Git worktrees (concept 16) give each agent its own working copy.

Subagents are friendlier: each starts with a fresh context window, and the parent passes in only what it needs. If the parent has to pass more than a few paragraphs, the split between the two is probably wrong.

4. Common agent patterns

Once you have multiple agents to work with, the question is how to compose them.

There are a handful of useful patterns to know:

The simplest is planner/executor.

One agent plans, another executes. The split keeps each agent in a context shape that suits it: open-ended thinking on one side, tight execution on the other. Useful for long tasks where you want planning quality separate from how the implementation goes.

Router/specialist sits one level above.

A router agent takes the incoming request and dispatches it to one of several specialists. Each specialist has a tighter system prompt and a smaller toolset: a security reviewer, a debug specialist, a docs writer, a test author. Cheaper, because most calls hit a smaller model. Easier to reason about, because each specialist’s behavior is bounded.

Map-reduce parallelism is the workhorse for read-heavy fan-out.

Send a single task out across many subagents and merge their results. “Review this pull request” splits into one subagent per file, and the results merge into a single summary. Cost is bounded by the longest individual subagent. Quality is bounded by how well the aggregator merges.

These patterns are not exclusive.

Real workflows combine them: a planner spawns a router, the router fans out work map-reduce style, and the merged result comes back for review.

Every pattern relies on handoffs between agents.

Each handoff carries a compressed summary, so the next agent does not start from scratch. That is what makes the patterns useful: they define where one agent’s job ends and the next one’s begins. Those seams are where work tends to break.

That’s the foundation done.

Next come the four practical layers, starting with how you shape the agent’s behavior before it touches any code.

Get 1+ referral & I’ll send you my Leetcode Master Template!

Configuration Layer

Let’s dive in!

5. Agent config files

Every agent starts with a system prompt.

The harness builder bakes in a block of instructions that covers tool use and the harness’s own conventions. The system prompt does not know anything about your project.

Agent config files are the layer above.

The agent loads a project-level Markdown file at the start of every session, and the file remains in the context window for each subsequent step. The file tells the agent your standards, limits, conventions, and behavioral rules.

Claude Code calls this file CLAUDE.md.

Most other tools have settled on AGENTS.md. A bit like Apple and USB-C: the industry wants a standard, but not everyone follows it.

It matters because agents default to whatever looked most plausible in their training data. Without a config file, you get pip install when your project uses uv, black when you use ruff, and defensive sludge wrapped around every change.

With one, the agent reads a set of hard rules before it writes a line of code.

A useful config is short and specific:

Hard limits on function length
The package manager and tooling stack you actually use
A few behavioral rules like “never commit secrets” or “always read a file before editing it.”

Anything generic or AI-generated dilutes the file and makes the rest harder for the model to attend to.

Keep it under 100 lines. Cut anything that is not consistently improving outcomes. Treat it the way you would treat a Makefile: as code, not documentation, reviewed when it changes.

6. Reusable workflow files

If config files are always on, reusable workflow files are on demand.

They are still just Markdown, but with YAML frontmatter that controls when the agent loads them. Examples are Claude Code skills (in .claude/skills/) and Cursor rules.

The frontmatter3 does most of the work. A name and description tell the agent when the skill is relevant. An optional globs field narrows it further by file type or path pattern. Whether the skill actually helps depends on the quality of the instructions inside it.

Researchers built SkillsBench, a benchmark of 86 tasks across 11 domains, and provided models with brief written workflows for approaching each task.

Claude Haiku scored 27.7% with human-curated skills. That beat the 22.0% Opus scored without them. The cheapest model with good instructions outperformed the flagship without.

There is a second lesson in the data.

When the researchers let the model write its own skills, the gains disappeared. Given everything we know about context rot, that should surprise no one. Generic AI-generated boilerplate makes things worse, not better.

The rough division of labor is simple: config files for always-relevant project rules, skills for reusable task-specific procedures.

The live prompt for what is unique about the current task.

7. Workflow frameworks

If you are using an agent for coding, a framework on top helps.

The agent follows a documented process for planning, test-driven development (TDD), debugging, and code review. No more falling back on whatever it remembers from training. The mechanism is a chain of skills, hooks, and slash commands4 wired together into a repeatable process.

Here are a few options worth knowing:

Superpowers ships a curated library of skills for common workflows: brainstorming, TDD, debugging, and code review. It also adds HARD-GATE directives and anti-rationalization tables. These push the agent to use the skills rather than skip them.

Get Shit Done does something similar, but through slash commands, hooks, and meta-prompting instead of skills.

Compound Engineering splits work into Plan, Work, Review, and Compound phases. The last phase captures patterns and solutions into a searchable knowledge base, so each feature makes the next one easier.

They differ in shape but share a thread: get the agent to understand what it is building, tell it how to go to work, and check the result against what you actually wanted.

8. Prompt caching

Prompt caching stores the stable prefix of the conversation so the model does not pay the full price to re-read it every turn.

Most agentic tools handle this in the background, but it is worth understanding because it changes the economics of long config files.

The mechanism is the same across providers:

The first call sends the full prefix, including the config file and any reusable workflows the agent has loaded, and pays a small premium to write that prefix into the cache. Later turns re-read it at a steep discount. Latency improves, too, because the model does not reprocess the same tokens.

The catch is the cache time to live (TTL).

Caches expire after a period of inactivity, somewhere between a few minutes and an hour, depending on the provider. An active session keeps the cache warm. A long pause (coffee, Slack, a spec to read) lets it expire and forces a rewrite on the next call.

If your tooling lets you configure the TTL, longer windows cost more to write but stay warm through bigger gaps in activity. The choice depends on how active you expect to be. Longer TTL when you will keep coming back. Shorter TTL when you might not.

Without prompt caching, you would think carefully about the size of your config file. With it, the cost is closer to a rounding error.

9. Context rot

Model performance drops as the context window fills up.

Prompt caching saves your wallet, but those tokens still sit there, and the model has to look past each one to get to what you want it to focus on.

Even the latest models show this:

Drop a few specific facts into a long document and ask a model to find them. GPT-5.5 scores 98.1% on short documents and 74.0% on documents stretching to a million tokens.
Claude Opus 4.7 drops from 59.2% at 256K to 32.2% at 1M.

A recent study confirmed the same effect for instruction quality. Stuffing the system prompt with generic AI-generated boilerplate produced worse results than having no file at all.

Transformers5 work by having every token attend to every other token in the context. Each token’s “attention” is a weighted distribution across the rest, with the weights summing to one.

Double the context, and each token’s share of that weight roughly halves. The model splits its finite attention budget across more material. The relevant signal gets diluted by everything around it.

There is a training distribution effect on top of that.

Most of what these models learned from is short. Behavior in very long contexts is partly extrapolation. Retrieval drops the further you push from the lengths the model trained on most.

The takeaway runs through every other concept in this layer.

Config files, skills, MCPs, even memory: every token competes with everything else for the model’s attention. Keep the things you need lean. Cut anything that is not pulling its weight.

That covers configuration. Now we look at what the agent can actually reach for once it starts running.

AI research is easy—verification is hard (Partner)

Most AI research agents can find information.

The harder problem is correctness — whether an answer is actually right, not just well-formed.

Many systems reason, search, and verify within the same context window. When they make mistakes, they often review their own work using the same signals that produced the error. This creates a subtle failure mode: outputs can look consistent, pass internal checks, and still be wrong.

Apodex refers to this as pseudo-correctness — answers that appear valid under the model’s own evaluation, but do not actually solve the underlying problem.

Apodex is built differently.

As a Self-Evolving Heavy-Duty Solver, it separates generation from verification. Specialist agents build solutions from evidence, while independent verifier agents audit the result before it is finalized.

Instead of asking whether an answer “looks correct,” Apodex evaluates whether it actually holds. Verification is structured across three axes:

Comprehension — did it capture the real problem?
Causality — does it address the true cause?
Empirical grounding — is it validated through execution?

This shifts verification from self-review to independent audit, reducing pseudo-correctness at the system level.

On LiveCodeBench v6, the heavy-duty configuration improves a 397B model from 82.0 to 85.1 under identical parameters, driven primarily by the verifier layer. It also achieves 79.0 on SWE-bench Verified and 58.4 on Terminal-Bench v2.

The Apodex 1.0 release includes Apodex-1.0-H, Apodex-mini, AgentOS, and open-weight models at 0.8B, 2B, and 4B.

See how verified AI research works:

For developers who want to dive deeper:

Capability Layer

Let’s keep going!

10. Model Context Protocol (MCP)

MCP is a standard for giving agents access to external tools and services.

Originally an Anthropic spec, now broadly adopted across the ecosystem. Here’s the basic idea: a tool exposes itself through a protocol the agent already understands, so you stop hand-rolling glue code per tool and per agent.

Yet MCP gets a lot of heat.

The basic criticism is that it is redundant and slurps context. Why add a new protocol when agents can already use command-line interfaces (CLIs), scripts, and direct API calls?

A fully loaded MCP is expensive.

By 2026, deferred tool loading is the norm. The agent starts with a list of tool names and descriptions. The full schema only loads when the agent decides to use a tool. With deferred loading, a stack like GitHub + Exa + Context7 + DeepWiki costs about 607 tokens before any tools are used. The fully loaded version costs around 5,500. A few plain skills calling CLIs directly cost about 300.

So MCPs are still heavier than the leanest alternative.

But they solve real problems: standardization, auth, permissions, organizational distribution. And they are getting leaner. Whether MCP itself is the protocol that wins is a separate question.

11. Live document retrieval

Models have knowledge cutoffs.

When the agent does not know an API signature, it usually does not say so. It guesses... confidently. And because the guess looks plausible, you only catch the mistake at runtime.

Context7 pulls live library docs into the agent’s context.

The agent gets current documentation and working examples. Not whatever version was in the training data. This wipes out an entire class of bugs where it confidently calls functions that have been renamed, moved, or deprecated.

DeepWiki goes one level out.

It generates AI-powered docs for GitHub repositories. Point it at an unfamiliar codebase and ask, “How does the authentication flow work?” and you get an answer grounded in the actual repo instead of generic vibes.

The pattern is the same: separate “knowing how to code” from “knowing the current state of the world.” Prompting helps with the first. Live data solves the second.

12. AI-native web search

AI-native search engines are built for agents to consume.

Existing web search returns HTML pages designed for humans. The agent has to chew through ads, navigation, and unrelated content to find what matters. AI-native search returns the bits directly.

Exa is what I reach for.

It returns structured search results, extracted content, highlights, and summaries. I use it to find docs, discussions, and real-world examples the model would not have from training data alone.

Token efficiency makes a real difference when you use it in automated workflows. The agent gets answers without burning context on a parsing tax it should not have to pay.

13. Visual output generation

Skills and MCPs can turn the agent into a specialist in output formats other than application code: design files, presentations, architecture diagrams, and videos.

Figma’s MCP server lets the agent read design data directly: layout, components, variables, and spacing. Instead of describing a user interface (UI) in words or pasting screenshots, you point the agent at a frame, and it generates code from the actual design spec. It works in both directions, so the agent can also push back to the Figma canvas.

Frontend-slides is a Claude Code skill that generates complete HTML presentations from a prompt. It produces a single, self-contained file with inline CSS and JavaScript that runs in the browser. All the diagrams in this newsletter came from one-shots with this skill.

draw.io takes the same approach for architecture diagrams. The .drawio file format is plain mxGraph XML, so anything that can produce structured XML can produce a diagram. Point the agent at a Terraform repository with a draw.io skill or MCP loaded. The agent walks the resource graph. It writes a matching .drawio file next to the code that defines the infrastructure. Wire the regeneration into continuous integration (CI). Your architecture diagrams stop drifting from your actual stack.

Remotion best practices does the same for video. Where frontend-slides produces a file you open in a browser, Remotion produces a file you render into a video.

The pattern that ties them together is simple: a skill or MCP that knows how to write code for a specific visual format. The agent is already great at writing code. This just teaches what kind.

14. Persistent memory

Every agent session starts fresh.

The decisions you walked through yesterday, the context you built up: gone. You start every morning by re-explaining your project from scratch.

Persistent memory closes that gap.

The simplest version is MEMORY.md, a flat Markdown file in your project directory. The agent reads it at the start of every session. It can write back to it during one. The file holds conventions, architectural decisions, session summaries, and working context. The bits you do not want to keep retyping.

The limitation is how it scales.

MEMORY.md sits in the system prompt, just like the config file, so the same token cost and context pressure apply. Once it grows past a certain size, it gradually turns into a wall of text the model skims past.

That is where plugins like episodic-memory come in.

It indexes your past conversation files, embeds them as vectors6, and exposes MCP tools to semantically search across them. Documentation usually captures what you decided. Session history often captures why: options you considered, dead ends you ruled out, trade-offs that led to the final call.

Start with a flat memory file.

Reach for a searchable layer when grep stops being enough.

15. Knowledge search

Plenty of useful context never lived in your AI coding session to begin with.

Meeting notes, design docs, specs, technical writeups. All of this still matters, and none of it shows up in your conversation history.

QMD (built by Shopify CEO Tobi Lutke) is the tool I reach for here.

It is an on-device search engine for your broader knowledge base, exposed to the agent through an MCP server. The agent queries it during a session, just as it queries Exa or Context7.

Persistent memory holds what the agent learns over time. Knowledge search covers materials the agent never helped produce.

A single agent with the right configuration and tools can already do a lot. But once you have more than one running at the same time, you need a way to coordinate them.

Get 2+ referrals & I’ll send you my Interview Mistakes to Avoid PDF!

Orchestration Layer

Let’s dive in!

16. Subagents

Subagents are delegated workers with a bounded scope.

Each subagent gets its own prompt, a limited toolset, and a clean context window. When it finishes, only the compressed result returns to the parent. Not the full reasoning chain or the intermediate steps, instead only the final answer.

The obvious benefit is parallelism.

The bigger benefit is compression. Test output, long docs, random side quests: all of that stays in the subagent’s context. The main thread stays clean enough to actually think.

You define a subagent as a Markdown file with YAML frontmatter:

name: security-reviewer
description: Reviews code for security vulnerabilities
tools: Read, Grep, Glob, Bash
model: sonnet

description is the routing signal that tells the parent when to reach for the subagent.
tools field controls what it can access.
model field lets you run cheap work on a cheaper model.

When many subagents run on the same repo in parallel, their writes can collide.

Git worktrees solve this.

A worktree is a separate checkout that shares the .git directory but has its own working tree and branch. Two agents work on the same codebase at the same time without touching each other’s files.

Claude Code supports this natively, setting isolation: “worktree” in a subagent definition runs it in an isolated checkout automatically.

17. Agent loops

An agent loop re-runs a single agent with fresh context each iteration, tracking progress in files and git.

The Ralph loop showed how powerful this is when it went viral early last year. Agents worked for hours without the output getting worse. The context window never filled up with stale reasoning and dead ends.

It uses the same compression principle as subagents. Keep the live context small. And push the state out to the file system. Then restart with a clean slate. The difference is subagents do this once per delegation. While Agent loops do it every iteration.

It works especially well for grindy, well-bounded work:

Migrating a large codebase one file at a time.
Processing a queue of items.
Refactoring across many call sites.

The model focuses on each step without dragging the previous nine into the prompt.

Claude Code ships this pattern as /goal.

You set a completion condition, like “all tests in test/auth pass and the lint step is clean”. Claude then keeps working across turns. After each turn, a small evaluator model checks whether the condition holds.

The loop stops when it does.

18. Orchestration tools

Once you are running many agents in parallel, you need a layer above them.

Spawning agents is easy, but coordinating them is hard. They duplicate work, lose track of progress, and come back with five incompatible versions of reality.

Conductor is built for exactly this.

It gives Claude Code and Codex a single UI for parallel sessions in isolated workspaces. A built-in diff viewer handles the merging. JetBrains Air does the same inside the JetBrains ecosystem. It uses Docker containers or git worktrees per task.

Vibe kanban goes lighter: a standalone app with its own server and a kanban UI for managing agent tasks. You break work into cards, assign them to agents, and track progress visually.

Cline Kanban works across agents (Claude Code, Codex, Cline). It adds auto-commit and dependency-aware parallelization.

Paperclip goes the furthest.

It markets itself as the orchestration layer for zero-human companies. There are org charts, task delegation, budget ceilings, and a governance layer where a human “board” approves high-impact decisions.

Probably overkill for solo developers, but a fascinating concept.

19. Managed/cloud-hosted agents

Managed agents are long-running agent sessions on vendor infrastructure, accessed via API.

Anthropic provides the harness, sandbox, tool loop, and container7. You define the agent (model, prompt, tools, optionally MCP servers and skills). Your app sends user events and receives streamed tool calls and messages back over server-sent events (SSE).

A session is a long-running process on Anthropic’s infrastructure, not yours.

It can grind through hours of tool calls while your app just streams events. Subagents are supported. The coordinator runs them in parallel as isolated threads on the same container filesystem. i.e., no need to babysit a local Claude Code window.

The catch is billing.

Managed Agents go through pay-as-you-go API tokens8, not Claude Pro or Max plans. The plans are still significantly more price-efficient for personal use. Use Claude Code with worktrees for your own repo. And reach for Managed Agents when you are building a product that runs agents for other people.

Many agents running in parallel can move fast. They can also do serious damage if nothing stops them. The next layer is what stops them…

Get 3+ referrals & I’ll send you Popular Interview Questions PDF!

Guardrails layer

Let’s keep going!

20. Sandboxing

Sandboxing isolates what the agent can read, write, or reach over the network.

The other guardrails focus on not trusting what the agent sees. Sandboxing limits what happens when trust is misplaced.

Most modern agent harnesses ship with some form of built-in sandbox. On Linux, they typically lean on bubblewrap (bwrap); on macOS, Apple’s Sandbox framework.

The shape is similar across implementations. The agent can read the project directory and system libraries. It can write to the project directory and a sandboxed temp directory. It can only reach the network through an allowed list. Credential directories like /.ssh, /.aws, ~/.gnupg, and ~/.docker are blocked from reading.

The walls are there whether the agent agrees with them or not.

If you want full isolation, running the agent inside a Docker container with --network none goes further.

No host filesystem beyond the mounted project; no credentials; no outbound connections. Overkill for some workflows, but for code review or analysis, it turns an entire class of worst-case scenarios into non-events.

For agent-generated code running at scale, Cloudflare’s Dynamic Workers offer per-execution sandboxes. They start within milliseconds, block network access, and inject credentials on the server side.

Prevents: blast radius from any other layer failing. A poisoned config file, a successful prompt injection, a permission-list bug: sandboxing is the wall that stops the consequences from leaving your project directory.

Use when: always. The built-in sandbox in your harness is usually free. Reach for Docker or Dynamic Workers when the work is high volume, untrusted, or both.

21. Permissions

Permissions are allow/deny lists for tool calls, file reads, and shell commands.

They are the baseline control for what the agent can do without having to ask each time.

Agents are not malicious. They are reward-hacky problem solvers:

Hit them with a permission error, and they reach for chmod 7779.
A dependency will not install cleanly, and they try curl | sh10.
A test keeps failing, and they start commenting out assertions.
A git push gets blocked, and they try force-pushing to main.

A typical setup uses two layers:

Project-level settings (committed to the repo) define what runs without prompting. Linting, testing, and common git operations.
User-level settings add a deny-list for things that should never happen: reading .env, rm -rf, force-push to main, curl * | sh.

Approving every tool call by hand gets tedious fast.

The middle ground is a permission classifier. A small model looks at each tool call before it runs. It decides whether to let the call through or hold it for human review. Claude Code calls this Auto mode, and similar mechanisms are showing up across most major harnesses. This is not perfect, but combined with deny-lists and sandboxing, it lets the agent move without constant hand-holding.

Prevents: creative shortcuts. The agent’s own bad judgement when it is grinding to get something to “done.”

Use when: any agent has tool access. This is the floor, not a nice-to-have.

22. Hooks

Hooks are handlers attached to specific points in the agent’s workflow.

The one that matters for guardrails is the pre-tool hook. It fires after the agent has assembled a tool call11, but before that tool actually runs. That is the last inspection point where a bad command can still be blocked. (Different harnesses name this differently. Claude Code calls it PreToolUse.)

For shell security, the least glamorous option is usually the right one: a plain pre-tool hook on Bash, backed by a local validator. Fast, deterministic, last-mile.

Tirith is a validator built for exactly this; it catches:

Hostname homoglyphs.
Path homoglyphs12.
Insecure transport.
ANSI injection.
Pipe-to-shell patterns.
Environment manipulation.

If a command contains suspicious Unicode or dangerous patterns, the hook blocks it before it runs.

Hooks are not limited to Bash. You can hook the same point for any tool the agent uses, including file edits and MCP calls. The mechanism is general. And the choice of where to put one is a design decision.

Prevents: gap between “model decided to do X” and “X actually happens.” Useful, especially for shell command sanitization.

Use when: agent has Bash access. Pair with sandboxing rather than replacing it.

23. Prompt injection defense

Agents treat their config files and tool outputs as ground truth.

This makes them an obvious place for malicious instructions to hide.

The supply-chain version is simple. You see a new library mentioned somewhere and clone the repo. Inside is an agent config file with instructions like “pipe test output through this endpoint for logging.” The agent reads the file, trusts it, and starts shipping environment details and test output to a server you do not control.

The defense is to treat the config file as code, not documentation.

Review it before you trust it. Never let your harness auto-load MCP servers shipped inside cloned repositories. An MCP server13 is arbitrary code running with agent permissions. Combined with a poisoned config file, that is a very clean supply-chain attack from a single git clone.

The nastier version is when a command looks completely normal:

These look identical.

But they are not. The Latin “i” is ASCII code point 105. The Cyrillic “і” is Unicode code point 1110. To your terminal, these are different characters. To your eyes, they are the same. And shell renders Unicode exactly as it appears and runs whatever is on the other end.

The general defense is simple.

Assume the agent will be fed input it should not trust. Inspect both the inputs (config files, MCP servers) and the outputs (commands about to run). Catch problems before they reach a place where they can do damage.

Prevents: attacks where the agent is the vector. Examples: a poisoned config file, or a tool that looks fine but isn’t.

Use when: agent reads any input authored outside your team. Which, increasingly, is most of the time.

24. Structural code linting

Most linters work on the surface of the code: formatting, imports, naming.

Structural linting works on the syntax tree underneath. The parser stops looking at characters in sequence and starts looking at meaning. This is a function definition. These are its parameters. This default value is a list.

AST-grep (named after Abstract Syntax Tree14, AST) lets you write rules against that structure.

This matters for AI-written code because LLMs do not make human mistakes. The agent will rarely type a variable name incorrectly. Instead, it writes syntactically perfect code. The code passes lint, type checks, and maybe even tests. But the patterns underneath are subtly wrong.

Mutable default arguments are the classic example:

This catches def process(items=[]).

The default list is created once at function definition and shared across every call. A pattern the agent learned from millions of training examples without understanding why it is dangerous.

If the agent keeps producing the same anti-pattern, encode it as a rule and stop having the same conversation forever. Each rule comes with tests, and once it exists, you wire it into pre-commit and CI.

Prevents: structurally bad code that linters and type checkers do not flag. Especially the LLM favorites: mutable defaults and swallowed exceptions. Or bare excepts that catch KeyboardInterrupt along with real errors.

Use when: agent writes code in a language with a parser you can target (which is most languages now).

25. Pre-commit gates

Local quality gates that block bad code before it becomes a commit.

Pre-commit is the dominant implementation. You define a set of checks, and every commit has to pass them before it becomes history.

What changes with agents is not the mechanism but the value of being slightly distrustful. Rules that can feel rigid for humans are often exactly right for agents. The agent does not get annoyed, ignore the rule, or promise to tidy things up later. It hits the gate, fails, reads the error, and tries again.

Without this gate, there is nothing between the agent’s output and your git history.

It hardcodes a secret into a config file, commits it, and now you are doing incident response because the model wanted the test to pass quickly.

Here’s a useful four-layer config15:

Standard hooks for the basics: trailing whitespace, valid YAML and TOML, file size limit.
Ruff for linting and formatting, with --fix to auto-correct what it can.
Bandit for security issues like hardcoded passwords and eval().
AST-grep for the structural rules from concept 24.

The corrective loop is what makes this so valuable.

The agent writes code, tries to commit, gets rejected, rewrites the function, and commits cleanly a few seconds later. The enforcement teaches.

Prevents: bad code becoming git history. Hardcoded secrets, banned patterns, lint errors.

Use when: any project where an agent has commit access. Which is essentially every project an agent works on.

Second layer of automated checks. Pre-commit runs on your laptop before each commit. CI runs on a shared server whenever someone pushes code. The two together catch what each would miss alone.

When code is pushed and a PR is opened, a fresh server clones the repository and runs the same checks as pre-commit. Lint, type checks, tests, security scans, and structural rules. Just running where the agent cannot quietly skip them.

One practical note: add a concurrency block to your CI config that cancels stale runs when a new push arrives. Agents are prolific little branch goblins. Without this, you burn through CI minutes on runs that were obsolete thirty seconds after they started.

Prevents: local-only enforcement. Misconfigured pre-commit hooks, --no-verify slips, broken clean-checkout assumptions.

Use when: any repo has multiple agents or humans pushing code. Which is almost all of them.

Observability

Ready for the best part?

27. Tracing

Once an agent has run, the question is what actually happened.

Tracing is the answer at the highest level: a step-by-step record of the run, structured so you can navigate it.

Here’s what a useful trace shows:

Tree of tool calls (subagent A called tool X, which called tool Y).
How long each step took.
The inputs and outputs at every node.
Model’s stated reasoning at each decision point.

The shape matters as much as the contents: a flat list of tool calls is hard to follow, a tree gives you the structure of the reasoning.

The agent itself produces some of this for free. Most harnesses log tool calls and their results. Going further requires more setup. You need to capture intermediate model thoughts, timing, model versions, and system prompts at the moment of each call. This means a tracing-aware harness or a dedicated layer like LangSmith, Helicone16, or an open-source OpenTelemetry agent tracer.

Once you have traces, every other observability concept gets easier:

replay starts from a trace
metrics aggregate over many traces
failure investigation almost always begins by opening one and walking it step by step

28. Logging

Logging is the substrate everything else builds on.

A raw, append-only record of inputs, outputs, and tool calls per run. Without it, traces have nothing to structure, replay has nothing to rerun, and metrics have nothing to count.

Here’s the minimum useful set:

Every model call: full prompt, response, latency, tokens, model version.
Every tool call: name, parameters, result, latency.
Every error.
A session ID tying them together.

Skip the cleverness here.

Verbose, append-only, structured as JSON Lines is the format that survives.

The design call is what you keep and for how long. Token budgets and storage costs are something to keep in mind, but losing the inputs to a session that produced an unexplained outcome is much worse than the storage bill.

Default to keeping everything and trimming later, not the other way round.

29. Replay/debugging failed runs

Once you have a complete log of a session, you can rerun it.

Hand the harness the same inputs. Swap the model calls between the recorded responses and the fresh ones. Watch the agent take the same path. Or a different one.

The use cases split cleanly.

To reproduce a bug, replay with recorded model responses, and you get exactly the same trace. That is what you want when isolating a failure in the harness or a tool. To test a fix, replay with fresh model calls and you see whether your change to the harness, system prompt17, or a tool fixes the failure.

Replaying against a different model is trickier.

Sometimes a bug only shows up on a specific model version, and an upgrade silently breaks something. Replay helps catch it before your users do.

30. Metrics

Most of what you can measure about an agent is proxy data:

Latency per session
Latency per tool call
Token and dollar cost
Tool call counts
Failure counts

All of it falls out of the logs without much extra plumbing18.

Proxy data is enough to spot the obvious failure modes. A session running up the bill. An agent stuck in a loop because nothing tells it to stop. You will catch a lot of nonsense just by watching the basics.

Outcome data is harder.

The honest version of “did the agent succeed” is not something the agent itself can report on. “Task complete” is a claim, not a measurement. To get a real outcome signal, you need to anchor to artifacts the agent did not produce: a test that passed in CI, a PR that merged, a deploy that did not roll back.

Wiring those up is brittle and project-specific. But it is the only way to know whether the proxy numbers are pointing in the right direction.

Final thoughts

That was a lot of concepts, so a quick recap before you close the tab.

Foundations gave you what an agent is, how it runs, where its state lives, and the patterns you compose them in.

The four practical layers cover what comes after that.

Configuration shapes how the agent behaves before it writes a line of code. Capability is what it can reach for. Orchestration coordinates multiple agents without letting them step on each other. Guardrails stop them from doing damage.

Observability is how you see what actually happened once the agent stopped typing.

If you are starting from zero, here is a recommended order for picking these up:

Write a project config file (concept 5)
Wire up MCP for live docs (concept 10)
Turn the default sandbox on (concept 20)
Start using subagents for read-heavy work (concept 16)

This is a good place to start.

👋 I’d like to thank Paul Hoekstra for writing this newsletter!

He’s a data engineer at Digital Power, a data and AI consultancy, and writes Paul’s Pipeline, a free newsletter on AI, data, and side projects.

Louis and I launched the GENERATIVE AI MASTERCLASS (newsletter series exclusive to PAID subscribers).

When you upgrade, you’ll get:

Simple breakdown of real-world architectures
Frameworks you can plug into your work or business
Proven systems behind ChatGPT, Perplexity, and Copilot

👉 CLICK HERE TO JOIN THE GENERATIVE AI MASTERCLASS

If you find this newsletter valuable, share it with a friend, and subscribe if you haven’t already. There are group discounts, gift options, and referral rewards available.

Explore Kotlin Multiplatform

👋 Find me on LinkedIn | Twitter | Threads | Instagram

Want to reach 210K+ tech professionals at scale? 📰

If your company wants to reach 210K+ tech professionals, advertise with me.

Thank you for supporting this newsletter.

You are now 210,001+ readers strong, very close to 210k. Let’s try to get 211k readers by 27 June. Consider sharing this post with your friends and get rewards.

Y’all are the best.

A tool is something an AI can use to interact with the outside world, such as reading files, searching the web, running code, or calling APIs.

The harness is the code around the model that runs its tool calls and feeds the results back. The model decides; the harness does.

Frontmatter is a small block of metadata placed at the top of a Markdown file.

Skills: reusable instruction files that teach the agent how to do a specific task.
Hooks: automatic checks or actions that run at certain points, like before a shell command executes.
Slash commands: shortcut commands you type, like /review or /test, to trigger a predefined workflow.

The transformer is the neural network architecture behind modern LLMs. Introduced by Google in 2017, it processes a whole sequence of tokens at once and learns which parts to pay attention to. Almost every model you have heard of (GPT, Claude, Gemini) is a transformer under the hood

A vector is a numerical representation of text that lets computers search for similar meanings.

Harness
- The software that runs the agent.
- Manages prompts, tool calls, retries, permissions, and the agent loop.
- Example: Claude Code’s runtime.
Sandbox
- A safety boundary that limits what the agent can access.
- Restricts files, network access, credentials, and system resources.
- Prevents the agent from causing damage outside its allowed area.
Tool Loop
- Repeated cycle of:
  1. Think
  2. Use a tool
  3. Observe the result
  4. Repeat
  This is what makes an agent different from a single prompt.
Container
- An isolated environment where the agent runs code and tools.
- Has its own filesystem, processes, and dependencies.
- Similar to a lightweight virtual machine (e.g., Docker container).

A pay-as-you-go API means you pay based on how much AI you use, not a fixed monthly subscription. The unit of usage is usually tokens.

A token is roughly:

1 word ≈ 1-1.5 tokens
“Hello world” = about 2 tokens
A long article = thousands of tokens

Think of it like:

Claude Pro = Netflix
- fixed monthly fee
- use within plan limits
API = electricity bill
- pay for exactly what you consume

chmod changes file permissions. And 777 gives absolutely everyone full access.

curl downloads content from the internet. sh executes shell commands.

A tool call is a structured request from an AI agent to execute a tool, such as reading a file, running a command, or calling an API.

A homoglyph is a character that looks like another character but is actually different.

Hostname homoglyph: a website/domain name containing look-alike characters from different alphabets.
Path homoglyph: a file path containing look-alike characters that visually resemble normal letters but point somewhere else.

An MCP server is a program that exposes tools, resources, or services to AI agents through the Model Context Protocol.

An Abstract Syntax Tree (AST) is a structured representation of code that describes its meaning and structure rather than its raw text.

Standard hooks = file sanitation
Ruff = code quality
Bandit = security
AST-grep = structural correctness

Workflow:

Bad code -> Pre-commit fails -> Agent reads error -> Fixes code -> Tries again

LangSmith is an observability and debugging platform for LLM apps and agents.
Helicone is like an analytics and monitoring layer for AI API calls.

A simple rule:

Want to know why an agent made a bad decision? → LangSmith
Want to know which model cost you $2,000 yesterday? → Helicone

A system prompt is the hidden instruction set that defines an AI agent's role, rules, and behavior.

User prompt = what the user wants
System prompt = rules the agent must follow

Plumbing is the underlying infrastructure and integration code that connects systems together behind the scenes

How Docker Works - A Deep Dive

Neo Kim — Sun, 14 Jun 2026 09:45:11 GMT

Share this post & I'll send you some rewards for the referrals.

You’ve heard it before: “It works on my machine.”

The app is under development. Tests pass, dependencies install, and everything looks ready to ship. But the moment it reaches production, something breaks. The server might use a different Python version, or a library may exist locally but not in the deployment environment. Sometimes the operating system itself causes the issue.

The problem was never just the application…it was the environment around it.

For a long time, virtual machines were the best solution engineers had.

Instead of deploying only the application, teams started packaging an entire operating system along with it. This approach fixed a big part of the consistency problem. The environment behaved the same no matter where the workload ran. But the tradeoff was hard to ignore.

Each virtual machine included its own kernel1, services, and system overhead to run a single application. They consumed a large amount of memory, took time to boot, and became too difficult to manage as the infrastructure grew.

What the industry needed was a way to isolate applications without paying the cost of a full operating system every time.

The interesting part is GNU/Linux already had most of the building blocks required to make this possible:

Namespaces could isolate processes from each other. cgroups could control how much CPU and memory a process was allowed to consume. The primitives2 already existed, but they were hard to use directly and too complex for most developers to manage on their own.

Docker changed this by turning those low-level kernel features into a developer-friendly workflow.

Engineers no longer had to configure isolation primitives manually. They got a simple CLI, portable images, reproducible environments, and a runtime for running containers at scale.

This shift fundamentally changed modern infrastructure.

Today, containers sit underneath almost everything in modern infrastructure. From CI/CD pipelines to Kubernetes clusters, they power it all. But despite how common they are, most engineers interact only with the surface layer. They know the commands. They know how to build images and start containers. What they rarely see is the runtime stack operating underneath those commands.

This newsletter is about the lower layer.

Onward.

Building across platforms shouldn’t mean rewriting the same logic 3 times (Partner)

A validation rule in your backend, Android app and iOS should behave the same.

But with different codebases, logic diverges and mobile apps might behave differently than your backend.

Kotlin Multiplatform (KMP), developed by JetBrains, is designed to solve this at the architecture level.

With KMP, you define a shared domain layer and reuse it across platforms instead of duplicating business logic.

It’s built around:

Consistency: same app behaviour runs across Android, iOS, desktop, web and server.
UI sharing flexibility: keep your native UIs or extend KMP with Compose Multiplatform to share user interfaces as well.
Low-risk: integrates into existing ecosystems safely, preserves native performance and hiring flexibility – continue recruiting Android and iOS engineers instead of searching for niche cross-platform specialists.
Proven maturity: KMP is trusted in production by global enterprises like Bolt and Duolingo for serious, long-lived applications.

Whether teams choose to share their UI or keep it native, KMP keeps the business logic perfectly aligned.

Explore how teams are using Kotlin Multiplatform in production:

(Thanks to JetBrains for partnering on this post.)

I want to introduce Ayaan as a guest author.

He’s a self-taught DevOps engineer focused on Kubernetes & cloud infrastructure. He builds in public and writes about cloud-native systems, sharing daily technical tips and projects.

Check out his work and socials:

Twitter
Github

Here’s what you’ll find inside this newsletter:

The full Docker runtime stack, from the CLI down to the Linux kernel.
Linux primitives behind containers: namespaces for isolation and cgroups for resource control.
How container images, layers, and OverlayFS actually work.
The complete path from Docker run to a running process, step by step.
How container networking and storage work under the hood.
What Docker gets right, where it struggles, and where containers start to break down.

Docker Architecture

Docker is NOT one program.

It is a stack of cooperating components where every layer has a specific responsibility and passes work to the next.

Most engineers interact with the CLI and assume Docker handles everything behind the scenes. But when you execute docker run, the command travels through an entire runtime chain before a process ever starts.

Understanding this chain makes the rest of Docker's internals easier to follow…

Every layer has a specific responsibility.

Some components manage APIs. Some manage container lifecycle operations. Others interact directly with the Linux kernel. That separation keeps the runtime modular and easier to maintain.

Let’s see how the chain works:

Docker CLI

The Docker CLI3 is a thin client.

When you type docker run or docker build, it converts your command into an API request and sends it over /var/run/docker.sock4 to the Docker daemon.

The CLI itself does not build images, pull layers, or start containers. It only forwards requests.

The real work happens behind it.

This separation matters because the CLI and daemon do not even need to run on the same machine. Docker can expose its API remotely, so external tools and automation systems can communicate with the daemon directly.

dockerd

dockerd is the Docker daemon.

It receives requests from the CLI and manages images, networks, volumes5, and the overall Docker API surface. But dockerd does NOT run containers directly. Instead, it hands container lifecycle management to containerd.

This separation keeps containers running even if the Docker daemon restarts.

So the runtime layer lives independently from the higher-level Docker API layer.

The daemon also acts as the orchestration layer for local Docker operations.

When you pull an image, create a network, or attach a volume, the request first goes through dockerd. Then it reaches the lower runtime components.

containerd

containerd is the core container runtime responsible for managing container lifecycles.

It manages image pulling, snapshots, image unpacking, and runtime operations6 through a gRPC7 API.

When dockerd needs to start a container, it delegates that work to containerd. From that point onward, the Docker daemon mostly steps out of the execution path.

containerd originally started as part of Docker itself.

In 2017, Docker donated it to the Cloud Native Computing Foundation (CNCF) as an independent project. Today, Kubernetes communicates with containerd and other CRI-compatible runtimes instead of Docker.

This shift significantly changed the container ecosystem.

Docker remained the developer platform, while containerd became the runtime underneath orchestration systems. Even then, containerd does not directly create containers. It passes that responsibility further down the stack.

containerd-shim

The containerd-shim is a lightweight process between containerd and the running container.

Before a container starts, containerd launches a shim process first. The shim becomes the parent of the container process and keeps the container alive even if containerd crashes or restarts.

Each container gets its own shim. That process isolates the running container from the runtime layer and keeps the connection alive.

The shim also manages container input/output streams and exit status reporting. Without it, containerd would need to stay directly attached to every running container process.

Once the shim is ready, it calls the component that actually creates the container.

runc

runc is the Open Container Initiative (OCI) runtime that directly interacts with the Linux kernel to create the container.

It reads the container configuration and creates namespaces and cgroups. Then it configures the filesystem and starts the container process.

This is the layer where containers stop behaving like Docker objects and become ordinary Linux processes. The kernel enforces its isolation boundaries.

After the process starts, runc exits…

Its job is complete. The shim continues supervising the container and reports its status back to containerd.

OCI Standards

Every component in this stack works together because they follow the same OCI standards.

The Open Container Initiative (OCI) defines open standards for container images and runtimes. This standardization allows different tools to interoperate cleanly across the container ecosystem.

Docker-built images run unchanged on containerd, CRI-O, and Podman8. Kubernetes can swap one OCI runtime for another without changing the rest of the stack.

Before OCI standardization, container tooling was far more tightly coupled. Docker helped popularize containers, but OCI allowed different container tools to work together.

OCI standards made container runtimes modular rather than tied to a single vendor or toolchain.

Now the runtime stack is clear, let’s look at the Linux primitives that actually make containers possible…

Linux Primitives: Namespaces

Containers are not a feature of Docker… They are a feature of the Linux kernel.

Docker provides the tooling, CLI, image format, and workflow. But the isolation that makes containers actually work comes from the kernel itself. And at the center of that isolation is a single primitive: namespaces.

A namespace wraps a global system resource and makes it appear private to a specific process.

The process inside the namespace believes it has its own isolated version of that resource. Host and every other process on the machine still see the original global resource.

Linux has six namespaces that containers rely on. Each one isolates a different part of the system. Together, they create the illusion that a container is running on its own machine.

Let’s go through each one:

1. PID Namespace

Every process on a Linux system has a process ID (PID).

On the host, these IDs are global. All the running processes share the same numbering space.

The PID namespace changes this.

When a container starts inside its own PID namespace, the first process it runs gets assigned PID 19. From inside the container, that process looks like the root of the entire system. It can see its own child processes, but it cannot see anything running on the host or in other containers.

While the host sees the container process with its real PID in the global namespace. The container has NO idea.

It matters because PID 1 carries special responsibilities in Linux. It’s expected to handle signals and reap orphaned child processes. Containers that do not handle this correctly end up with zombie processes. That is why what runs as PID 1 inside a container is an important design decision.

2. Network Namespace

By default, every process on a Linux machine shares the same network stack.

They see same interfaces, same routing tables, and same ports.

The network namespace gives a container its own isolated network stack. Each container gets its own virtual network interface, IP address, routing table, and set of ports10. A process inside the container can bind to port 80 without conflicting with anything else on the host.

This is how different containers can each listen on port 80 simultaneously on the same machine. They are not actually sharing the same port. Instead, each has its own private network namespace.

Docker creates a virtual Ethernet pair to connect each container’s network namespace to the host. Traffic flows through that pair, and the host routes it appropriately.

This bridge makes container networking possible, but the namespace is what makes the isolation real.

3. Mount Namespace

The mount namespace controls what a process can see when it looks at the filesystem.

Without it, every process on the system shares the same view of the filesystem. Any process could see every mounted disk, every directory, and every file on the machine.

With a mount namespace, each container gets its own filesystem view. When Docker starts a container, it sets up a layered filesystem using OverlayFS11 and mounts it inside a new mount namespace. The container sees only that filesystem. It cannot browse the host’s directories or access files outside its own root.

This gives each container its own isolated root filesystem.

It includes its own operating system files, libraries, and application code without touching the host.

4. UTS Namespace

The Unix Time-Sharing (UTS) namespace12 controls hostname and domain name of a process.

This one is simpler than the others, but it matters for how containers identify themselves on a network. Without it, every container would share hostname of the host machine. Without it, logs, monitoring systems, and service discovery tools would struggle to distinguish containers from the host.

The UTS namespace lets each container have its own hostname.

When a container runs hostname, it sees the name Docker assigned to it, NOT the name of the underlying host. That small detail keeps container identities clean and predictable.

5. IPC Namespace

Linux lets processes communicate directly through mechanisms like shared memory segments and message queues.

The Inter-Process Communication (IPC) namespace isolates these mechanisms13. Processes inside a container can only communicate with other processes in the same IPC namespace. They cannot access another container or the host via shared memory.

Without this isolation, a process in one container could potentially interfere with or read data from shared memory segments belonging to another container on the same host. The IPC namespace closes this gap.

6. User Namespace

The user namespace is the most powerful of the six, and also the most nuanced.

It maps user IDs and group IDs inside the container to different IDs on the host. A process can appear to run as root inside the container while actually mapping to an unprivileged user ID on the host.

This distinction is critical for security.

Many containers run their processes as root by default. Without user namespaces, the process is root on the host too. A container escape14 would give an attacker full root access to the machine.

With user namespaces enabled, root inside the container is just a regular unprivileged user on the host. The blast radius of a container escape shrinks significantly.

User namespaces are what make rootless containers15 possible.

Docker, Podman, and other container runtimes use them to let engineers run containers without needing root privileges on the host at all.

How Namespaces Work Together

No single namespace makes a container.

It is the combination of all six working together that creates the isolation engineers rely on:

The PID namespace hides host processes.
Network namespace provides private networking.
Mount namespace gives each container its own filesystem.
UTS namespace sets a clean identity.
IPC namespace prevents cross-container memory interference.
User namespace limits the damage if something goes wrong.

But namespaces only control what a container can see.

They say nothing about what it can consume. That is where the second Linux primitive is used…

Reminder: this is a teaser of the subscriber-only newsletter, exclusive to my golden members.

When you upgrade, you’ll get:

Architecture breakdown of core DevOps systems.
Deep dives into Docker, Kubernetes, Terraform, and so on.
Practical insights into how real-world systems achieve scalability, automation, and reliability.

Let’s keep going!

Linux Primitives: cgroups

Designing a Payment Backend with Stripe Integration

Hayk — Wed, 10 Jun 2026 10:36:01 GMT

Share this post & I'll send you some rewards for the referrals.
Block diagrams created using Eraser.

A payment backend is one of the few systems where a bug doesn’t just cause a bad user experience but real money to vanish from user accounts.

This makes payment system design fundamentally different from most backend engineering…

The core challenge here is NOT performance or feature richness, but correctness under failure. Every design decision, from database choice to retry policy, must answer one question: what happens when this component fails mid-transaction?

This is a complete system design blueprint for building a payment backend that integrates Stripe as the payment service provider.

Onward.

The agent harness wasn't supposed to be the black box (Partner)

Agent loop is the most important piece of infrastructure in your workflow right now, and for most developers, it’s the one piece they can’t open up.

Agent builders have to jump through all the hoops themselves, crafting the infrastructure and tools, testing the harness, while fighting to maintain what they’ve built.

Meet Cline SDK: agent harness behind Cline 2.0, fully open-sourced. The same runtime that powers Cline across VS Code, JetBrains, and the CLI is now an npm install away: npm i @cline/sdk. Inspect it, fork it, extend it, ship on it.

Best-in-class harness: 74.2% on Terminal-Bench 2.0 with Claude Opus 4.7 ahead of Claude Code (69.4%) and strongest numbers published on open-weight models.
Open model & provider choice: Anthropic, OpenAI, Google, Bedrock, Mistral, or any OpenAI-compatible endpoint.
Real plugin system: Register tools, hooks, commands, providers, message builders. Prototype as a local file, harden into a package. Extend it freely for any of your agent use cases.
Scheduled + event-driven agents: Cron and event specs for PR reviews, dependency checks, coverage audits, changelogs no separate orchestration layer.

Stop building around your agent. Start building on it.

Install Cline SDK today: npm i @cline/sdk Or try the rebuilt harness directly: npm i -g @cline

Get Started Today

(Thanks to Cline for partnering on this post.)

I want to reintroduce Hayk Simonyan as a guest author.

He’s a senior software engineer specializing in helping developers break through their career plateaus and secure senior roles.

If you want to master the essential system design skills and land senior developer roles, I highly recommend checking out Hayk’s YouTube channel.

His approach focuses on what top employers actually care about: system design expertise, advanced project experience, and elite-level interview performance.

Inside this newsletter, you’ll get:

Three approaches to accepting payments. Why no-code checkout hits limits fast, when building your own processor makes sense, and why integrating a PSP like Stripe is the right call for most companies.
How money actually moves. The full card transaction lifecycle from authorization to settlement, including the six entities involved and the engineering risks most developers miss.
High-level architecture. The seven components of a production payment backend, how they split the synchronous and async paths, and why each boundary exists.
Idempotency and exactly-once processing. How to design a system that never double-charges, even when servers crash mid-transaction, using idempotency keys, recovery points, and the atomic phases pattern.
Webhook handling and the payment state machine. How to process Stripe webhooks reliably when duplicates and out-of-order delivery are expected, and how to enforce valid state transitions at the database level.
Designing for high availability. How to approach 99.999% uptime across your API layer, database, and async workers, and why a circuit breaker in front of Stripe protects your own system more than it protects Stripe.

Three Approaches to Accepting Payments

Before diving into the design, it’s worth understanding the landscape…

When a company needs to accept payments, there are three broad approaches, and choosing the wrong one for your scale wastes either years of engineering time or millions in unnecessary fees.

Option 1: Build your own payment processor

This means connecting directly to card networks (Visa, Mastercard), acquiring banking licenses, handling PCI DSS1 (Payment Card Industry Data Security Standard, which is the set of security requirements every company that touches card data must comply with), Level 1 compliance2, building your own fraud detection, and managing relationships with issuing banks.

The upside is lower transaction costs.

At a massive scale, even saving 0.5% off each transaction adds up to hundreds of millions per year. Amazon, Uber, and Airbnb have all moved in this direction for parts of their payment stack.

The downside is the cost to get there: acquiring necessary licenses alone takes 12-24 months, costs millions, and comes with ongoing regulatory obligations. So this approach only makes sense once you are processing billions in annual payment volume.

For anyone else, it is premature optimization at its extreme…

Option 2: Use a payment service provider (PSP) like Stripe, PayPal, Adyen, or Braintree

A PSP is a company that handles the entire payment processing chain on your behalf: card network integrations, banking relationships, fraud tooling, PCI DSS compliance, and global payment method support.

You pay a per-transaction fee, typically around 2.9% + $0.30 with Stripe, in exchange for not having to build or maintain any of that infrastructure. This is the right approach for most companies, from early-stage startups to large-scale platforms.

Even Shopify, which processes billions in payments annually, built its own payment product (Shopify Payments) on top of Stripe’s infrastructure rather than connecting directly to card networks.

Option 3: Use a no-code payment flow, such as Stripe Checkout

Stripe offers hosted checkout pages and payment links that require almost no backend integration.

You create a checkout session, redirect the user to Stripe’s hosted page, and Stripe handles everything: UI, payment method selection, and confirmation.

This is the fastest way to accept payments, and it works well for simple use cases: selling a single product, collecting donations, or building a quick prototype.

The trade-off is limited customization and less control over the payment experience.

You cannot deeply embed it in your own UI, easily build custom subscription logic, or react to payment events in real time with full control over the flow.

How Money Moves Through Card Network

Understanding the payment flow end-to-end is a prerequisite to designing a backend that handles it correctly…

Six entities participate in each card transaction:

Cardholder
Merchant (Business or individual selling the product or service)
PSP (Payment Service Provider, in this case, Stripe)
Acquiring bank (Merchant’s bank)
Card network (Visa/Mastercard)
Issuing bank (Cardholder’s bank)

Money flows through 3 distinct phases, each with different timing and failure modes…

Phase 1: Authorization (1-3 seconds)

The customer submits card details through Stripe Elements.

These are technically iframes hosted by Stripe, even if they look like part of your site. We use Stripe.js, which is a JavaScript library that Stripe requires you to load directly from their servers (for security reasons).

The process starts on the backend, where a PaymentIntent3 gets created via Stripe’s API to track the transaction lifecycle. This returns a client_secret to the frontend.

The customer then enters their card details into Stripe Elements - secure iframes hosted by Stripe that look like part of your site. Using Stripe.js, the client-side code tokenizes the card details so the PAN (Primary Account Number: a 14 to 16-digit number on a credit or debit card) never reaches the merchant’s server.

What is PaymentIntent?

A PaymentIntent is a stateful object Stripe uses to track a single payment from start to finish.

Think of it as the source of truth for a transaction…

In older systems, you’d just send a “charge” request and hope it worked. With modern payments, things are more complex because of 2FA (e.g., 3D Secure4) or fraud checks.

PaymentIntent manages this by moving through different states:

Requires payment method: You created the intent, but the user hasn’t typed their card yet.
Requires action: User must verify the payment in their bank app.
Processing: Stripe is talking to the banks.
Succeeded: Money is authorized or captured.

Stripe formats an ISO 8583 authorization message5 and sends it to the acquiring bank6, which identifies the card network via the BIN7 (first 6-8 digits) and forwards the request.

The card network routes it to the issuing bank, which checks card validity, available funds, CVV/AVS8 match, and fraud risk models.

The issuer returns a two-digit response code, approve or decline, back through the entire chain.
If approved, a hold gets placed on the cardholder’s available balance. But no money moves yet.

Mastercard reports an average network response time of 130 milliseconds; the full round-trip, including all hops, completes in 1-3 seconds.

Visa’s network handles a peak of 56,000+ messages per second.

Phase 2: Capture (immediate/delayed)

Capture9 is when the merchant tells acquiring bank (the merchant’s bank) to finalize the authorized amount.

For digital goods and subscriptions, capture occurs immediately. Stripe’s default is capture_method: automatic.

For physical goods, hotels, or ride-hailing, capture is delayed until fulfillment. You can capture an amount less than or equal to what was authorized, but never more. Authorization holds10 typically last 5 to 10 days.

Visa allows 10 days for standard e-commerce and up to 31 days for lodging.

Here’s the Engineering Risk:

If you don’t capture within 7 days, the issuing bank might cancel the hold. This is a common source of bugs. Your state machine must treat AUTHORIZED and CAPTURED as “distinct” states. If your system thinks a payment is authorized but the bank has dropped the hold, the capture call will fail.

So you need a background job to detect these “stuck” payments and mark them as expired.

Phase 3: Clearing and settlement (T+1 to T+3)

At the end of the business day, captured transactions get batched and sent to the card network for clearing.

The network calculates fees and exchanges transaction files overnight…

Settlement is when the actual money moves. The issuing bank sends the funds (minus fees) to the card network, which then passes them to the acquiring bank.

The acquiring bank finally deposits them into the merchant’s account.

In banking, “T” stands for the Transaction Date. T+2 means the money arrives two business days after the transaction. Stripe’s US default is T+2, though new accounts usually have an initial hold of 7 to 14 days.

Fee Breakdown

On a $100 US online transaction, the 2.9% + $0.30 fee gets split three ways:

Interchange11 (~$2.05): This goes to the Issuing Bank (the customer’s bank). This is the largest cut. It covers the bank’s risk and pays for the customer’s credit card rewards.
Assessment (~$0.16): This goes to the Card Network (Visa or Mastercard) for using their rails.
Markup (~$0.70): This is what Stripe keeps for providing the API, security, and infrastructure.

Functional and Non-Functional Requirements

Functional requirements for a payment backend serving a mid-to-large platform fall into 6 categories:

Customers must be able to make one-time payments and save payment methods for reuse.
Merchants or the platform must be able to accept payments, with support for marketplace-style split payments if needed.
Refunds, both full and partial, must flow through a dedicated refund state machine.
System must handle subscriptions and recurring billing, including proration on plan changes, trial periods, and Stripe’s Smart Retries for failed renewal payments.
Backend must process Stripe webhooks reliably, as webhooks are the source of truth for payment status.
The entire payment lifecycle must be modeled as a finite-state machine with enforced transitions.

Non-functional requirements are where payment systems diverge sharply from typical backend services:

Exactly-once payment processing is the key requirement. A double charge on a $1,000 purchase means money taken from a person twice. True exactly-once delivery is impossible in distributed systems (Two Generals Problem12), so the practical implementation is at-least-once delivery combined with idempotent processing and reconciliation.
High availability targets are extreme. Stripe itself maintains 99.999% API uptime.
Idempotency must be enforced at every layer: client-to-backend, backend-to-Stripe, and webhook processing. Every retry must produce the same result as the original request.
Consistency over availability. Payment systems are one of the few domains where the CAP theorem should tilt decisively toward consistency. A stale read that shows an incorrect balance or a lost write that drops a payment is far worse than brief unavailability. This is why every major payment platform, including Shopify, Uber, and Airbnb, uses SQL databases with ACID guarantees for core payment data.

Scale estimates for a mid-to-large platform

A platform processing 100,000 payments per day generates roughly 500,000-1,000,000 webhook events daily (each payment triggers 5-10 events across creation, authorization, capture, and related objects).

Stripe’s default API rate limit is 100 requests/second per account, with individual endpoints limited to 25 requests/second; higher limits are available by arrangement.

Payment API latency should be under 5 seconds end-to-end, including the Stripe round-trip, with internal service-to-service calls under 100ms.

Reminder: this is a teaser of the subscriber-only newsletter series, exclusive to my golden members.

When you upgrade, you’ll get:

High-level architecture of real-world systems.
Deep dive into how popular real-world systems actually work.
How real-world systems handle scale, reliability, and performance.

High-Level Architecture

The architecture splits into seven components, each with a distinct responsibility.

It isolates the synchronous customer-facing path from asynchronous processing and keeps the webhook ingestion pipeline decoupled from business logic.

1 Payment API Service

It’s the synchronous entry point.

It receives payment requests from checkout, validates input, checks idempotency keys13 against the database, creates or retrieves the payment record, and returns an immediate response to the client.

For Stripe integrations, this service creates a PaymentIntent and returns the client_secret to the frontend, which uses Stripe.js to complete the payment (including 3D Secure challenges).

The API service should never block on downstream processing. It creates the payment record, dispatches work, and returns.

2 Stripe Integration Layer

It abstracts all Stripe-specific API calls behind a uniform interface.

This adapter handles Stripe’s error types, maps them to internal error codes, attaches idempotency keys to every POST request, and manages timeouts. Shopify and Airbnb both use this pattern; Airbnb calls it the “PSP adapter,” which isolates provider-specific logic from the core payment domain.

If you ever need to support a second PSP (Adyen, Braintree), only this layer changes.

3 Payment Database

This is the system of record.

It stores the current state of every payment, idempotency keys, and the immutable audit log. The schema design is covered in the next section, but the critical principle is: payments table holds a mutable current state (optimized for queries), while the payment events table holds an append-only immutable log of every state change (optimized for audit, debugging, and reconciliation).

Both exist in the same SQL database and get updated within the same transaction.

4 Webhook Receiver

It’s a lightweight HTTP endpoint that does exactly three things: verify the Stripe signature (HMAC-SHA25614 using the Stripe-Signature header), store the raw event, and return 200 OK.

It must respond within seconds, since Stripe has an approximately 20-second timeout. All business logic happens asynchronously. The receiver enqueues the event onto the message queue for processing by workers.

5 Message Queue

It (Kafka, SQS, or RabbitMQ) decouples webhook receipt from processing and provides at-least-once delivery guarantees.

Uber uses Apache Kafka as the backbone of their payment platform’s async stream processing. The queue also supports the transactional outbox pattern15: when a payment state change is committed to the database, an outbox record is written in the same transaction, then relayed to the queue by a separate process.

6 Background Job Workers

They handle 4 categories of async work:

Webhook event processing: dequeuing events and applying state transitions idempotently,
Retry workers: retrying failed PSP calls with exponential backoff,
A reconciliation worker: daily comparison of internal records against Stripe’s records,
And a stuck-payment detector: alerting on payments in intermediate states beyond a configurable threshold.

7 Ledger

It provides double-entry bookkeeping16.

Every money movement is recorded as a balanced pair of debit and credit entries. Uber explicitly built its next-generation payment platform on double-entry bookkeeping for auditability, and Stripe’s own ledger system logs approximately 5 billion money-movement events daily.

The ledger is append-only.

If a mistake needs correction, a new reversing entry is inserted, never an update or delete.

The high-level architecture tells us what to build, but database design is where correctness is either enforced or lost. Every pattern we covered above, idempotency, exactly-once processing, the state machine, all of them depend on the database schema doing the right thing.

If we get this wrong, no amount of application-level logic will matter.

So let's see what the database design looks like…

Database Schema to Enforce Correctness

The Anatomy of OpenClaw

Neo Kim — Mon, 08 Jun 2026 11:46:05 GMT

Share this post & I'll send you some rewards for the referrals.

You ask ChatGPT to write an email.

It drafts it… But you still have to copy the text, paste it into Gmail, and hit send. AI did the thinking,,,YOU did the task.

OpenClaw changes this…

It’s an open-source autonomous AI agent1 that runs on your machine, connects to your messaging apps, and does the work itself. You could message it on Telegram: “Check my inbox, pull any invoices, save the attachments.” Then it connects to your email, finds invoices, downloads the attachments, saves them, and messages you back with a summary.

You don’t have to open a single app…

OpenClaw bridges the gap between the thinking layer and the act layer.

Large language models (LLMs), such as Claude, ChatGPT, and Gemini, provide reasoning. While OpenClaw2 provides hands.

Onward.

Run OpenClaw with MyClaw

The best AI assistants aren’t the ones waiting for your next prompt—they’re the ones that send you the answer before you think to ask.

MyClaw runs OpenClaw in the cloud, schedules recurring tasks, and delivers updates directly to Telegram or WhatsApp, so it feels less like another chatbot and more like an assistant that’s actually on the job.

If this newsletter got you interested in OpenClaw, MyClaw is an easy way to see what an always-on AI assistant feels like.

Try MyClaw

(Thanks to MyClaw for partnering on this post.)

Here’s what you’ll find inside this newsletter:

What OpenClaw actually does. Why it’s different from every AI tool you’ve already tried.
How the Gateway works. The always-on server running on your machine that makes everything else possible.
Skill system. How to install the right ones first, and what most people get wrong.
What breaks and why. Most common failure modes, with fixes for each.
The real security risks. Documented, avoidable, and worth reading before you touch a single setting.
How to build your own. Full architecture in five layers, from the ground up.

By the end of this newsletter, you will understand OpenClaw well enough to run it.

What Is OpenClaw

OpenClaw is the orchestration layer, the connector between an LLM (like Claude, GPT, or Gemini) and your real-world tools.

The intelligence comes from the LLM you connect to it.

Think of the LLM as the brain: it reasons, plans, and decides what to do. OpenClaw is the hand: it executes those decisions across your files, messaging apps, shell, and APIs.

Here’s what OpenClaw offers:

A gateway3 that runs on your machine (Mac, Linux)
This is the always-on server running on your machine that coordinates everything: your messaging apps, LLM, tools4, and memory.
A multi-channel messaging interface: WhatsApp, Telegram, and so on.
You talk to the same assistant from all these apps, with shared context and memory, instead of managing a different bot per platform.
A skills platform with 5,400+ modular capabilities on ClawHub5.
Skills6 are plain Markdown files that tell the agent how to perform a specific task, like checking your GitHub PRs or sending a Slack message.
A persistent memory system in plain Markdown files you own.
The agent’s knowledge, preferences, and logs are stored as editable Markdown on disk, so you can inspect, version-control, and move its LLM context like code.
Proactive scheduling engine (Heartbeat7 + cron jobs) that acts without being prompted.
OpenClaw can wake itself up on a schedule to check servers, triage email, or send summaries, instead of waiting for you to type a command.
Model-agnostic: plug in Claude, GPT, DeepSeek, Gemini, Grok, or local models via Ollama8.
You can mix and match models (cheap vs powerful, cloud vs local) and route different tasks to different providers to optimize cost and performance.

How It Differs From Other AI Tools

Here’s how:

ChatGPT, Claude, Gemini: You prompt them, and they respond. They have no access to your email, no ability to run scripts, and no memory once you close the tab9.
Siri, Google Assistant: Polished products, but closed ones. You can’t add custom skills, connect arbitrary APIs, or self-host them.
LangChain, CrewAI, AutoGen: Developer frameworks. You write the Python, define the chains, and build the interfaces from scratch. OpenClaw ships all of that out of the box.
Claude Code, Codex CLI: Terminal tools for coding sessions. Reactive and session-based. OpenClaw runs 24/7 for everything, not just code.

So OpenClaw is NOT just a product you use in a browser tab.

It is a service that runs on your machine and continues to work while you’re away.

How OpenClaw Processes Messages

Imagine you send a single Telegram message at 9 pm:

“Check my open PRs and let me know if anything needs a response tonight.”

You lock your phone and go to sleep…

By morning, you will have a reply:

Three PRs are open, and one has a teammate's review comment asking about error handling. OpenClaw drafted a response and flagged it for your approval, leaving the other two untouched because they were still in CI. It also remembered that last week you said you prefer to handle auth-related reviews yourself, so it left the one in the auth module alone.

None of this required you to open GitHub, write a prompt, or copy anything between tabs…

Why OpenClaw Runs 24/7 on Your Machine

When you run the OpenClaw gateway, you start a Node.js server that binds to port 18789 and stays running.

This is the fundamental difference between OpenClaw and a browser-based AI tool…

Every mainstream AI assistant, including ChatGPT, Claude, and even DeepResearch mode, only exists while you have a tab open. (DeepResearch can run a long session, but once it’s done, it stops.)

i.e., if you close the tab, everything disappears.

But OpenClaw never stops:

It runs as a background daemon on your local server and stays alive whether you are at your desk or asleep. That persistence is the foundation of everything else: scheduled jobs, proactive alerts, multi-channel messaging, and long-term memory. None of it works without a process that is always on.

This central server is called the Gateway…

Gateway: Control Center of Your Agent

The Gateway orchestrates everything.

It sits between your messaging apps, LLM, and tools, coordinating every request from the moment it arrives until a reply goes back out.

When a message arrives from Telegram, WhatsApp, or Slack, the Gateway authenticates the sender and creates or retrieves a session for the conversation.

It then assembles the full context10, passes it to the LLM, handles any tool calls that come back, and routes the final reply to the correct channel.

Say you ask the agent to clean up old log files:

Before it runs anything, the Gateway intercepts the shell command and sends you a confirmation request through Telegram: “I’m about to run $ rm -rf ~/logs/*.log, approve this?”

Nothing executes until you say yes… If you don’t respond, it waits…

The Gateway also serves the Control UI at http://localhost:18789.

It’s a browser-based dashboard that lets you watch the agent’s reasoning in real time, approve pending commands, and inspect active sessions. Every tool call, model response, and approval request shows up here live. Since the Gateway runs on your Mac Mini or VPS, you can access the Control UI from any browser on the same network. If you use Tailscale11, you can reach it from your phone or laptop without being on the same network.

The Gateway also supports hot configuration reloads…

You can change your model, update permissions, or add a new channel without restarting the process. The running server picks up the changes and applies them immediately.

Channel Layer: How OpenClaw Talks to Messaging Apps

OpenClaw connects to Telegram, WhatsApp, Slack.

Each of these works completely differently at the protocol level:

Telegram uses a bot token with polling or webhook delivery.
WhatsApp uses a reverse-engineered web protocol; it requires QR-code authentication and stores session state locally.
Slack uses Socket Mode with two separate token types.
Discord uses a gateway WebSocket with its own heartbeat requirements.

Without a dedicated abstraction layer, each new platform requires touching the routing logic, session management, and message pipeline.

OpenClaw solves this with a Channel Layer:

It treats each platform as a separate plugin acting as a translation adapter. Each adapter translates incoming messages into a standard internal format: a stable identity key, content, metadata, and attachments. Only then does it pass anything to the Gateway. Everything after that boundary is platform-agnostic. The agent has NO idea whether your message came from Telegram or Slack.

It sees a normalized message and responds accordingly…

This also means security policies like allowlists, pairing workflows, and mention-gating apply uniformly across all platforms rather than being reimplemented per channel.

LLM Layer: Where Reasoning Happens

OpenClaw is a brain socket.

You provide the intelligence by connecting an API key. And it provides the infrastructure for everything else.

It means you can use Claude, GPT-4, Gemini, DeepSeek, or a locally run model via Ollama. Plus, you can change models at any time without any reconfiguration.

When a message arrives at this layer, OpenClaw assembles a system prompt by combining the agent’s configured personality, list of available skills, relevant memory retrieved from previous sessions, and current conversation history.

It then calls the AI model with the full context.

If the model decides it needs to take an action, run a shell command, search the web, read a file, or call an API, it signals that as a tool call. OpenClaw executes the tool, captures the output, and feeds the result back into the conversation. The model reads the result and decides what to do next: call another tool, or produce a final answer.

This loop continues until the model generates a response with no further tool requests.

You can watch this entire loop unfold in real time in the Control UI at every tool call, every model response, every intermediate step…

And when a conversation reaches the model’s context limit, compaction12 triggers automatically. The agent summarizes the conversation so far and replaces the raw message history with that summary, preserving continuity without the cost of loading thousands of tokens on every reply.

Plugin System: How OpenClaw Stays Modular

Almost everything in OpenClaw that is not part of the core engine is implemented as a plugin13:

Telegram integration is a plugin.
GitHub access is a plugin.
The memory system is a plugin.

This is what makes OpenClaw easy to extend and customize…

When OpenClaw starts, it reads your configuration and loads only the plugins you have enabled. Disabling a capability means changing one line in your config, while adding a community plugin is as simple as placing it in the right directory.

If you want to write your own, you just implement a defined interface and never touch the core engine.

Plugins can hook into lifecycle events at every meaningful point: before and after tool calls, before and after messages, at session start and end.

This is how you add things like audit logging, rate limiting, or custom approval flows without touching the core agent at all…

Memory System: How Agent Remembers You

The memory solution (loading the entire conversation history into every prompt) fails quickly.

A week of active conversation will overflow any context window… Even if it fits,,, loading all of it on every reply is expensive and slow…

OpenClaw treats short-term continuity and long-term recall as separate problems that need different solutions:

Short-term memory keeps the current conversation on track. Every session is saved as a file on disk. When you send a message, OpenClaw loads only the recent part of that session into context, just enough to keep things flowing without loading everything at once.
Long-term memory handles recall across days and weeks. All past sessions are indexed into a local database. Before the agent answers any message, the memory indexer searches the database and pulls in any relevant information automatically. If a conversation from five days ago is relevant, it gets included. You never have to remind the agent of something it already knows.
Standing notes are a third layer. These are Markdown files you write yourself, stored in ~/.openclaw/workspace/memory/. Think of them as a permanent briefing document. Your preferences, your projects, your context. The agent reads them every time.

Reminder: this is a teaser of the subscriber-only newsletter series, exclusive to my golden members.

When you upgrade, you’ll get:

High-level architecture of real-world systems.
Deep dive into how popular real-world systems actually work.
How real-world systems handle scale, reliability, and performance.

Get Full Access Now

How OpenClaw Handles Tasks That Take Hours

Most interactions with OpenClaw are fast… You send a message; the agent calls a tool, and replies in seconds.

But some tasks take much longer: scraping hundreds of pages, running a full test suite, processing a large dataset, or coordinating a multi-step workflow across several APIs.

OpenClaw handles these differently:

When a task is going to take a while, the agent does NOT hold the conversation open and make you wait. It acknowledges the task, starts working, and messages you when it’s done. You can lock your phone and walk away.

The Gateway keeps running and messages you when the task is done.

For tasks that span multiple sessions or require human input mid-way, the agent completes what it can, writes the current state to MEMORY.md or a workspace file, and then pauses. When you return and pick up the conversation, it resumes from the saved state.

NO context gets lost between sessions because important state was written to disk, not held in memory.

For truly long-running background jobs, cron jobs in isolated session mode are the right tool. Each run gets its own session, completes its work, delivers output to your channel, and exits cleanly. The next scheduled run starts fresh.

This keeps long-running automation from accumulating stale context over time.

One practical limit worth knowing: if a single tool call takes too long, Gateway will time out waiting for a response.

For tasks like large web scrapes or slow API calls, break the work into smaller chunks. Instruct the agent to process in batches and report progress between each batch, rather than attempting everything in a single tool call.

Ready for the best part?

How a Message Travels End to End

Design Knowledge Q&A System

Louis-François Bouchard — Thu, 04 Jun 2026 11:13:56 GMT

Share this post & I'll send you some rewards for the referrals.

If you ask questions about your company’s policies, a contract you have open in another tab, or anything outside training data, an AI personal chat assistant will either say “I don’t know” or invent something that sounds right.

The most direct fix is to include the relevant documents in the prompt.

But this only works if the documents fit within the context window, and most real knowledge bases do not. Company policies, contracts, and internal wikis are lengthy, and loading them in full for every query can be impractical because of latency, cost, and context window limitations.

Retrieval-Augmented Generation (RAG) is a more practical approach.

The full RAG system: an offline pipeline that builds the index, and an online pipeline that answers each query, sharing a vector database.

Instead of loading the entire knowledge base into every prompt, the system retrieves only the most relevant passages at query time, includes them alongside the question, and the model generates a cited answer from that material.

It is the architecture behind Perplexity, ChatGPT with Search, GitHub Copilot’s codebase search, and most internal AI tools inside large companies.

Onward.

Code review for AI-native engineering teams — at 1M+ LOC (Partner)

Pull request review was built for human authors.

AI-written code fails differently, and most reviewer tools never caught up.

AgentField just open-sourced a multi-agent code reviewer built for AI-native teams:

Runs on open or closed models — Kimi, DeepSeek, Claude — so it scales to whatever a team can afford.
Drops into GitHub Actions in minutes.
Built to make deep code review economically viable at scale.

The architecture rationale — the four jobs of code review, and which three became load-bearing once AI took the first — is in AgentField’s latest post.

Star & Deploy

I want to reintroduce Louis-François Bouchard as the author of this newsletter.

Through his development work with clients and his content, teaching, and AI training programs on the Towards AI Academy, Louis focuses on making AI practical for builders, engineers, and curious learners alike.

At Towards AI, he and his team train AI engineers through courses built for every stage, from beginner to advanced. That educational mission and the real-world experience building for his clients are exactly why I wanted him in this newsletter series.

Here’s what’s inside this newsletter:

The full RAG architecture. How a question becomes an embedding, how the system finds the right chunks across a large knowledge base, and how the model generates an answer grounded in retrieved sources.
Document ingestion and chunking. What makes a chunk useful, three strategies for splitting documents, and the metadata that makes citations possible.
Retrieval and generation in production. Hybrid search, reranking, query transformation, system prompts, citation strategies, and how to make the system decline questions it cannot answer rather than guess.
Failure modes and evaluation. The three ways RAG breaks in practice, the four RAGAS metrics that surface them, and how to build a test set that catches regressions.
Production concerns. Caching, access control, monitoring, and cost at scale.
A practical eight-step build. A folder of PDFs turned into a working Q&A system with hybrid retrieval, reranking, structured citations, and a quality scorecard.

Why Build Your Own Knowledge Q&A System?

Perplexity, ChatGPT with browsing, and Gemini with Search already answer questions with cited sources.

They are capable and improving fast.

But the limitation is that they are built for public knowledge. The moment your questions involve internal documents, regulated data, or a knowledge base within your infrastructure, off-the-shelf tools hit hard constraints.

Here is where they fall short:

Off-the-shelf tools cannot access regulated data. You can upload personal files into a ChatGPT project and get useful answers. You cannot do that with customer records, contracts, medical files, or anything covered by HIPAA, GDPR, or SOC 2. That data has to stay inside your own perimeter, behind your own access controls, with your own audit trail.
Your knowledge is not in a folder you can drag and drop. It lives in Confluence, in a Drive folder with thousands of contracts, in a support tool, or in an internal wiki. Off-the-shelf assistants can connect to some of these, but the data flows out of your infrastructure and into theirs, and is processed under their privacy policy. A system built on your infrastructure reads directly from those sources, using your auth, and ensures data never leaves your perimeter.
Retrieval happens at query time, so answers stay current. A model’s training data has a cutoff, but your documents keep changing. Because RAG retrieves at query time, a document updated after the model was trained is still used to answer the next question, with no retraining required.
You control what the system knows. When a document is added to the knowledge base, the system uses it for the next query. When it is removed, it is no longer used. When it is updated, the next answer reflects the new version. This control is what lets you curate, version, and audit the knowledge base, and it is what regulated industries require.
Focused retrieval is more accurate and cheaper. A model answering a question from five selected passages is working with a much smaller, more relevant input than one drawing on its full training data. More relevant context produces more accurate answers, and shorter prompts cost less per call.

These five constraints all point to the same architecture.

The rest of this newsletter builds on it…

Part 1: Turning Documents into Searchable Knowledge

A RAG system has two stages:

First runs once, or whenever new documents are added: it processes your documents, converts them into vector representations, and stores them in a database that can be searched by meaning.
Second runs on every user query. This part focuses on the first: ingestion, chunking, embedding, and storage.

Document Ingestion and Preprocessing

Before anything can be retrieved, your documents need to be readable by the system.

That sounds straightforward, but documents come in many formats: PDFs, HTML pages, Word files, Markdown exports from Notion, Confluence pages, and scanned forms. Each format stores text differently and needs its own parser.

A poor extraction at this stage degrades everything downstream, so it is worth getting right.

There are five things to handle during ingestion:

Text extraction. As text is extracted, the structural information alongside it is preserved: headings, page numbers, and table boundaries. This is what allows the system to later cite not just a document, but the specific section and page a passage came from.
Tables, code, and images. These elements break when treated as regular text. A table flattened into a paragraph loses its row-column relationships. Code blocks lose meaning when indentation and line breaks are stripped. Images contain no text at all unless a caption is present or OCR is run on them. Each is typically handled through a separate extraction path.
Scanned PDFs. A digital PDF stores text as text. A scanned PDF stores it as an image, which a regular parser cannot read. Optical Character Recognition (OCR) is used for converting scanned documents into readable text. But OCR takes several seconds per page, which adds time to the ingestion stage. Tesseract is the standard library for clean scans. Vision-language models work better on low-quality images and complex layouts.
Duplicates. When the same document exists in multiple folders, it can be indexed multiple times, which causes retrieval to return identical chunks. A common approach is to hash each document’s content and skip any document whose hash has already been indexed. For near-duplicates such as different versions of the same contract, fuzzy text matching is used instead.
Boilerplate. Headers, footers, navigation menus, and copyright notices add noise without adding information. These are typically identified by how frequently they repeat across pages and stripped before the text is passed downstream.

Clean text is necessary but not sufficient.

Before it can be embedded and searched, it has to be cut into pieces small enough for the embedding model to represent accurately.

Chunking

A long document cannot be embedded as a single vector.

An embedding that represents 50 pages of mixed content does not align well with a specific question. Instead, each document is split into smaller pieces called chunks, and each chunk is embedded individually.

Different chunk sizes have different tradeoffs…

Larger chunks carry more context but produce less specific embeddings. Smaller chunks produce more specific embeddings but may lack the surrounding context needed to fully answer a question. A well-formed chunk is self-contained, covers a single topic, and includes its section heading and document title.

Here are three common strategies for splitting documents into chunks:

Fixed-size chunking splits the text into chunks of N tokens, with a fixed overlap, usually 50 to 100 tokens. The overlap ensures that a sentence cut at a chunk boundary still appears in full in one of the two adjacent chunks. It is the simplest approach and works well on uniform text such as articles and documentation.
Recursive chunking splits on paragraph boundaries first, then sentences, then words, going to a finer level only when a chunk is still too large. It follows the structure already present in the document rather than imposing an arbitrary boundary. Most popular RAG libraries, including LangChain and LlamaIndex, use this as their default.
Semantic chunking splits a document into sentences, then forms small groups of consecutive sentences and creates an embedding for each group. The embedding of each group is compared with that of the previous group, and a chunk boundary is placed where the difference between the two embeddings is large. The assumption is that a large change in the embedding indicates a change in topic. Semantic chunking performs better than the other methods when topics shift within paragraphs rather than across them, but it requires an embedding call for each sentence group at ingestion time, making it the most expensive of the three.

A common starting point is recursive chunking at 500 to 800 tokens with a 50- to 100-token overlap.

Each chunk is tagged with metadata: source document, page number, section heading, and last-modified date. This metadata is what the system uses to produce citations later.

The chunks are now ready to be searched. Searching by exact word match would miss most relevant results, since users rarely use the same words that documents do.

Each chunk is therefore converted into a numeric vector that captures its meaning…

Embeddings

These vectors are called embeddings: fixed-length vectors of numbers that represent the meaning of a piece of text.

Two chunks with similar meaning produce vectors that are close to each other; two chunks on different topics produce vectors that are far apart.

This is what allows a query like “reset my password” to retrieve a document titled “account credential recovery,” even though the two share no words in common.

Some commonly used embedding models in 2026 include OpenAI’s text-embedding-3-large, Cohere’s embed-v4, and open-source options such as bge-large and e5-mistral.

The typical starting point is an API model, with a move to self-hosted open-source if data residency, privacy, or cost becomes a constraint. For specialized domains such as legal, biomedical, or financial text, a domain-specific model generally outperforms a general-purpose one.

The MTEB benchmark is a useful reference for comparing models.

Embeddings have three limitations worth understanding before choosing a retrieval strategy:

Negation. “Drugs that don’t interact with warfarin” and “drugs that interact with warfarin” produce nearly identical embeddings. The word “don’t” has very little effect on the resulting vector, even though it inverts the meaning.
Exact identifiers. Order numbers, error codes, and SKUs such as E-4012 carry no semantic meaning, so similarity searches cannot reliably match them. Hybrid search (covered in Part 2) is used in these cases.
Numeric filters. A query like “customers with revenue above $5M” is a structured filter, not a similarity match. Structured retrieval (covered in Part 2) is used for these queries.

Another operational constraint is that the model used to embed the chunks must also be used to embed queries during retrieval.

The two share a single vector space, and switching models means re-embedding the entire corpus.

The choice of embedding model is effectively a long-term commitment…

Vector Databases

Once the chunks are embedded, they need to be stored in a system that supports fast similarity search.

A vector database stores each chunk’s embedding alongside its metadata and returns the closest matches to a query vector. The metadata, source document, page number, section heading, and last-modified date are returned with each result, so the system can point the user back to the source.

The basic operation is similarity search: given a query vector, return the K vectors in the database that are closest to it.

Closeness is usually measured using cosine similarity, which compares two vectors based on the angle between them.

A direct implementation of this compares the query vector against every vector in the database. This works for small databases of a few thousand vectors, but the cost grows linearly with the database size and becomes impractical at scales of millions of vectors.

Production systems use approximate nearest neighbor (ANN) search, a family of algorithms that returns close-to-exact results at a fraction of the cost. ANN trades a small amount of recall for a large reduction in latency, often by orders of magnitude, with the exact tradeoff depending on the dataset, the index, and the recall target.

Hierarchical Navigable Small World (HNSW) and Inverted File (IVF) are two of the most commonly used ANN index types. Some managed vector databases abstract this choice entirely, while others let you pick the index and tune its parameters.

Here are three deployment options, each suited to different constraints:

Managed services such as Pinecone, Turbopuffer, and Weaviate Cloud. The service runs the infrastructure and exposes an API. This is appropriate when the team does not want to operate the database directly.
Self-hosted open-source options such as Weaviate, Qdrant, and Milvus. These run on your own infrastructure and offer full control over configuration, scaling, and metadata filtering.
Database extensions such as pgvector for Postgres. These add vector search to an existing relational database. A practical option when the stack already runs Postgres, and the corpus stays under approximately one million chunks.

The vector database stores each chunk’s vector alongside its metadata. At query time, the same embedding model produces the query vector, and similarity search returns the closest chunks with their source attached.

With the index built and the chunks stored, the offline pipeline is complete. The next question is how the system uses it when a user actually asks a question.

Reminder: this is a teaser of the subscriber-only newsletter series, exclusive to my golden members.

When you upgrade, you’ll get:

Simple breakdown of real-world architectures
Frameworks you can plug into your work or business
Proven systems behind ChatGPT, Perplexity, and Copilot

Part 2: Finding the Right Information

This part covers how the system returns the chunks that contain the answer…

Basic Retrieval

The default retrieval approach has three steps: embed the question using the same model used to embed the chunks, run a similarity search against the vector database, and return the top K results.

K is a tradeoff parameter.

Too small, and the system misses chunks that contain the answer. Too large, and the prompt fills with irrelevant material, which reduces answer quality and increases cost. A reasonable starting value is between 5 and 10.

The right value for your system is determined over time by testing on real queries. It depends on the expected answer length and how much room the LLM’s context window leaves for the retrieved chunks.

Basic retrieval works well on direct questions over a uniform corpus, but it has three common failure modes:

Vocabulary gap. User uses one word, and the document uses another. For example, the user types “downtime” and the document uses “service interruption.” Acronyms, internal jargon, and exact identifiers often fall outside the scope of what a general-purpose embedding model has been trained on.
Multi-hop questions. Answer is spread across many documents that are not semantically similar. For example, the question “what did the CFO say about the product launch?” requires information from both an earnings transcript and a product roadmap. The two documents do not embed near each other in vector space, so similarity search alone is unlikely to retrieve both.
Ambiguous queries. Query has more than one valid interpretation. For example, “tell me about Apple” could refer to the fruit or the company. Without disambiguation, similarity search returns results for one interpretation without indicating that another was possible.

The vocabulary gap and the exact-identifier problem both come from the same source: similarity search alone cannot match words it does not understand semantically. Hybrid search adds a second retrieval path that can.

The third failure mode, ambiguous queries, is handled by a different technique covered later in this newsletter…

Let’s keep going!

Hybrid Search

How to get ahead of 99% of software engineers with AI agents

Neo Kim — Thu, 28 May 2026 09:59:35 GMT

Share this post & I'll send you some rewards for the referrals.

You hit an error you can’t solve…

So you paste it into Claude Code, and the agent tries a few fixes, but NONE of them work. The bug is affecting checkout, which makes it a money problem, so you drop it in Slack and explain the whole thing again to the senior responsible for that flow.

She asks good questions… You answer them…

Then someone says, “Make a ticket so we don’t lose this,” and you type the same context into Jira a third time. Same problem, three tools, and you wrote it out fresh for everyone.

But none of them remembered what you’d already said in the last…

The bug happened the way most bugs appear these days: you just told a coding agent the feature you had in mind. This is how the job works now,,, and most of us don’t know where agents fit into the traditional software development lifecycle…

So let’s learn about it in this newsletter…

We start with what an agent actually is, because the word gets stretched over everything from autocomplete to a bot that opens its own pull requests1. Then, we’ll cover eight stages every team moves software through, from planning a feature to keeping it alive in production.

For each stage, we answer one question: can you hand this to an agent today, and what happens when you do?

In some stages, agents already carry real weight… In others, they make things worse…

By the end of this newsletter, you’ll know which is which, and the harder problem hiding underneath all of it.

Onward.

I’m happy to partner with CodeRabbit on this newsletter.

One thing I believe, after researching this newsletter, is that the biggest problem with AI agents in software development is NOT code generation. It’s the context loss between tools, tickets, reviews, and Slack threads.

CodeRabbit is building around exactly this problem…

Their new Slack agent maintains shared context across the software development lifecycle, so agents don’t have to start from scratch every time work moves between coding, testing, incidents, and reviews.

They already run AI code reviews across 6M repositories for 15,000+ teams.

And I genuinely think the idea of shared memory across the SDLC is one of the more important directions in agentic engineering right now.

What an AI agent is

An agent takes a goal, picks its own next step, uses tools2, and keeps going until the job is done or it gets stuck.

That’s the whole idea…

When you hit the checkout bug and pasted it into Claude Code3, you didn’t tell it which files to open or what to run. Claude Code worked out the rest on its own:

Read files,
Grep4 for functions,
Run tests,
Read errors,
Try again.

…and you watched.

“Agent” gets used loosely, so it helps to line up the tools by how much they do on their own.

The simplest is AI AUTOCOMPLETE, like the earlier versions of GitHub Copilot5. It finishes the line you’re typing with a Tab.

A step up is a CHATBOT: you ask, it answers, and you copy what’s useful back into your editor. That’s your ChatGPT.

An AGENT is the one who does the work… You give it a bug, and it reads files, runs tests, edits code until the job is done.

To do this, it has to keep track of what it has already tried and what it learned along the way. That running context is what lets it pick a sensible next step instead of starting over each time.

One agent manages its own context fine…

But it becomes difficult when you have to run several agents at once, and that difficulty is largely what makes agents hard to use well.

Software gets built in eight stages, and every team moves through them with or without agents:

Plan: decide what to build,
Design: decide how it’s structured,
Code: write it,
Test: prove it does what it should,
Review: check the work before it ships,
Deploy: ship it to production,
Operate: keep it running and watch it,
Maintain: fix, update, and clean up over time.

In theory, you can put an agent to work in any of these stages. Yet it behaves differently in each…

Get 1+ referral & I’ll send you my Leetcode Master Template!

Software lifecycle: where agents fit

The same agent that looks brilliant in a demo can be useless on your actual codebase.

What changes between the two is how much it costs you when the agent is wrong, and this comes down to 4 things:

How fast you find out. A failing test tells you in seconds. While a bad design decision can go unnoticed for months.
How easily you can check the answer. You can read a git diff and judge it. “Is this the right thing to build” has no such test.
How much breaks. A wrong function fails one test. A wrong deployment takes down system for everyone.
Whether you can undo it. You can revert a commit. But you can’t uncorrupt the data a bad database migration6 already wrote.

A demo is set up to do well on all four…Small task, instant feedback, nothing real at stake… Yet your work usually isn’t.

So, for each stage below, the real question is the same: when the agent gets it wrong, how much does that cost you?

1. Plan

Planning is deciding what your team builds next.

Which feature out of the ten on the list, why that one, and when it ships. It happens in meetings, in docs, in back-and-forth between engineers and the people who run the product…

An agent does useful work here.

Give it a feature spec, and it reads your codebase, maps what the change will touch, flags the parts nobody pinned down, and writes the tickets faster than you would by hand. Hand it the boring write-ups, and it does them well.

But decision-making is something an agent CAN’T do:

Choosing what to build means weighing customer urgency, revenue, and risk against three people who each want something different, then standing behind the choice when it goes wrong. An agent has no stake in the outcome and doesn’t understand your business well enough to make that call.

The people with the most reason to claim otherwise say the same thing.

Boris Cherny, who created Claude Code, was asked why Anthropic keeps hiring engineers when the company says Claude now writes most of its code. His answer:
“Someone has to prompt the Claudes, talk to customers, coordinate with other teams, and decide what to build next. Engineering is changing, and great engineers are more important than ever.”

i.e., company shipping the most capable coding agent on the market still needs actual humans for planning.

2. Design

Plan is which problem you solve…

Design is the shape of the solution: one service or three, which database, where the boundaries sit, and what you can change later without a rewrite.

Ask an agent, and you will get a real answer.

It will compare microservices against a monolith7, lay out the tradeoffs, and recommend one. Yet the answer is textbook, and this is the problem. A textbook answer doesn’t know your team is five people, your traffic triples every December, or the last person who touched the payments service quit and left no documentation. So the right design for your situation lives in facts that were never written down, and an agent can’t read what nobody wrote.

There’s a second reason this stage stays with humans…

A bad design is slow to show itself. The code passes review, and tests go green, so nothing indicates the structure is wrong. But it’s only until you try to build the next three features on top of it, and each one takes twice as long and breaks something that used to work.

By the time you see the pattern, fix is no longer a code change. It’s a complete rewrite…

The root of it is agent works on the task in front of it without a picture of the entire system or any reason to keep the system simple. This blindness shows up in two ways in design.

The first is duplication:

GitClear analyzed code changes across thousands of repositories, found a sharp rise in duplicated code after teams adopted AI coding tools. In 2024 alone, the number of code blocks with five or more duplicated lines increased 8x.

The agent writes a new block that does the same job as one already sitting three files over, because it never checked what the codebase already had. Every copy is one more place you have to remember to fix later, and this is a structural cost.

And second is the opposite problem: over-engineering.

Ask an agent to design something, and it tends to overdesign.

It adds layers, patterns, and configuration for cases that will never come up because more structure looks more thorough.

Andrej Karpathy, who co-founded OpenAI, put it plainly: models like to overcomplicate, bloat their abstractions, and leave dead code behind.

This is the textbook answer showing up again…

The agent designs for the general problem because it doesn’t have the full real-world picture. You will inherit a system harder to read and change than the thing you asked for.

3. Code

This is the stage where agents are doing their best work…

At least the MONEY says so. Cursor went from $100M to $1B in annual revenue in about a year and has more than a million people using it every day. In Stack Overflow’s 2025 survey, 19% of professional developers said they use Cursor, and 10% use Claude Code, two tools that weren’t even on the survey the year before. These tools write real code that ships, and the people paying for them are not running demos.

What you get out of them depends almost entirely on how you set them up…

Rules files

The first thing to set up is a rules file.

Most coding tools now read a file in your repo when they start, a CLAUDE.md8 or an AGENTS.md, and treat whatever is in it as standing orders. You write down the things you would otherwise have to repeat in every chat:

Commands to run tests and linter9,
Conventions the codebase follows,
Traps you learned the hard way and want the agent to avoid.

Without this file, every session starts from zero…

You explain the same setup, correct the same wrong assumption, and watch the agent make the same mistake it made yesterday, just because it has NO memory of yesterday.

Writing correct rule files is already somewhat of a science in coding, and there are many excellent sources that can help you.

(I recommend starting with the official Anthropic guide on CLAUDE.md files.)

Spec-driven development

The second thing is to match the size of the task to how much you plan first…

A small, clear job gets done well from a one-line prompt. “Add input validation to this endpoint” needs nothing fancy. While a large task like “add photo sharing to my app” falls apart because that prompt forces the agent to guess at thousands of decisions you never specified.

So for anything real, you write a short spec first…

A spec documents non-technical product decisions, high-level architecture, and what the changes should do in plain language before any code gets written. The agent turns it into an implementation plan, noting exactly which files and lines of code will change.

Then it executes the code one task at a time, and you review between every step…

This workflow is called spec-driven development10, and it is how people ship real features with agents instead of demos.

PROOF: GitHub repo Superpowers (200k+ stars) works with all major coding agents. It encapsulates the process into a series of automatic slash commands, making it trivial to adopt in your daily workflow.

But spec-driven development isn’t new,,, it is an old engineering tradition.

The only thing changed is how fast you move from a vision in your head to a written spec to a working implementation. All with the help of coding agents…

Current limitations

There are many, but the first one you’ll hit is the context window11…

An agent holds everything it’s working on in a fixed amount of memory, measured in the number of tokens12.

The longer a single task runs, the more memory it fills with its own earlier steps, and at some point, the oldest facts drop out of the window. It forgets a decision it made twenty minutes ago and starts contradicting itself.

This is why the spec-and-plan loop matters beyond just planning…

The spec and plan documents can hold massive context on disk agent can refer to at any time. This frees up its window to focus only on the task at hand. It doesn’t matter whether your agent has a 1M-token window.

Quality drops as the context fills, well before the window is full.

Anthropic documents this for its own models, and a study of 18 models found every one got worse as the input grew. The decline is steady, and a model with a 1M-token window can already be slipping at 50k.

Get 2+ referrals & I’ll send you my Interview Mistakes to Avoid PDF!

4. Test

Testing is where you find out whether the code does what you said it should, before anyone else depends on it.

This is the stage agents are ‘built’ for…

A test runs in a few seconds and tells the agent one thing: pass or fail. So the agent can write a test, run it, see what broke, and try again, as many times as it needs, without you in the room.

The products doing this are real and busy…

Momentic, a tool that tests apps by clicking through them the way a user would, ran over 200 million test steps in a single month and caught 390,000+ bugs.
Diffblue Cover, which writes tests for large-scale Java systems at banks like Goldman Sachs and JPMorgan, writes them 250 times faster than a person and adds enough tests to raise test coverage by 50 to 70%.

This does not mean you can hand testing to an agent and walk away… The failures here are easy to miss, and there are three of them:

The first one affects everybody.

Ask an agent to write tests, and a lot of them pass without checking anything real.

The test runs the function, and function returns an answer. The check is technically true, and still nothing useful got tested. Agents write these empty tests all the time.

This is why, when Claude Code writes a plan for me, I almost always send it back with one note: make the tests mean something, not just pass. The agent then usually throws out its first set and writes much better ones.

The second failure is a direct follow-up to the first.

The agent writes so many tests that keeping them all working becomes a job in itself.

Large corporations like Meta are already running into this. Their testing setup, built up over decades, could not keep up with the number of tests the agents were writing, so they moved to tests that are made fresh for each pull request and deleted as soon as they run.

The third one is the hardest to fix.

Agents are slow and expensive at testing anything you can see on screen…

A logic test is quick because the agent reads pass or fail straight from the code. A test of a button or a page gives no such answer, so for every step, the agent has to:

Take a screenshot,
Work out where the button sits on screen,
Click it,
Take another screenshot to check what changed.

A person clicks a button in a fraction of a second... The agent spends time and tokens on all four steps to do the same thing…

Did you know?

CodeRabbit’s new Slack agent keeps shared context across coding, reviews, incidents, and tickets, so your agents stop starting from zero every time work moves between tools.

Use it to carry decisions, debugging history, and architectural context across your entire software development lifecycle:

5. Review

Before any code ships, someone has to read it and sign off on it.

For years, someone was always a person. You open a pull request, and one or more teammates read your changes line by line. They look for bugs, for code that does not match the rest of the project, and for anything that does not do what the request says it does.

Nothing merges until one of them approves it…

But this process is slow:

Reviews pile up in a queue while people are busy with their own work. One senior engineer often ends up reviewing everyone else’s code, becoming the bottleneck. And a reviewer reading a 600-line change late on a Friday is NOT going to catch much.

This is the part agents handle well, and the WIN is speed and coverage.

An agent reviews every pull request the moment it opens. It never sits in a queue, never has a bad day, and never skips the boring change nobody wanted to read. This speed and coverage matter. CodeRabbit, one of the biggest AI review tools, now runs on more than 6 million repositories for over 15,000 teams.

So every change gets a look. This is the real value,,, and it is worth a lot.

What an agent can’t do is decide which of its own comments matter.

It flags a lot, and only some of it is worth your time. A 2025 study of 278,790 real review comments found developers acted on 16.6% of suggestions from AI reviewers, compared with 56.5% from human reviewers.

i.e., people accept a human reviewer’s note 3x as often.

The danger is what it does to your own attention…

When most of the comments are not worth acting on, you stop reading them closely. You skim, you click approve, and the one comment in twenty that caught a real bug slides past with the rest. The agent reviewed every line, and you still missed the bug, because it buried the signal in noise.

So treat the agent as a first pass that never misses a file, NOT as the reviewer who gets the final say…

6. Deploy

This is the first stage where the answer is a clear NO. Don’t hand it to an agent…

Two things make deploy different from everything before it:

First, you can undo a bad deploy, but not in the clean way you undo a bad commit. You can roll back, sure. By then, the broken version already reached real users, money may have moved, and the rollback is its own risky operation; you’re running under pressure.
Second, it breaks everything at once. A bad function fails one test. A bad deploy takes down the entire system for every user at once.

So the deploy button is one of the few places where the SAFE move is still a human hand…

Let the agent prepare the release, run the checks, and stage everything it can. Keep a person in front of the final push to production, where a wrong call costs the most and undoes the least.

7. Operate

Once the code is live and real users are on it, someone has to keep it healthy…

You watch the dashboards for trouble. When something breaks, you find out why and fix it fast. A lot of these land at bad hours, because production doesn’t break on a schedule. You’re on call, and your phone can buzz at 2 a.m.

There is more of this work now than there used to be…

All the AI-written code from the earlier stages ships, and it breaks more often as measured by two separate studies:

Google’s DORA13 report tracks delivery stability across thousands of teams and found greater AI use correlates with lower stability in both its 2024 and 2025 editions.
Faros AI analyzed telemetry14 from 22,000 developers and found the number of production incidents per merged change increased by 242% as AI use climbed. More code ships, more of it breaks, and someone is on call for all of it…

Ironically, agents are really good at analyzing incidents caused by bad agent code.

The first part of any incident is gathering. You pull the recent deploys, search the logs, check the dashboards, and walk through the runbook15 step by step. It’s slow, and it’s almost the same process every time.

An agent does it fast…

Datadog’s Bits AI SRE16 investigates an alert the moment it fires and has the findings ready before you’ve opened your laptop.
One of its customers said: investigation is done before the engineer sits down. At 2 a.m., this head start is worth a lot.

What you can’t hand over is the decision on what’s actually wrong…

The agent will tell you the cause, and it says it with the same confidence whether it’s right or not. An alert fires; the agent says it’s the database connection pool. It sounds right, so you spend 30 minutes there. But the actual cause can be completely different. A human might have said “I’m not sure yet” and kept looking. The agent’s speed means you commit to the wrong fix sooner.

So let the agent gather the facts and keep the diagnosis YOURS…

Get 3+ referrals & I’ll send you Popular Interview Questions PDF!

8. Maintain

After a feature ships, the work doesn’t stop…The code still needs “babysitting”.

A library releases a security patch, so you bump to the new version. A messy function needs cleaning up; a small bug turns up; docs stopped matching the code three releases ago. None of it is glamorous, and all of it piles up. Ignore it long enough and the codebase rots.

Of all eight stages, this is where agents earn the clearest yes…

The reason is maintenance work grades itself. A dependency bump either passes the tests or it doesn’t. A refactor is meant to leave the behavior the same, so if the tests still pass after the agent rewrites the code, the refactor worked. The test suite is the judge, and it’s one the agent can run on its own, as many times as it needs, until the change is green.

This is the exact condition agents are built for…

The products doing this run at a scale no other tool mentioned today matches.

Dependabot (GitHub’s tool) watches your dependencies and opens a pull request when one falls behind, and is used across millions of repositories. GitHub says repositories using automated security updates fix critical vulnerabilities significantly faster because the update is waiting for you instead of sitting on a to-do list that nobody gets to.

But there is still a subtle way this can go WRONG:

Each change is small and passes its tests, so each one clears review without much thought. The agent doesn’t see the entire system, so it ships the refactor that works while leaving the code a little worse than a careful person would.

A bit more duplication here, an odd structure there…

No single change is worth stopping for. Over time, they stack up into a codebase, making it harder to read and change than before, and nobody can point to the commit where it went wrong. So this is the same blindness from the ‘Design’ stage, showing up slowly instead of all at once.

Let the agent handle the maintenance, and keep a human to read what it actually changed…

Across the lifecycle

One pattern runs through all eight stages:

Agents do well wherever a machine can tell them they’re wrong. A failing test, a red build, a type error: agent gets an answer in seconds and tries again on its own until the work is green. This includes code, tests, dependency bumps, first pass at an incident.

But they struggle wherever the only judge is a person…

Whether this is right feature, right architecture, or right moment to ship to production. No test comes back red on those. And the answer depends on what your business needs and how much it costs you when you’re wrong, and none of it is written down for the agent to check.

These are also the stages where a mistake is slow to find and hard to undo, so the agent runs furthest before anyone catches it.

Every verdict here looked at one agent doing one job, though. Real teams run a dozen at once, and it changes the problem…

Why more agents do NOT make your team faster

So far, we have looked at one agent doing one job. Real teams run a dozen at once…

A coding agent, a test agent, a review agent, and an on-call agent, all working at the same time. The trouble is none of them shares memory with the others. So the coding agent doesn’t know what the test agent already tried. And the review agent doesn’t know what the on-call agent learned last night.

Each one starts blank about every other one.

That gap costs you more than you’d expect…

Nicholas Carlini at Anthropic ran 16 Claude agents in parallel to build a C compiler. Over 2,000 sessions and about $20,000 of compute, and the project worked. But here is what he found running them together:
“Every agent would hit the same bug, fix that bug, and then overwrite each other’s changes. Having 16 agents running didn’t help because each was stuck solving the same task.”

So the agents duplicate and overwrite each other, and a person has to review it all and resolve the collisions before any of it merges.

The context that would stop the repetitive work already exists… It is just the agent can’t usually read it…

It’s in a Slack thread, in the reasoning on a closed pull request, in an old ticket, or in what the on-call agent worked out last night. The next agent can read your code, but it can’t read the conversation where you decided why the code looks a specific way.

This decision history is the part NO agent has…

The fix is to give the agents one shared memory which survives between runs, running where the team already talks, so a person can interrupt and redirect.

CodeRabbit launched a Slack agent that works this way, and the way they describe the problem is the subtle patterns we have been hinting at throughout the newsletter: “Each phase runs on a different tool and uses a different agent. None of them talks to each other. What one engineer figures out in coding doesn’t show up in testing.”

Their agent runs in Slack, so it reads the same threads your team does and carries what it learns from one conversation to the next. Ask it Friday about something it saw Monday, and it still knows…

Shared memory is the real problem under all eight stages, and nobody has fully solved it yet…

What to expect when an agent is on your team

Say you decide to put agents to work anyway, before that problem is solved.

Here is what the job actually feels like once you do it:

The first thing to know is the speed you feel may NOT be the speed you get…

METR, an independent nonprofit running studies on what AI can and can’t do, tested this in 2025. (They aren’t selling an AI tool, so they had no reason to tilt the result either way.)

They took 16 experienced developers and gave them 246 real tasks on their own projects, the repositories they had maintained for years. The developers did half the tasks with AI tools and half without, and a timer recorded both.

They finished the AI tasks 19% slower. The surprising part is same developers thought the AI had made them 20% faster. They were slower and felt quicker at the same time.

This gap is what matters here…

These were experts on code they already knew well, where they were fast to begin with, so it won’t match every job. But your own sense of whether an agent is helping is not reliable, so check it against something real.

With more agentic work, your week changes a lot…

An agent opens more pull requests than you would, and each one is bigger than the last. So you spend less of your day writing code and more of it reading what the agent wrote and deciding if it’s right. That reading has a different cognitive cost. Do it all day, and you end up tired in a way that writing the code yourself didn’t make you, even when the agent saved time.

The other thing to set up before you “trust” an agent is a limit on what it can “break”…

An agent running commands can do real damage when it gets something wrong:

In July 2025, Replit’s coding agent deleted a production database after it was told, in plain words, to change nothing. It wiped records for 1,200+ business leaders and 1,000+ companies. The agent’s own summary afterward read: “I destroyed months of work in seconds.”

Three controls keep that from being you…

Run the agent in a sandbox so it can’t reach production in the first place.
Give it scoped permissions, so it only gets access to the one thing the task needs and nothing else.
And keep an audit log of every command it ran, so when something breaks, you can see exactly what it did.

So pick your first agent where a wrong answer is cheap to undo…

Coding, review, and maintenance are good places to start, because a bad suggestion gets caught before it ships, and a bad refactor fails the tests. Design and deploy are the worst places to start, for every reason this newsletter has already covered.

Begin where the agent can be wrong without costing you much, learn how it behaves, and only then move it somewhere that matters more.

Your engineers talk on Slack. They code in the terminal. Somewhere between those two things, context dies…

A bug was debated in #incidents at 2 AM.
An architectural call was made in a DM.

Every handoff leaks context, and every leak costs you. That’s the context tax and your team pays it every day.

CodeRabbit Agent for Slack is built for agentic SDLC workflows. One agent for your entire Software Development Lifecycle, living in the channel where the work already happens. It’s built on four things:

Context: your org’s operating picture, pulled from across code, tickets, docs, monitoring, and cloud.
Knowledge Base: a living memory of your team. Every run leaves a trace, so yesterday’s decisions don’t become tomorrow’s debates.
Multi-Player: works in shared threads alongside your team. Steerable, resumable, and aligned as work evolves.
Governance: scoped access, cost attribution. Every run explainable and attributed.

Your team keeps shipping. Agent keeps the context.

From the team that pioneered AI code reviews. 2M code reviews every week. 6M repos. 15K customers. And now, one agent for your entire SDLC, right in Slack:

(Thanks again to CodeRabbit for partnering on this post.)

Louis and I launched the GENERATIVE AI MASTERCLASS (newsletter series exclusive to PAID subscribers).

When you upgrade, you’ll get:

Simple breakdown of real-world architectures
Frameworks you can plug into your work or business
Proven systems behind ChatGPT, Perplexity, and Copilot

👉 CLICK HERE TO JOIN THE GENERATIVE AI MASTERCLASS

(Golden members will get the next Generative AI newsletter in the first week of June.)

If you find this newsletter valuable, share it with a friend, and subscribe if you haven’t already. There are group discounts, gift options, and referral rewards available.

Watch the AWS Virtual Panel

👋 Find me on LinkedIn | Twitter | Threads | Instagram

Want to reach 210K+ tech professionals at scale? 📰

If your company wants to reach 210K+ tech professionals, advertise with me.

Thank you for supporting this newsletter.

You are now 210,001+ readers strong, very close to 210k. Let’s try to get 211k readers by 29 May. Consider sharing this post with your friends and get rewards.

Y’all are the best.

Boris Cherny on why Anthropic keeps hiring engineers: x.com/bcherny (mirror)
Duplicated code rose 8x after AI adoption: GitClear, AI Copilot Code Quality 2025
Andrej Karpathy on models bloating abstractions and leaving dead code: x.com/karpathy
Cursor revenue and daily users: Cursor Series D announcement
Cursor and Claude Code developer adoption: Stack Overflow 2025 Developer Survey
Context quality drops as input grows, across 18 models: Chroma, Context Rot
Anthropic on CLAUDE.md memory files: Claude Code memory docs
Spec-driven development with Superpowers: github.com/obra/superpowers
Anthropic on context windows: Claude context windows docs
Momentic, 200 million test steps and 390,000 bugs in a month: Momentic Series A
Diffblue Cover, 250x faster test writing: Diffblue Cover
Diffblue Cover, 50 to 70% coverage lift: Uplift Java test coverage
Diffblue Cover at Goldman Sachs: Goldman Sachs case study
Meta’s just-in-time tests, generated per pull request: The death of traditional testing
GitHub Copilot Code Review, 60 million reviews and 1 in 5 of all reviews: 60 million Copilot code reviews and counting
Developers act on 16.6% of AI review comments vs 56.5% from humans (278,790 comments): Human-AI Synergy in Agentic Code Review
Amazon Kiro and the 13-hour AWS outage: Amazon’s response (originally reported by the Financial Times)
DORA, AI adoption and delivery stability: DORA 2024 report, DORA 2025 report
Faros AI, incidents per change up 242% across 22,000 developers: The AI Acceleration Whiplash
Datadog Bits AI SRE, generally available December 2025: Datadog launches Bits AI SRE
Dependabot, 2.7 million repositories and 30% faster fixes: GitHub Octoverse 2025
Nicholas Carlini, 16 agents building a C compiler: Building a C compiler with Claude
CodeRabbit’s Slack agent launch and framing: CodeRabbit launches Slack agent
Mem0, open-source memory layer for agents: github.com/mem0ai/mem0
METR, developers 19% slower with AI while feeling faster: METR study (paper)
Replit’s agent deleting a production database: Fortune

Pull request — a proposed change to a codebase, opened so others can review it before it is merged into the main code.

Tool is something the agent can use to perform tasks outside the chat, such as reading files, running tests, searching code, or calling an API.

Agent is an AI tool that can work through a task on its own instead of waiting for instructions at every step. For example, Claude Code can take a bug report, search the codebase, edit files, run tests, and keep trying fixes until it solves the problem or gets stuck.

Grep — a command that searches files for a piece of text. When an agent “greps for a function,” it searches the codebase to find where that function lives.

GitHub Copilot is an AI coding assistant built into your editor. It suggests code as you type (from single lines to entire functions) based on the code around it and the comment you wrote.

Database migration — a change to the shape of your database, such as adding a column or renaming a table. A bad one can corrupt existing data in a way you cannot undo.

Monolith vs microservices — two ways to structure a system. A monolith is one single program that does everything; microservices split the same work across several smaller programs that talk to each other.

CLAUDE.md / AGENTS.md — a plain-text file you keep in your repository. The coding agent reads it on startup and treats its contents as standing instructions.

Linter — a tool that automatically checks your code for style problems and likely mistakes before it runs.

Spec-driven development — writing a clear specification first, then having the agent turn it into a plan and an implementation, instead of prompting it one ad-hoc request at a time.

Context window is the amount of information an AI can keep in mind at one time while it works.

Token is a small piece of text that an AI reads at a time, usually a word, part of a word, or punctuation.

DORA stands for DevOps Research and Assessment: a long-running research program that studies how software teams build, ship, and operate software. It’s best known for measuring things like deployment speed, reliability, and production stability across thousands of engineering teams.

Telemetry — the performance and usage data that software automatically reports about itself, such as error rates and response times.

Runbook — a written, step-by-step checklist for handling a specific kind of incident, so whoever is on call can follow the same procedure every time.

SRE (Site Reliability Engineering) — the practice of keeping production systems running reliably, treating operations problems as software problems to be solved with code.

How Virtual Machines Work - A Deep Dive

Neo Kim — Mon, 25 May 2026 07:25:33 GMT

Share this post & I'll send you some rewards for the referrals.

I have a huge huge huge announcement for everyone…

INTRODUCING: DevOps Mastermind

This newsletter series will help you become a fast-growing and high-paying engineer.

If you’ve ever thought:

“I use Docker & Kubernetes, but I still don’t fully understand how they work.”
“I want to master DevOps fundamentals instead of just memorizing commands.”
“I should learn how modern infrastructure & real-world systems actually work.”

Then this is for you.

Here’s what you’ll get inside DevOps Mastermind:

Architecture breakdown of core DevOps systems.
Deep dives into Docker, Kubernetes, Terraform, and so on.
Practical insights into how real-world systems achieve scalability, automation, and reliability.

Onward.

Your laptop is already running an operating system.

Now imagine starting another one on the same machine. A completely separate system with its own files, memory, and processes. You can run Linux inside Windows, or Windows inside a Mac. It boots independently, runs programs, and behaves as if it has full control over the hardware.

But it doesn’t.

Underneath, there is still only one physical machine. One CPU, one pool of memory, one disk. Yet multiple such systems can run at the same time, each isolated from the others and each functioning as if it were the only system present.

This idea changed computing…

Instead of buying a new server each time, you could split one powerful machine into many smaller ones. Each workload gets its own space. Nothing interferes with anything else.

That shift is what made cloud computing possible…

When you run an app on Amazon Web Services, Microsoft Azure, or Google Cloud Platform, it runs inside a virtual machine. The entire cloud infrastructure is built on this foundation.

But how does it actually work? How can one machine present itself as many? How are resources shared without breaking isolation?

Build strong data foundations for agentic AI at scale (Partner)

AI experiments are easy.

But building trusted AI systems that scale across an enterprise is hard. That’s exactly what leaders from Mercedes-Benz, Yahoo, Regeneron, and AWS discuss in this on-demand virtual panel on agentic AI.

Here’s what you’ll learn inside:

Practical strategies from enterprise leaders: See how teams move from isolated AI experiments to production-scale systems.
Real-world guidance on governance and architecture: Learn how industry experts approach analytics, unified governance, and continuous learning.
Frameworks for scaling intelligent systems: Discover how to build AI systems that drive business value.
Pragmatic infrastructure decisions: Learn how to select the right databases and data foundations for AI-based applications.

Watch the AWS panel and see how enterprise teams are building trusted AI systems with speed, governance, and control.

(Thanks to AWS for partnering on this post.)

I want to introduce Ayaan as a guest author.

He’s a self-taught DevOps engineer focused on Kubernetes & cloud infrastructure. He builds in public and writes about cloud-native systems, sharing daily technical tips and projects.

Check out his work and socials:

Twitter
Github

Here’s what’s inside this newsletter:

The difference between emulation and virtualization, and why one simulates hardware while the other shares it.
What a virtual machine actually is, how it presents itself as a full computer, and why the guest operating system believes it owns the hardware.
How the hypervisor works, the different types it comes in, and how it controls access to CPU, memory, storage, and networking.
What happens under the hood when a virtual machine boots, from resource allocation and virtual hardware setup to kernel startup and runtime execution.
How features like live migration, isolation, and resource controls make virtual machines practical for modern infrastructure.
Where virtual machines fit today, why cloud platforms depend on them, and how they differ from containers.

Emulation vs Virtualization

Before we get into how virtual machines work, there is one distinction worth getting clear early.

Emulation is when software pretends to be completely different hardware. Retro gaming emulators work this way. Your laptop takes each instruction from the old console and turns it into something your modern chip understands.

It works, but it is slow…

Virtualization is different. Instead of pretending to be different hardware, it creates a version of the same hardware your machine already has. Most instructions run directly on the real processor with no translation needed.

The result is near-native speed.

Think of it this way:

Emulation is like hiring a translator for every sentence in a conversation.
Virtualization is like realizing that both people already speak the same language.

This distinction matters because speed is what made virtualization practical at scale. You cannot run thousands of emulated systems on a single server. With virtualization, you can.

And that is exactly what cloud providers do…

What a Virtual Machine Actually Is

A virtual machine is a complete computer abstraction built on top of a physical machine.

It’s presented with a CPU, memory, storage, and a network interface, just like a physical system. The operating system inside it interacts with these components as if they were real hardware. It has no awareness that it is sharing the underlying machine with anything else.

From its perspective, it owns the system.

This behavior is made possible by a layered design.

At the bottom is the physical hardware, which provides the actual compute, memory, and storage resources. Above it sits the hypervisor, which controls how those resources are allocated and accessed. On top of that, each virtual machine runs its own operating system in isolation.

The hypervisor is the layer that makes the illusion possible.

Because the operating system never interacts with the hardware directly.

Instead, every request for CPU time, memory access, disk I/O, or network communication is handled through the hypervisor. It decides how resources are shared and ensures that each virtual machine remains isolated from the others.

This abstraction enables three key capabilities:

Isolation: Each virtual machine runs independently, so a failure in one doesn’t affect others.
Portability: Virtual machines can move between physical systems with minimal changes.
Efficient resource use: Multiple workloads share the same hardware instead of leaving it idle.

These properties are what made modern infrastructure possible.

Next, let’s look at the hypervisor: what it is, the different types, and how it powers virtual machines…

Hypervisor: Brain Behind the Illusion

The hypervisor is the software layer that enables virtualization.

It sits between the physical hardware and the virtual machines running above it. Every request a virtual machine makes, whether for CPU time, memory, or storage, goes through the hypervisor. It’s responsible for mapping those requests to the underlying hardware resources.

This layer controls all access to the hardware.

Type 1 vs Type 2

Not all hypervisors follow the same structure. The main difference comes from where they run.

Type 1 (bare-metal): It runs on the physical hardware, with no host operating system in between. Examples include VMware ESXi, Microsoft Hyper-V, and Kernel-based Virtual Machine (KVM). Cloud environments and data centers use these because they provide direct hardware access and minimal overhead.
Type 2 (hosted): Runs on top of an existing operating system, like a regular application. Tools such as VirtualBox and VMware Workstation fall into this category. They are easier to use but introduce additional overhead.

The difference lies in how the hypervisor accesses hardware and the number of extra layers it adds…

Full Virtualization vs Paravirtualization

There are also different ways a hypervisor presents hardware to a virtual machine.

In full virtualization, guest operating system runs without modification and behaves as if it were on real hardware. The hypervisor handles all the work required to maintain that abstraction.

In paravirtualization, the guest operating system is aware that it is running in a virtual environment. Instead of relying only on emulated hardware, it asks the hypervisor to handle some tasks, which reduces overhead.

The trade-off is between compatibility and performance.

Full virtualization runs any unmodified operating system. Paravirtualization is faster because the guest works with the hypervisor.

Hardware-Assisted Virtualization

Early hypervisors relied on software, making virtualization complex and inefficient. This changed when processor manufacturers introduced hardware support such as Intel VT-x (Intel Virtualization Technology for x86) and AMD-V (AMD Virtualization).

With these features, CPU itself can differentiate between virtual machines and the hypervisor. Sensitive operations pass control to the hypervisor when needed. It manages access and keeps everything secure.

This made virtualization practical at scale.

Nested Virtualization

It is also possible to run a virtual machine inside another virtual machine, a setup known as nested virtualization. You use this for testing, development, and running tools like Kubernetes locally in virtual machines (VMs).

But each additional layer introduces overhead and increases complexity. As you add more virtualization layers, performance drops. This makes the approach unsuitable for most production workloads.

It is useful in controlled scenarios, but not a general-purpose design.

Now you know what the hypervisor does. The next question is how it shares the CPU across multiple virtual machines. The answer is CPU virtualization.

Let’s look at how it works…

CPU Virtualization

The CPU is the brain of any computer.

Every instruction, process, and calculation goes through the CPU. So when you create a virtual machine, the question becomes: how do different operating systems share one CPU without getting in each other’s way?

That is what CPU virtualization solves…

The Privilege Model

Operating systems are designed to be in charge.

When your OS needs to access hardware, manage memory, or control processes, it does so with full authority. The processor has a concept called privilege levels to enforce this.

Think of it like levels of trust, where the most trusted software gets the most access.

In a normal system, the operating system sits at the highest privilege level.

But in a virtual environment, the hypervisor needs to sit above the operating system. It needs to intercept certain instructions and control what the guest OS can and cannot do. So the guest OS gets moved down to a lower privilege level, even though it still believes it is fully in charge.

The guest OS never knows the hypervisor demoted it.

This is the fundamental trick behind CPU virtualization. The guest OS continues to behave as if it owns the processor. But when it tries to do something that needs high-level access, the processor steps in and hands control to the hypervisor.

Everything continues smoothly, just one level deeper…

vCPU and Scheduling

When you create a virtual machine, you assign it CPUs, but these are not physical cores. They’re virtual CPUs (vCPUs) that represent processing capacity managed by the hypervisor.

The hypervisor schedules vCPUs onto physical cores in the same way an operating system schedules processes.

At any given moment, a physical core can execute only one thread of work. The hypervisor switches between virtual machines, giving each one a slice of CPU time. This switching happens fast enough that each virtual machine appears to have continuous access to the processor.

Modern CPUs may expose multiple logical threads per core through technologies like Intel Hyper-Threading or AMD SMT (Simultaneous Multithreading), but CPU time is still shared and scheduled across workloads.

In reality, all virtual machines share it.

This is called time-slicing. It’s the same idea your operating system uses to run multiple applications at once.

Hypervisors can also assign more vCPUs than there are physical cores, a practice known as CPU overcommit. For example, a machine with 16 cores might run virtual machines totaling 32 or more vCPUs.

This works as long as not all virtual machines are busy at the same time.

When demand rises across multiple VMs, they compete for CPU time. This causes contention and reduces performance.

Overcommit is a useful tool, but it depends on workload behavior.

VM-exit and VM-entry

Let’s go one level deeper into what actually happens when a guest OS tries to do something it is not allowed to do on its own.

When a virtual machine tries to run a privileged instruction, the processor stops it. Control is immediately transferred to the hypervisor. This event is called a VM-exit.

The hypervisor then figures out what the guest was trying to do and handles it safely.

Once the hypervisor has dealt with the request, it hands control back to the virtual machine exactly where it left off. This is called a VM-entry. From the guest OS’s perspective, nothing unusual happened. The CPU executed the instruction and continued running.

The entire handoff takes microseconds.

This mechanism is what makes isolation possible. The guest OS cannot access hardware it shouldn’t. Every sensitive operation triggers a VM-exit, which the hypervisor controls. But it also introduces overhead. Every VM-exit and VM-entry takes time, and if they happen too frequently, performance suffers.

Keeping VM exits to a minimum is one of the core challenges of CPU virtualization.

NUMA Awareness and CPU Topology

This matters more in large systems and performance-sensitive workloads, but it’s still worth understanding at a high level…

Modern servers often have more than one physical processor. Each processor has its own local memory. Accessing memory that belongs to a different processor is slower than accessing your own. This architecture is called Non-Uniform Memory Access (NUMA).

The name describes the problem exactly. Not all memory accesses are equal.

In a virtual environment, this creates a subtle issue. A virtual machine might run on cores from one processor but end up accessing memory that belongs to another. The guest OS does not know this is happening. But the performance hit is real.

The solution is NUMA-aware scheduling.

The hypervisor tries to keep a virtual machine’s vCPUs and its memory on the same NUMA node. When it succeeds, the VM runs faster. When workloads grow and resources get spread across nodes, the performance gap becomes noticeable.

For most small deployments, this never comes up. For high-performance databases or real-time systems running on large servers, it matters a lot.

Putting it together, CPU virtualization is not a single mechanism but a combination of coordinated techniques.

The privilege model restricts direct hardware access. vCPU scheduling creates the illusion of dedicated processing power. VM-exits and VM-entries enforce safe boundaries for sensitive operations. NUMA awareness ensures that physical hardware layout does not introduce hidden performance penalties.

Each part addresses a specific challenge.

Together, they allow multiple virtual machines to share a single physical CPU while operating independently and efficiently.

Next, let’s look at how memory is virtualized and how virtual machines manage their view of system memory…

Reminder: this is a teaser of the subscriber-only newsletter, exclusive to my golden members.

When you upgrade, you’ll get:

Architecture breakdown of core DevOps systems.
Deep dives into Docker, Kubernetes, Terraform, and so on.
Practical insights into how real-world systems achieve scalability, automation, and reliability.

Let’s keep going!

Memory Virtualization

AI Agents: State, Memory, Consistency - A Deep Dive

Neo Kim — Sun, 17 May 2026 09:40:01 GMT

Share this post & I’ll send you some rewards for the referrals.

The hardest part of building an AI agent1 has nothing to do with the model…

Calling a language model is the EASY part…

Keeping the agent coherent across a workflow that spans hours or days is where everything actually gets difficult: remembering what was said, tracking what’s already been done, and staying steady when the user changes their mind.

Even with stronger models each year, agents break on those exact failures…

They forget earlier turns, repeat questions they already asked, and behave inconsistently across long tasks. The model is rarely the cause. The cause is a missing or badly designed state and memory, plus the consistency rules that keep the two in line.

This newsletter walks through all three…

100+ Claude Code hacks to ship code 10X faster (Partner)

Top engineers at Anthropic and OpenAI say AI now writes 100% of their code.

If you’re not using AI, you’re spending 40 hours doing what they do in 4.

These 100+ Claude Code hacks fix that and help you ship 10x faster.

100+ Claude Code hacks used by top engineers — free
The Code newsletter — learn the latest AI tools, tips, and skills to code faster with AI in 5 minutes a day

Claim your free playbook

(Thanks to the Code for partnering on this post.)

I want to introduce Sivasankar as a guest author.

He is a Technical Director and GenAI practitioner with over 20+ years of experience in architecting and building Big Data, Cloud, and GenAI solutions.

His work focuses on helping organizations design practical and scalable intelligent systems. He enjoys exploring new ideas through hands-on DIY projects and practical experimentation, often turning real-world use cases into working prototypes.

You can connect with Sivasankar (Shiva) on LinkedIn.

Here’s what you’ll find inside this newsletter:

State vs memory. What each one is for, how they differ, and why conflating them is the root cause of most agent bugs.
Memory lifecycle. How an agent creates, updates, summarizes, and deletes memory so it doesn’t rot over time.
Consistency under change. How state reacts to new instructions right away, and how long-term memory updates only after a change holds up.
A reference architecture. The four layers (brain, state, memory, external systems) and what each one owns.
Scaling and failure modes. Where stateful agents break at scale, and the three ways memory itself goes wrong.

By the end, you’ll know how to design agents that track the right things, recover from failure, and stay truthful with the data they don’t control.

Let’s start with the state...

Tracking what the agent is doing with the state

State is the information an agent keeps about the current task: the plan, active constraints, what’s already been done, and what comes next.

It’s the agent’s picture of NOW…

The easiest way to see why it matters is to compare an agent with it to one without it.

Stateless vs Stateful

A stateless agent keeps nothing between requests…

Each input is a fresh start, which is fine for one-shot jobs like “translate this sentence to French, summarize this paragraph, or explain what an API is”.

A stateful agent tracks the current task, session, or workflow…

At each decision point, it checks the state to answer three questions: what step am I on, what has already been done, and what should I do next? Those answers drive the next action, check constraints, and stop the agent from repeating itself.

State matters the moment a workflow has more than one step…

The user books a flight (step one done, constraints locked), asks for a hotel (step two, new constraints layered on), then changes the return date (earlier step invalidated, replan downstream).

Without a state, none of those three questions has an answer.

Where state lives

State is maintained on the agent side (backend server that runs the reasoning loop), and NOT on the client side (chat UI, mobile app, or API caller the user interacts with).

Clients disconnect, refresh, and retry. And agent still needs a steady view of progress.

So where that view lives depends on how long the workflow runs…

Short-lived workflows that end in a single session can remain in an in-memory store. Anything that has to survive restarts needs external storage: a database, a KV store (Redis-style lookups by id), or a serialized state object managed by an orchestration framework2.

LangGraph3, for example, comes with SqliteSaver and PostgresSaver as ready-made checkpoint backends.

At each step’s start, agent loads the latest state snapshot and rebuilds its context. That load makes a chain of steps feel like one chain rather than a set of separate calls.

Checkpointing and recovery

Loading state on each step only works if something wrote it in the first place…

That’s what checkpointing4 does: it saves state at the milestones you pick as meaningful, not every tiny step. If the agent crashes, times out, or gets redeployed mid-workflow, it reloads the last checkpoint and resumes from there.

Picture the travel agent picking a return flight when the API call fails…

Without checkpointing, the agent would have to re-ask the user for dates, budget, and outbound flight details. With checkpointing, the saved state already contains all that, so on retry, the agent picks up at the failed step and continues.

The same setup covers internal errors in the agent’s own code and redeploys that restart the infrastructure underneath it.

This pattern runs in production at real scale…

LinkedIn’s internal SQL Bot5 is a hierarchical multi-agent system built on LangGraph, and its long-running query workflows rely on the framework’s checkpoint layer to pause, inspect state, and resume without losing context mid-task.

State versioning

Long-running agents change over time…

New fields appear, steps get reordered, logic shifts. Without versioning, in-progress workflows break the moment the schema changes, and the stored state written yesterday becomes unreadable tomorrow.

Versioning keeps yesterday’s state readable…

It migrates old formats forward and blocks execution paths that no longer make sense under the new schema. That way, workflow changes roll out without killing ongoing tasks.

This is most evident in production systems where workflows span minutes, hours, or even days…

Travel agent: state in action

Storage, checkpointing, and versioning click together once you watch a concrete plan move through them…

The user says, “Plan a complete trip for me.” The agent then walks a sequence:

Pick the date and destination.
Select an outbound flight.
Then select a return flight.
Choose accommodation.
Confirm the itinerary.

After each step, the state moves forward…

Once the outbound flight is selected, the state says select the return flight next. And checkpoints occur after preferences are confirmed, outbound flight is selected, return flight is selected, and the hotel is booked.

If the hotel step ‘fails‘ because of a network error, the agent restarts, reloads the state, sees that the dates and flights are already locked in, and retries only the hotel step.

NO re-asking and out-of-order suggestions…

That’s what the state gives you.

Next up is memory: information the agent carries across tasks...

Share this post & get rewards for the referrals.

Carrying information across tasks with memory

If the state is the agent’s picture of now, memory is what spans across: past decisions already made, preferences built up over time, conversations it can refer back to, and reference data it queries from other systems (flight schedules, hotel listings).

Agents store memory in three flavors, separated by how long the memory persists and whether the agent owns the data or just reads it from elsewhere.

Short-term memory

Short-term memory holds the current context for the active task…

It’s short-lived and cleared when the task ends. Most implementations keep it in in-memory structures, session variables, or within the LLM6 prompt. The lifecycle is simple: create it when a task begins, update it as the agent reasons and uses tools, and clear it once the task wraps up.

Use it for one-task decisions that don’t need to outlive the current workflow.

Long-term memory

Long-term memory stores information across interactions: user preferences, past bookings, and recurring patterns.

The storage options are vector databases7, KV stores, document stores, or relational databases. A write only happens when the information is actually worth keeping, and retrieval kicks in at the start of new tasks or when a relevant signal shows up in the current one.

Every so often, the agent summarizes or merges old entries to prevent the store from growing out of control…

Use this layer when a preference is steady, behavior repeats, or a piece of information should carry into future interactions.

The big win of long-term memory is NOT loading the entire history into the prompt each turn. The agent stores what’s worth keeping and pulls in only what the current decision needs.

Mem08 is one production system built on this idea:

Tested on LOCOMO9, a benchmark for long conversations, the Mem0 layer reduced p95 retrieval latency10 by 91% against a full-context baseline11. Token cost dropped by over 90% on the same test.

On an LLM-as-judge evaluation12, it scored 26% higher than OpenAI’s memory system.

External memory

External memory is a large reference data that the agent queries on demand instead of storing: flight databases, hotel listings, knowledge graphs, and internal systems of record.

These are external sources that hold the authoritative version of a piece of data, like a booking system or a pricing API…

These live in cloud databases, APIs, knowledge graphs, or external RAG13 indexes. Nothing gets loaded into the prompt all at once. When a decision depends on data that must remain official, auditable, and shared across applications, the agent queries the system of record14 directly.

Memory lifecycle

Even a good memory goes bad if you never manage it. Managing it means walking every entry through four stages:

Create when new information arrives (user picks a flight).
Update when details change (user adds a second city).
Summarize to shrink detail (only the times and locations that matter, not every API response).
Delete when the information is no longer needed or the retention window ends (clear past trip details once the booking is complete).

Skip this cycle, and long-term memory grows without limit.

Old preferences start contradicting newer ones, and the agent makes decisions based on a pile of half-truths…

Travel agent: memory in action

One trip exercises all three memory types at once:

Short-term memory holds the current trip (cities, dates, flight search results).
Long-term memory holds repeat preferences (airline, seat class, meal choice).
External memory provides live data that the agent doesn’t own (current flight schedules, hotel availability, seat prices).

The hard part for the agent isn’t storing any of this…

It’s deciding when the state should override memory, when memory should update from what just happened, and when neither should move because the real answer lives in an external system.

That’s the next problem...

100+ Claude Code hacks to ship code 10X faster (Partner)

Top engineers at Anthropic say AI now writes 100% of their code.

Are you using AI to write yours?

These 100+ Claude Code hacks show you exactly how. Sign up for The Code and get:

100+ Claude Code hacks — free
The Code newsletter — learn the latest AI tools and skills to code faster in 5 mins a day

Claim your free playbook

Staying consistent when preferences change

Real users change their minds mid-task.

They try a new option, correct themselves, adjust what they want, and sometimes say one thing on Monday and the opposite on Tuesday. An agent that can’t handle this feels wrong: repetitive, stuck on old instructions, or too eager to rewrite its long-term knowledge on every passing comment.

The core rule is simple: react fast with state, learn slowly with memory.

Reacting fast, learning slowly

State updates as soon as a new instruction arrives…

The current task reflects it right away. But memory updates only after the change looks steady and reusable.

The question is when memory writes happen…

LangChain’s team frames two options: the agent can write “in hot path” (save the new fact before replying to the user) or “in the background” (a separate process updates memory during or after the conversation).

Background writes are what make “learn slowly” work, because they give the system time to check whether the change holds up over a few more turns before it gets saved.

If memory learned too fast, it would rot15…

A rare exception becomes the rule, a short-term constraint becomes permanent, and future tasks go wrong with no clear cause.

Three roles keep this clean:

State tracks what the agent is doing now,
Memory stores what it has learned over time,
Consistency rules keep the two in check when they disagree.

Before any of these rules run, the agent has to pull the right memory for the current decision.

Retrieving only what’s relevant

Agents don’t load all memory…

Instead, they query it like a database, using metadata on each entry:

topic: flights [seats, airlines, class, timing]
topic: hotels [room type, location, amenities]
topic: food [dietary restrictions, cuisine]
topic: budget [price sensitivity, limits]
duration: short-term | long-term
confidence: confirmed | inferred

The current state picks which slices to pull.

Booking a flight pulls flight and budget memory, and leaves hotel memory alone. This keeps the working context focused and stops old or unrelated memory from leaking into the decision.

Rolling back on corrections

When a user corrects the agent, treat it as a constraint change, NOT an error.

The agent always plans under a set of constraints:

seat type = aisle
budget <= 800
departure = morning
hotel near the city center

When the user says, “I want a window seat”, they aren’t saying you made a mistake. They’re saying update the constraints. The agent’s job is to apply the new constraint without throwing away the work that’s still valid.

On a correction, the agent traces which steps depended on the old constraint…

It rolls back to the earliest affected step, marks everything downstream as invalid, and leaves the earlier valid steps alone.

But memory doesn’t update on the first correction…

The agent first checks intent: whether the change is a one-time request, task-specific, or a lasting preference (indicated by phrases like “always a window seat”). Only after the change looks steady does long-term memory update.

When the system of record wins

NOT all memory is truth…

Prices, availability, and inventory come from external systems, not from the agent’s stored preferences. External systems define ground truth; memory shapes decisions without overriding real data, and when the two disagree, the system of record always wins.

Anthropic names this pattern directly in its agent design guidance:

Agents must “gain ‘ground truth’ from the environment at each step (such as tool call results or code execution) to assess its progress.” That keeps the agent in sync with reality, and hallucinations drop because it isn’t guessing at live data it could have just fetched.

Travel agent: consistency in action

The agent already knows 2 things:

Long-term memory: user usually prefers aisle seats.
Short-term memory: user wants the cheapest flight for this trip.

Right now, the agent is comparing flights and choosing seats…

Then the user says:

“I want a window seat this time.”

Here’s what the agent does:

Reads the new request.
Pulls only seat-related preferences (memory).
Understands “this time” means a temporary change and labels it accordingly.
Rolls state back to the flight selection step because seat preferences affect flight choices.
Removes earlier seat and pricing decisions that depended on aisle seats.
Recalculates the best flight options using the new window-seat preference.
Updates short-term memory for this trip ONLY.
Keeps long-term memory unchanged because preference may NOT be permanent.

Later, the user says:

“I prefer window seats going forward.”

Now the agent understands this is a lasting preference…

So it:

Read this as a lasting change.
Saves window seats as the new default preference.
Updates long-term memory.
Uses this preference for future trips.

Now, imagine the airline system says there are no window seats available…

The agent tells the user that window seats are unavailable and follows the airline’s live data. It does not remove the preference from memory. Instead, real-world data overrides stored preferences.

So far, the travel agent has shown up in pieces. Now it’s time to run it end-to-end...

Share this post & get rewards for the referrals.

Build travel agent from three pieces

Travel Agent AI helps users plan and book trips…

It talks to the user, tracks the current task, remembers useful preferences, and calls external tools to check flights, hotels, and prices.

Every request flows through the same loop…

Workflow loop

User input. The user asks for a plan or a specific booking.
Intent parsing and reasoning. The agent figures out the goal and picks the next step.
State update. Records which step is active and the constraints in play (dates, airline preference, budget).
Memory access. Pulls short-term, long-term, or external memory based on what this step needs.
Tool execution. Calls APIs for flights, hotels, or pricing.
Response generation. Gives a suggestion or confirmation from the current state and retrieved memory.
State/memory update. Writes the new selections and any session context back to storage.
Next step or loop. Continues to the next task in the plan.

Example prompts

A typical conversation might run across four turns:

“Help me plan a trip to Paris next month.”
“Book the outbound flight first.”
“I prefer window seats and morning flights.”
“Now add a hotel near the city center.”

Across these turns:

State tracks the active step.
Short-term memory stores choices for this trip.
Long-term memory stores recurring preferences.
External data provides live prices and availability.
Tools connect the agent to flight, hotel, and booking systems.

Agent’s tools

The agent reaches out through flight search APIs, hotel availability services, and pricing and booking systems…

Each step feeds the next, and the saved state keeps the plan aligned with live data even when something fails partway through.

The loop shows when things happen…The next question is what each piece is responsible for...

Reference architecture for a stateful agent

Zoom out from the runtime loop, and a stateful agent splits into four layers:

Each one has a single job, and the lines between them are what keep the agent steady.

Agent brain (reasoning engine). An LLM sits at the center, planning multi-step workflows, reading user intent, and picking the next action from the current state and retrieved memory.
State layer. This is where the current workflow lives: which steps are done, which are next, and what constraints are active. It also holds short-term overrides and session-specific choices, and lets the agent roll back when a user changes a preference without restarting the entire plan.
Memory layer. Short-term memory carries this-task context; long-term memory carries lasting user preferences and habits. Together, they feed the brain with context that matches past decisions and patterns.
External systems (system of record). APIs, databases, and live data sources provide official information such as seat availability, current prices, and cancellation policies. Memory can suggest, but the system of record always wins in a conflict.

The brain decides, the state tracks, memory remembers, and external systems check.

Mix up those responsibilities, and the agent guesses when it should check a live source, or keeps memory that should have been dropped. Keep them separate, and every layer pulls its own weight.

Share this post & get rewards for the referrals.

Deciding what fits in the context window

The four layers only work if they fit…

Every LLM runs inside a finite context window16, and state, memory, tool schemas, tool outputs, and user messages all fight for that space.

What takes up the context window

Every step’s context is the sum of what you send the model in that step…

This includes system and developer instructions, current state, retrieved memory snippets (short-term, long-term, and external), tool schemas and instructions, tool outputs from API responses and search results, and the user’s messages themselves.

A useful mental model for the budget (not a formula to measure):

Context size = instructions + state + memory + tools + data + user input

Visibility is what matters here, not precision…

Tool definitions and raw API responses often consume more context than the model’s actual reasoning.

Anthropic ran into this in their own Claude Code setup:

The full tool definitions17 alone used about 134,000 tokens of context. To fix this, Anthropic introduced Tool Search, a lazy-loading system that loads a tool’s full schema only when the agent actually needs it.

This reduced tool-definition overhead by 85%, increased usable context per session from 122,800 tokens to 191,300, and improved MCP tool-use accuracy for Claude Opus 4 from 49% to 74%.

If the agent loads everything all the time, the context window fills up quickly…

When that happens, older messages drop out first. This usually removes the earliest and most important information, like the user’s original goals, constraints, or instructions.

Retrieve only what’s relevant

Use tags, scope, and recency to pull what matches the current step, not everything the agent has ever learned.

Match retrieval to the current state: booking a flight pulls flight and budget memory, nothing else.

Relevance is really a question of how much history you want to pay for right now…

Summarize and remove old checkpoints

In long workflows, the agent’s state keeps growing after every step…

Keeping every raw log and tool response makes the context large, expensive, and noisy. Instead, the agent can periodically create a short summary of completed work using an LLM.

The summary should keep only the important information:

decisions already made
constraints that are now fixed
tasks that still need to be completed

The agent should remove detailed logs that no longer affect future decisions.

And summaries should not stay forever either.

If an old summary is no longer useful for future tasks, the agent should archive it or delete it. Outdated summaries can confuse the agent just as outdated memory does.

Keep memory outside the prompt

The agent should not keep all its memory in the prompt at all times…

Instead, memory should remain in external storage and be retrieved only when needed. Adding too much memory directly into the prompt quickly fills the context window, increases cost and latency, and makes reasoning less focused.

Most stored memory is only occasionally useful and unnecessary for every step…

Before adding memory to the context, the agent should ask:

Does this change what I should do next?
Is this information still valid?
Would removing it break the current plan?

If the answer to all three questions is “no”, the memory should stay out of the context.

Good agents do NOT succeed by remembering everything…Instead, they retrieve only the information that matters right now.

Cost, latency, and reliability trade-offs

Every reliability improvement comes with a cost…

Retrieving more memory increases latency. Extra tool calls increase costs. Additional consistency checks slow the response down.

At scale, fast agents are usually cheaper but less reliable. More careful agents are safer but slower. You cannot optimize for speed, cost, and reliability simultaneously.

In practice, teams focus on accuracy where mistakes are expensive, such as bookings or payments. Also, they cache tool results when possible to avoid repeated API calls.

Plus, batching API requests reduces overhead. Teams also monitor context size and tune memory retrieval using real usage data instead of guesses.

Now, imagine the travel agent after handling hundreds of trips…

Its long-term memory now contains many user preferences, budgets, airlines, and booking patterns. If the agent retrieves too much memory for every request, the context becomes large and inefficient.

A scalable travel agent:

retrieves only trip-related preferences
summarizes old planning steps
trusts live pricing APIs over stored prices
keeps prompts small and focused as history grows

Without this discipline, the agent gradually becomes slower, more expensive, and less reliable…

Efficiently managing context is only part of scaling an agent system.

The other challenge is handling memory failures correctly…

100+ Claude Code hacks to ship code 10X faster (Partner)

Top engineers at Anthropic say AI now writes 100% of their code.

Are you using AI to write yours?

These 100+ Claude Code hacks show you exactly how. Sign up for The Code and get:

100+ Claude Code hacks — free
The Code newsletter — learn the latest AI tools and skills to code faster in 5 mins a day

Claim your free playbook

How memory goes wrong

Even if an agent manages context well, its memory system can still fail…

Memory makes agents more capable, but it also creates new ways for them to behave incorrectly. In real systems, three common memory failures occur repeatedly.

These failures happen at different stages of the memory pipeline:

What gets stored
What stays over time
What gets retrieved later

1 Stale memory

The agent relies on outdated information…

Example: The agent remembers that the user prefers window seats, but the user recently switched to aisle seats. Without an update, the agent keeps suggesting window seats.

Why it matters: Old preferences can override newer user intent if the system does not update or expire memory correctly.

2 Wrong information captured

The agent stores incorrect information…

Example: During a conversation, the agent mistakenly saves the user’s phone number as their airline loyalty number.

Why it matters: Once incorrect data enters long-term memory, it can affect many future tasks and become difficult to trace back to the original mistake.

3 Retrieval failures

The correct memory exists, but the agent fails to retrieve it when needed…

Example: The user almost always books business class, but the agent fails to retrieve that preference and instead recommends economy.

Real-world example: In July 2025, a Replit coding agent deleted a live production database during an active code freeze, even after the user repeatedly typed “DON’T DO IT.”

The instruction existed in the agent’s context, but the system failed to apply it correctly. The agent later generated thousands of fake user records to hide the issue. Replit’s CEO publicly apologized and later introduced an automatic separation between development and production environments.

Why it matters: Stored memory is useless if retrieval, ranking, or relevance systems fail to surface the right information at the right moment.

Storing memory is easy. But managing it correctly over time is the hard part.

Preventing these three failures is a major part of building reliable AI agents…

Remember the right things, NOT everything

Here’s a checklist for designing stateful, reliable AI agents:

Separate state from memory. State is the current workflow. Memory is short- and long-term knowledge.
Checkpoint and roll back. Roll back only the steps affected by a correction. Keep the rest.
Keep memory outside the prompt. Retrieve only what’s needed. Apply short-term and long-term rules differently.
Budget context carefully. Treat it as a working set. Summarize old checkpoints. Drop stale summaries.
Let the system of record win. External systems override stored memory when they disagree.
Balance cost, latency, and reliability. Spend precision where mistakes are expensive.
Monitor in production. Track context usage, retrieval hits, tool calls, and response times.
Add safeguards. Freshness rules, clear approval before writing to long-term memory, scope-aware updates.

When the three pieces work together, agents stop reacting and start adapting…

They survive failure, stay honest with real-world data, and scale without their context budget falling apart. The trick isn’t more memory or a bigger model. It’s knowing what to track, what to recall, and when to act.

👋 I’d like to thank Sivasankar for writing this newsletter!

He is a Technical Director and GenAI practitioner with over 20+ years of experience in architecting and building Big Data, Cloud, and GenAI solutions.

Louis and I launched the GENERATIVE AI MASTERCLASS (newsletter series exclusive to PAID subscribers).

When you upgrade, you’ll get:

Simple breakdown of real-world architectures
Frameworks you can plug into your work or business
Proven systems behind ChatGPT, Perplexity, and Copilot

👉 CLICK HERE TO JOIN THE GENERATIVE AI MASTERCLASS

(Golden members will get the next Generative AI newsletter in the first week of June.)

If you find this newsletter valuable, share it with a friend, and subscribe if you haven’t already. There are group discounts, gift options, and referral rewards available.

👋 Find me on LinkedIn | Twitter | Threads | Instagram

Want to reach 210K+ tech professionals at scale? 📰

If your company wants to reach 210K+ tech professionals, advertise with me.

Thank you for supporting this newsletter.

You are now 210,001+ readers strong, very close to 210k. Let’s try to get 211k readers by 29 May. Consider sharing this post with your friends and get rewards.

Y’all are the best.

Anthropic Engineering. Building effective agents. December 2024. https://www.anthropic.com/research/building-effective-agents
Anthropic Engineering. Effective context engineering for AI agents. https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents
Anthropic. Tool use with Claude. Claude API documentation. https://platform.claude.com/docs/en/agents-and-tools/tool-use/overview
LangChain. Memory overview. LangGraph documentation. https://docs.langchain.com/oss/python/langgraph/memory
LangChain. Persistence. LangGraph documentation. https://docs.langchain.com/oss/python/langgraph/persistence
Lewis, P., et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Facebook AI Research, 2020. https://arxiv.org/abs/2005.11401
Packer, C., et al. MemGPT: Towards LLMs as Operating Systems. UC Berkeley, 2023. https://arxiv.org/abs/2310.08560
Pinecone. What is a Vector Database & How Does it Work? https://www.pinecone.io/learn/vector-database/
Chhikara, P., et al. Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory. 2025. https://arxiv.org/abs/2504.19413
Anthropic Engineering. Advanced tool use. https://www.anthropic.com/engineering/advanced-tool-use
Anthropic Engineering. Building effective agents. December 2024. https://www.anthropic.com/research/building-effective-agents
LangChain. Memory for agents. LangChain Blog, 2024. https://www.langchain.com/blog/memory-for-agents
LangChain. Is LangGraph used in production? LangChain Blog. https://www.langchain.com/blog/is-langgraph-used-in-production
AI Incident Database. Incident 1152: Replit AI agent deleted production database. https://incidentdatabase.ai/cite/1152/

AI agent is a system that uses AI to understand goals, make decisions, and take actions to complete tasks.

Orchestration framework. A library that manages how an agent’s steps run, how state is stored between them, and how failures are handled. LangGraph is one common example.

LangGraph is a framework for building AI agents and multi-step workflows with language models. It helps manage state, memory, tool usage, and workflow execution across many steps.
SqliteSaver is a built-in LangGraph component that saves agent state and checkpoints in a local SQLite database. It is useful for development or smaller applications.
PostgresSaver is a built-in LangGraph component that saves agent state and checkpoints in a PostgreSQL database. It is better suited for production systems that need scalability and reliability.

Checkpointing. Saving the agent’s state at safe points so a crash, timeout, or redeploy doesn’t lose progress. On restart, the agent reloads the last checkpoint and continues.

LinkedIn’s SQL Bot is an internal AI assistant that lets employees ask questions in natural language and automatically converts them into SQL queries to retrieve data from LinkedIn’s data warehouse.

Large Language Model (LLM). A neural network trained on large amounts of text to predict the next token in a sequence. In this newsletter, the LLM is the reasoning engine at the core of the agent.

Vector database is a database designed to store and search embeddings - numerical representations of text, images, or other data created by AI models. Instead of matching exact keywords like a traditional database, a vector database finds information based on semantic similarity, meaning similarity in meaning.

Mem0 is an open-source memory system for AI agents and AI assistants. It helps agents remember important information across conversations instead of sending the entire chat history to the model every time.

LOCOMO. A public benchmark of long, multi-session conversations used to test how well a system can answer questions about things said much earlier in the dialog.

p95 latency. Time it takes to complete an operation in 95 out of 100 calls. It describes the slow tail of the system, not the average.

Full-context baseline. Putting the entire conversation history into the prompt on every call, with no picking or shortening. Used as a point of comparison for memory systems that try to be smarter about what they include.

LLM-as-judge evaluation. Using another language model to score the quality of answers, instead of a human rater.

RAG: retrieval-augmented generation, where an agent fetches relevant documents at query time.

System of record. The external system that holds the authoritative version of a piece of data: a booking system, a pricing API, an inventory database. In a conflict with stored memory, the system of record always wins.

Memory rot happens when the agent saves too many weak or temporary signals as permanent knowledge.

Context window. The maximum number of tokens an LLM can process in one request. Instructions, state, memory, tool schemas, tool outputs, and user messages all fight for this space.

Tool definition is a description of a tool that tells the AI agent:

What the tool does
What inputs it accept
What output it return
When and how to use it

You can think of it like an API contract or a function specification for the model.

How ML Systems Actually Work: From Data to Deployment

Paolo Perrone — Tue, 12 May 2026 09:02:04 GMT

Share this post & I'll send you some rewards for the referrals.

You’ve probably walked into a machine learning (ML) system design interview, been asked to “design a recommendation system,” and suddenly everyone’s throwing around terms like embeddings, feature stores, and model drift, and you’re nodding along while silently drowning.

Most prep resources offer algorithm deep-dives or LeetCode-style drills.

But first, what you actually need is a solid understanding of how ML systems work: vocabulary, building blocks, and how they fit together.

This newsletter is a plain-English field guide to ML system design.

You don’t need to know TensorFlow, write Python, or understand gradient descent math. Just what these terms mean in practice, so you finally understand how these systems actually work.

Along the way, we’ll trace how real systems (like Netflix recommendations, Uber’s ETA predictions, and Spotify’s Discover Weekly) actually work under the hood. Don’t worry about memorizing every detail; this guide exists so your next ML interview feels less like an interrogation and more like a conversation.

Let’s start with the question every ML interview is really asking.

Treat coding agents like services, not terminals (Partner)

You use Claude Code, Codex, Cursor, or Gemini in your terminal every day. What if you could call them in code?

AgentField just open-sourced harness orchestration.

Call those same coding agents programmatically from Python, TypeScript, or Go.
Compose them into real automations: ship a PR, audit a cloud, run a code review - end-to-end pipelines, not toy demos.

For the architecture rationale, their latest post - harness-as-membrane - is worth a read.

Fork and build

(Thanks to AgentField for partnering on this post.)

I want to introduce as a guest author.

He’s an ML engineer with 8+ years of production AI experience, read by 1M+ AI/ML engineers on LinkedIn and Substack. He runs a tech content agency serving NVIDIA, Google, and Y Combinator, and writes The AI Engineer, the newsletter that makes engineers dangerously good at AI engineering.

What is an ML System?

Traditional software runs on rules you write by hand.

If the temperature is above 90°F, turn on the AC. If the user clicks “buy,” charge the card. Every decision is hardcoded in code.

ML systems work differently.

Instead of writing rules, you feed the system examples (thousands or millions of them), and it figures out the rules on its own. Show it a million past rides, and it learns to predict how long your Uber will take. Show it billions of watch sessions, and it learns which movie to recommend next.

This sounds magical, but it’s just data engineering. Data flows in, patterns get extracted, predictions flow out, and the system keeps learning from its own results.

Every concept in this newsletter maps to a stage in this pipeline.

Master the pipeline, master the interview…

Raw Material: How Data Becomes Intelligence

You can have the fanciest model in the world. If your data is garbage, your predictions will be garbage.

1. Features

A feature is a single, measurable piece of information model uses to make predictions.

Think of features as the columns in a spreadsheet that describe each example.

If you’re predicting whether someone will click on an ad, your features might include the user’s age, the time of day, the device they’re using, and how many ads they’ve already seen this session. Each of these gives the model a different angle on the same question.

In Spotify’s Discover Weekly, features might include your most-played genres, the time of day you usually listen, and how often you skip songs. The model doesn’t “hear” the music. Instead, it reads the features.

The quality of your features often matters more than the sophistication of your model. Which raises the question: where do good features come from?

2. Feature Engineering

Feature engineering is the process of transforming raw data into features that help a model learn better.

Raw data is messy and unstructured. Feature engineering is about cleaning it up, reshaping it, and extracting the signal.

Think of it like preparing ingredients before cooking. You don’t throw a whole chicken into the pan. You cut it, season it, and lay it out first.

For example, Uber’s raw data might say “ride requested at 2024-12-25 08:47:32 UTC.”

Feature engineering transforms that into: day of week (Wednesday), time of day (morning rush), holiday (yes), weather (rainy). Each transformation gives the model a more useful signal than the raw timestamp alone.

Sometimes the most powerful features aren’t in the raw data at all. A feature like “average number of rides this user takes per week” requires aggregation across many records. That kind of creativity is what separates good feature engineering from great ones.

But features alone aren’t enough. The model also needs to know what the “right answer” looks like.

3. Labels & Ground Truth

A label is the answer you’re training the model to predict. Ground truth is what actually happened in the real world.

If you’re building a spam filter, the label for each email is “spam” or “not spam.” If you’re predicting home prices, the label is the actual sale price. The model learns by comparing its guesses against these known answers.

Ground truth sounds simple, but it’s often surprisingly hard to get right.

When Netflix asks “did the user enjoy this movie?”, the ground truth is ambiguous. Did they watch to the end? Did they rate it? Did they watch something similar next? Different definitions of ground truth lead to different model behaviors.

💡 How you define your label shapes what the model actually learns. Choose carefully. Pick the wrong definition of success, and the model will optimize for the wrong thing.

With features and labels defined, the next step is splitting the data so the model can actually learn and prove it learned the right things.

4. Training, Validation & Test Sets

You don’t test a student with the same questions they practiced on.

The same principle applies to ML. The data gets split into three groups, each with a specific job.

Training set is what the model actually learns from. It sees these examples during training.
Validation set is your mid-training checkpoint. It tells you which model version performs best.
Test set is the final exam. If the model performs well here, it’s ready for the real world.

Most teams split their data 70/15/15 or 80/10/10.

The exact ratio depends on how much data you have, but one rule is non-negotiable: test set must never influence model development. Otherwise, you’ve let the student peek at the final exam.

At Netflix, this might mean training on viewing data from January through October, validating in November, and testing in December. This time-based split is especially important for ML systems because real-world data changes over time, a pattern we’ll revisit in Section 5.

Now that we have our data organized, there’s one more data problem that trips up many real-world systems.

5. Class Imbalance & Data Leakage

Class imbalance happens when one outcome is far more common than the others.

In fraud detection, maybe 1 in 10,000 transactions is fraudulent. In medical diagnosis, most patients don’t have the rare disease. The model can achieve 99.99% accuracy by simply predicting “not fraud” every time and still be completely useless.

Fixing class imbalance usually involves techniques like oversampling the rare class, undersampling the common class, or adjusting the model’s loss function to penalize mistakes on rare examples more heavily.

Data leakage occurs when the answer is hidden within the training data.

You’re building a model to predict late deliveries, but your dataset includes “customer complaint filed.” That complaint only exists because the delivery was already late. Of course, the model looks accurate. Remove that feature, and it’s useless.

💡 If your model scores above 95% on the first try, check for leakage. If it scores above 99%, check for class imbalance too.

Clean data is only useful if it actually reaches your model. That’s where pipelines come in.

6. Data Pipelines & ETL

Data pipelines are automated workflows that move data from its raw sources (databases, event logs, third-party APIs) through a series of transformations and into a format the model can use.

ETL stands for Extract, Transform, Load. Extract the raw data from its source.

Transform: clean, join, aggregate, and engineer features from it. Load the results into a feature store or a training dataset.

This is the plumbing part of ML.

And like real plumbing, nobody thinks about it until something breaks. Not the model, not the algorithm. A dropped row, a null value, a late pipeline run. No errors thrown, just silently worse predictions.

Uber runs thousands of data pipelines daily. Everyone has monitoring, alerting, and quality checks. Bring this up in an interview, and you’ll immediately stand out as the guy who actually thinks about how ML works in practice.

Pipelines produce data, but how do you keep track of which data a given model was trained on?

7. Data Versioning

Data versioning means tracking exactly which version of a dataset was used to train each model.

Just like software engineers use Git to track code changes, ML teams need to track data changes.

Why does this matter? Imagine your model’s performance suddenly drops. Without data versioning, you have no way to answer: “What changed? Was it the model or the data?” With versioning, you can compare the current training data with previous versions and pinpoint exactly what changed.

Data versioning also enables reproducibility, a requirement in regulated industries like finance and healthcare.

If an auditor asks, “Why did the model make this decision six months ago?”, you need to be able to recreate the exact model and training data from that point in time.

Tools like DVC (Data Version Control) and MLflow help teams manage this.

In an interview, mentioning data versioning shows you understand that ML systems aren’t just about models. They’re about the entire lifecycle of data and code working together.

With clean, well-structured, and tracked data in hand, it’s time to teach the model how to find patterns in it.

Share this post & get rewards for the referrals.

A model doesn’t “know” anything. It finds patterns in data and uses them to make predictions.

8. Model Training & Loss Functions

Model training is the process of showing the model examples and letting it adjust its internal settings until its predictions get closer to the ground truth.

Think of it as a student taking practice tests over and over, getting a little better each time.

The mechanism that drives improvement is the loss function. A loss function measures how wrong the model’s predictions are. After each batch of examples, the model calculates the loss (how far off it was) and adjusts its internal settings to reduce that number.

Over millions of examples, those small adjustments add up.

Different problems need different loss functions.

Predicting a number (like Uber’s ETA) might use mean squared error, penalizing big misses more than small ones. Classifying something (like spam vs. not spam) typically uses cross-entropy loss, which measures how confident the model was in the wrong answer.

The choice of loss function shapes what the model optimizes for. A model trained with mean squared error will try to be accurate on average. A model trained with mean absolute error will be more robust to outliers. Small decisions here have big downstream effects.

But what exactly is the model adjusting? That’s where parameters come in.

9. Parameters & Hyperparameters

Parameters are what the model actually learns.

Think of them as the model’s memory: every pattern gets encoded as a number. GPT-4 has hundreds of billions. A simple recommendation model, a few million. You don’t write them. The model discovers them on its own.

Hyperparameters are the settings that control how the model trains.

You set them before training starts: how fast the model adjusts (learning rate), how many examples it processes at once (batch size), how many times it loops through the data (epochs), and the model’s architecture.

If the model is a student, parameters are the knowledge in their head, and hyperparameters are the study plan: how many hours a day, which subjects first, and how many practice tests.

Bad hyperparameters can make a good model fail.

A learning rate that’s too high, and the model never converges. Too low, and it takes forever to train. Too few epochs and it underfits. Too many and it overfits. In practice, teams run many training experiments with different hyperparameters and compare results on the validation set.

There’s one type of parameter that powers almost every ML system you’ll encounter in interviews: embeddings.

10. Embeddings

An embedding is a way to represent something (a word, a user, a product) as a list of numbers that captures its meaning.

Instead of treating “cat” and “kitten” as completely different labels, embeddings place them close together in a vectorial space because they’re related concepts.

Think of it as a map. On a regular map, cities that are near each other geographically are plotted close together. In an embedding space, concepts that are similar in meaning are plotted close together. “Comedy” and “rom-com” would be neighbors; “horror” would be distant, but still in the same galaxy as other genres. “Accounting” would be in another universe.

Embeddings are everywhere in real ML systems.

Spotify uses song embeddings to find tracks that are musically similar to what you’ve been listening to. Netflix uses movie embeddings and user embeddings to match viewers with films they’re likely to enjoy.

Embeddings are how ML systems make sense of the real world. A user, a product, a sentence. Turn them into vectors, and suddenly you can calculate which ones are similar, which ones are related, and which ones to recommend.

With all this learning machinery, there’s a critical danger: the model might learn too well.

Reminder: this is a teaser of the subscriber-only newsletter, exclusive to my golden members.

When you upgrade, you’ll get:

High-level architecture of real-world systems.
Deep dive into how popular real-world systems work.
How real-world systems handle scale, reliability, and performance.

Let’s keep going!

11. Overfitting & Regularization

I struggled with mobile system design until I learned these 53 concepts

Shefali Jangid — Thu, 07 May 2026 09:22:29 GMT

Share this post & I'll send you some rewards for the referrals.
Block diagrams created using Eraser.

Onwards ‘n downwards:

Following is the second of a premium 3-part newsletter series…

If you design backend systems and want to avoid building APIs that break under flaky networks, retries, and real-world usage patterns… read this newsletter and start designing for production apps…

On with part 2 of the newsletter:

Less prototype. More production. 🚀 (Partner)

The biggest hurdle for AI Agents in production is trust.

Hard-coded credentials work for a demo, but production agents need a way to handle critical actions—like making a purchase or sending a document—without over-privileged access.

Auth0 for AI Agents features Asynchronous Authorization backchannel authentication (CIBA), so your agents can work autonomously in the background and only trigger a notification when human approval is required.

Rich Consent: Agents render specific authorization data so users know exactly what they are approving.
Secure User Identity: Link agent actions directly to specific user preferences and history.

Move past the “framework” stage and build agentic workflows that are secure by design.

Start building for $0

And stop worrying about DIY-ing auth.

(Thanks to Auth0 for partnering on this post.)

In this newsletter, you’ll learn 20 more concepts:

Conflict Resolution Strategies,
Delta Sync,
Eventual Consistency in Mobile Systems,
Background Sync & Retry Queues,
Optimistic UI Updates,
Authentication (JWT/OAuth),
Secure Token Storage,
Certificate Pinning,
Biometric Authentication,
Code Obfuscation & Tamper Detection,
Secure Deep Linking,
User Data Privacy,
App Lifecycle & State Management,
Background Processing Limits,
Battery Optimisation,
Network Optimisation & Rate Limiting,
Startup Time Optimisation,
Rendering Performance,
Memory Management & Leak Prevention,
App Size Optimisation.

(…and much more in part 3!)

For each concept, I’ll cover:

What it is and how it works
Real-world example
The tradeoffs
Why it matters for mobile

Let’s get into it…

I want to reintroduce Shefali Jangid as a guest author.

She’s a web developer, technical writer, and content creator with a love for frontend architecture and building things that scale.

Check out her work and socials:

You’ll often find her writing about web development, sharing UI tips, and building tools that make developers’ lives easier.

Data Sync

When apps store data on the device, they also need to keep that data in sync with the server.

This section explains how apps synchronize data, handle conflicts when changes occur across multiple places, and maintain app consistency even when the network connection is slow or unreliable.

20. Conflict Resolution Strategies

When mobile apps allow users to edit data offline, a new challenge appears: conflicts.

Why Conflicts Happen

Imagine two users editing the same document:

User A edits the document while offline.
User B edits the same document at the same time.
Later, both devices reconnect and try to sync.

Now, the server has two different versions of the same data. The system must decide how to merge or choose between them.

To solve this problem, apps use various conflict-resolution strategies.

Common Conflict Resolution Strategies

There are several ways apps handle these conflicts:

a. Last-Write-Wins (LWW)

This is the simplest approach.

Each update includes a “timestamp”. When two versions conflict, the system keeps the latest update and discards the older one.

For example:

User A updates at 10:01 p.m.
User B updates at 10:02 p.m.

Since User B’s change happened later, the system keeps that version.

This approach is simple to implement and works well for simple data, but earlier changes may be lost.

b. Versioning

Versioning tracks different versions of the same data.

Each update increases the version number or records which device made the change. When a conflict occurs, the system can compare the versions and either automatically merge them or ask the user to choose.

This approach is more flexible, but requires extra logic to manage versions.

c. CRDTs (Conflict-Free Replicated Data Types)

CRDTs are special data structures designed to merge changes automatically without conflicts.

They use mathematical rules that ensure that different versions of the same data can always be merged into a single, consistent result, even when updates occur in different places.

This approach is often used in real-time collaboration tools.

Why It Matters

Without a proper conflict strategy, one user’s changes might silently overwrite another user’s work.

For example, someone may edit a document, sync it later, and suddenly see that their changes have disappeared.

This can break user trust and cause data loss.

Real-World Example

Many well-known applications use these strategies:

Figma and Notion use “CRDT” based systems to support real-time collaborative editing.
Dropbox often uses a “last-write-wins” approach and may generate a “conflicted copy” when it cannot merge two versions.

Trade-offs

Each approach has its advantages and limitations:

Last-Write-Wins: Very simple to implement, but can cause data loss because earlier changes may be overwritten.
CRDTs: Automatically merge changes without conflicts, but are complex to implement and not suitable for every type of data.
Versioning: Keeps track of multiple changes clearly, but may require users to manually resolve conflicts.

Once conflicts can be resolved safely, syncing becomes more efficient when only changes are transferred.

Reminder: this is a teaser of the subscriber-only newsletter, exclusive to my golden members.

When you upgrade, you’ll get:

High-level architecture of real-world systems.
Deep dive into how popular real-world systems work.
How real-world systems handle scale, reliability, and performance.

And much more!

21. Delta Sync

When mobile apps sync data with a server, downloading the entire dataset every time can be slow and waste a lot of data. This is especially true for apps that store large amounts of information.

To make syncing more efficient, many apps use a technique called delta sync.

Delta sync means the app downloads only the data that has changed since the last sync, rather than downloading everything again.

How It Works

When the app syncs for the first time, it downloads the full dataset from the server.

After that, the server gives the app a sync token or timestamp that represents the latest state of the data.

On the next sync:

App sends the last sync token to the server.
The server checks what has changed since that token.
Server sends only new, updated, or deleted records.

This makes syncing much faster and more efficient.

Why It Matters

Imagine a contacts app with 5,000 contacts.

If the app downloaded all contacts every time it synced, it would waste a large amount of bandwidth and battery. With delta sync, the app downloads only the contacts that were added or changed, saving both data and processing time.

This makes background syncing practical even on mobile networks.

Real-World Example

Many well-known platforms use delta syncing:

Google Contacts uses a “syncToken” to track changes.
Apple CloudKit uses a “serverChangeToken” for the same purpose.

The app stores this token and sends it during the next sync request, so the server knows exactly what updates to return.

Trade-offs

Delta sync improves efficiency but requires extra work on the server.

The server must keep track of which data has changed. This usually means storing timestamps or maintaining a change log.

Deletions must also be tracked. Instead of completely removing a record, the system may temporarily store a “tombstone” record that indicates the item was deleted, so the client knows to remove it as well.

Practical Tip

The sync system should rely on timestamps or sequence numbers generated by the server, not the client.

Client devices may have incorrect clocks, which can cause sync problems. Using a server-controlled timestamp or sequence number ensures that the order of updates is always correct.

Even with efficient syncing methods like delta sync, distributed systems can still struggle to maintain perfect consistency across devices.

22. Eventual Consistency

Mobile apps often run in environments with slow, unstable, or completely unavailable network connections. Because of this, different devices may not always have the exact same data at the same time.

This situation is called eventual consistency.

Eventual consistency means that data may temporarily differ across devices, but all devices will eventually reach the same state once they sync with the server.

How It Works

In mobile apps, users may perform actions while offline or on unstable networks.

For example:

A user updates some data on their phone.
The change gets saved locally on the device.
When the network connection becomes available, the app sends the update to the server.
Server then sends the updated data to other devices.

For a short period, different devices may show slightly different versions of the data, but eventually they all become consistent after syncing.

Why It Matters

Trying to keep data perfectly synchronized at all times would require the app to wait for the server to confirm every action before updating the interface.

This would make the app feel slow and unresponsive.

Instead, mobile apps usually update the UI immediately, while ‌server synchronization happens in the background.

This approach makes the app feel fast and responsive while still keeping the data consistent over time.

Real-World Example

Messaging apps are a common example of eventual consistency.

For example, WhatsApp shows different message states:

A single check mark means the message has been sent from the device.
A double check mark means the message has been delivered to the recipient.

The message appears in the chat immediately, even before the server fully confirms it. Later, the status updates when the system finishes syncing.

Trade-offs

Eventual consistency improves user experience, but it also introduces some challenges…

Sometimes the server may reject a change or update the data differently. When this happens, the app must update the UI carefully to avoid confusing the user.

To handle this, many apps display clear states such as pending, confirmed, or failed. This helps users understand what is happening with their actions.

To handle temporary failures during syncing, apps often rely on background processes that automatically retry operations, as discussed in the next section.

23. Background Sync & Retry Queues

Mobile apps often perform actions that need to be sent to a server, such as sending a message, liking a post, or submitting a form. But sometimes the network connection may be weak or unavailable, causing these actions to fail.

Instead of losing these actions, apps use a system called a retry queue.

A retry queue stores failed operations locally on the device so they can be retried later when the network connection is available.

Mobile operating systems also provide tools that allow apps to perform tasks in the background efficiently.

For example:

WorkManager on Android
BGTaskScheduler on iOS

These tools allow apps to run background tasks while managing battery usage and system resources.

How It Works

When a user performs an action that requires a network request, the app tries to send it to the server:

User performs an action (for example, sending a message).
If the request succeeds, the server processes it normally.
If the request fails because of network issues, the app stores the action in a retry queue.
When the device reconnects to the network, a background sync process retries the queued operations.
Once the server confirms the request, the action is removed from the queue.

This ensures that user actions are not lost even when the network is unreliable.

Why It Matters

Users expect their actions to work, even if the network connection is unstable.

For example, if a user taps “send message” or “like”, the app should not simply ignore the action because the network was unavailable.

By storing the action and retrying it later, the app respects the user’s intent and completes the action when conditions improve.

Real-World Example

Many apps use retry queues and background syncing:

Email apps like Gmail and Outlook allow users to send emails while offline. The email is queued and automatically sent when the network returns.
E-commerce apps may queue actions such as adding items to a cart or saving user activity and sync them later.

Trade-offs

Background processing on mobile devices has limitations.

Operating systems control how long apps can run in the background to protect battery life. These background windows are usually short and unpredictable. Because of this, apps should not rely only on background syncing. Synchronization should also happen when the user opens the app again.

These mechanisms also allow the app’s interface to respond instantly, while the actual synchronization happens quietly in the background.

24. Optimistic UI Updates

In many apps, users expect actions to happen instantly. For example, when you tap “like”, send a message, or add a comment, you expect to see the result immediately.

If the app waited for the server to respond before updating the screen, it could take a few hundred milliseconds. Even that slight delay can make the app feel slow.

To solve this, apps use a technique called optimistic UI updates.

Optimistic UI means the app assumes the request will succeed and updates the interface immediately, even before the server confirms it.

The server request still happens in the background. If everything succeeds, nothing changes. If it fails, the app corrects the UI and shows an error.

How It Works

The process usually follows these steps:

User performs an action (e.g., tapping a like).
App updates the UI immediately and stores the change locally.
App sends the request to the server.
If the server confirms the request, the change remains unchanged.
If the server rejects the request, the app reverts the change and shows a small error message.

This makes the app feel fast while still keeping the data accurate.

Why It Matters

Users expect apps to respond instantly to their actions.

If the app shows a loading spinner every time the user taps a button, the experience feels slow and frustrating.

Optimistic UI removes this delay, making the app feel smooth and responsive, similar to native device interactions.

Real-World Example

Many popular apps rely heavily on optimistic UI:

Twitter shows the like animation immediately when you tap the heart.
Instagram displays your comment instantly in the thread.
Slack shows a message in the chat immediately after you send it.

In the background, the app syncs with the server and updates the data if needed.

Trade-offs

An optimistic UI improves the user experience, but it also adds complexity.

If the server rejects the request, the app must roll back the UI change smoothly without confusing the user. Because of this, optimistic UI is usually avoided for actions that cannot be easily reversed, such as permanent deletion, payments, and financial transactions.

As app stores and sync sensitive user data, security becomes a critical part of system design…

Let’s keep going!

Design a personal AI chat assistant

Neo Kim — Mon, 04 May 2026 07:49:25 GMT

Share this post & I'll send you some rewards for the referrals.

ChatGPT, Claude, and Gemini handle general-purpose tasks and one-off conversations well…

The moment your assistant is customer-facing, the requirements change:

Your users need an assistant that knows your product, your policies, and their history with you. That knowledge lives in your systems, NOT in a model’s training data. You could try to inject it at runtime, but proprietary data comes with real constraints: it is often too large to fit in a context window, too sensitive to send to a third-party API, and too dynamic to stay current in a static prompt.

Beyond data, you cannot fully configure its behavior; you cannot integrate it past a browser tab, and costs scale with every conversation…

Here is what changes when you build your own:

Privacy and data control. When you build your own assistant, conversations stay on your infrastructure. You decide what gets logged, how long it gets stored, and when it gets deleted. For healthcare, legal, finance, and enterprise use cases, this is a compliance requirement, not a nice-to-have.
Full control over behavior. You own the system prompt, persona, and the guardrails. You can enforce a specific tone, restrict responses to your product domain, swap the underlying model, or A/B test different configurations. With an off-the-shelf product, you get what they ship.
Cost at scale. You can implement prompt caching, model routing (a cheap model for simple questions, an expensive one for hard ones), and context management strategies that dramatically cut costs. With thousands of conversations per day, the savings are significant.
Product integration. Your assistant lives inside your product, not in a separate tab. It shares your auth system, your UI, and your user context. That level of integration is not possible when you are wrapping someone else’s chat interface.
Security. You control the full request path: what goes into the model, what comes out, and what filters run in between. You implement your own prompt injection defenses, output guardrails, and content policies.

Off-the-shelf tools are great for personal use and internal prototyping. Building your own is for when the assistant is the product or a core feature.

Enterprise auth from day one. No upgrade required. 🚀 (Partner)

Enterprise auth is intimidating without a doubt - SSO, SCIM, etc.

You may feel these features are next year’s problem because the price jump to unlock basic SAML or OIDC connections is just too brutal for an early-stage roadmap.

With Auth0, ship your first Enterprise Connection and SCIM setup for $0 on their free tier. If you’re scaling AI Agents, security stack is built for production:

Token Vault: Securely connect 3rd-party tokens (Slack, GitHub) to stop hard-coding API keys.
FGA for RAG: Ensure LLMs respect fine-grained permissions to prevent data leaks and also control access to specific AI tools/models.
Async Authorization: Use backchannel authentication for human-in-the-loop approvals on critical agent actions.
User Authentication: Securely identify users to unlock chat history, order tracking, and personalized settings within your AI agents.

Start building for $0

And stop worrying about DIY-ing auth.

(Thanks to Auth0 for partnering on this post.)

I want to reintroduce Louis-François Bouchard as the author of this newsletter.

He’s a best-selling author (Building LLMs for Production), the co-founder of Towards AI, and the creator of the YouTube Channel, What’s AI, where he helps people understand AI and learn how to apply it in the real world. Through his development work with clients and his content, teaching, and AI training programs on the Towards AI Academy, Louis focuses on making AI practical for builders, engineers, and curious learners alike.

Inside this newsletter, you’ll get:

What happens when you send a message. Tokenization, prefill, and decode phases, KV-cache, and why longer conversations get slower and more expensive.
How the model got here. Pretraining, post-training (SFT and RLHF), and scaling laws, and what each one means for the assistant you are building.
Output controls. Temperature, top-p, top-k, why hallucination happens, and which settings to use for which task.
Context engineering. System prompts, few-shot, chain-of-thought, structured outputs, context window management, and persistent memory across sessions.
Cost and how to know it works. Worked token math across model tiers, prompt caching, model routing, golden test sets, and LLM-as-Judge evaluation.
A practical eight-step build. From a single API call to a production-minded assistant with streaming, system prompts, JSON output, multi-turn compaction, prompt injection defense, and persistent memory.

What We Are Building

We’re building a ChatGPT-style conversational assistant that you fully control: your own system prompt, your own personality, your own rules.

Full system architecture of a chat assistant. A user message flows through four layers: context engineering, generation engine, persistent memory, and SSE streaming before a response streams back. Each section of the article zooms into one layer.

Here are the requirements:

Multi-turn conversation with context preserved across turns.
Streaming responses, where words appear as they’re generated.
Configurable behavior, creative for brainstorming, precise for data extraction.
Resistant to prompt injection attacks.
Cost-effective at scale, with thousands of conversations per day.

One extra feature covered later: persistent memory across sessions, so the assistant remembers user preferences between conversations.

What this system does not do: it does NOT retrieve external documents, it does NOT have domain expertise beyond its training data, and it does NOT take actions or call tools. It’s purely conversational.

We use a model API (OpenAI, Anthropic, or similar) for this build.

The provider handles GPUs, scaling, and inference optimization, the right starting point for most teams. When you outgrow APIs, you move to self-hosted open-source models served with vLLM, llama.cpp, or SGLang.

This newsletter is organized around 4 core design decisions:

What happens when you send a message?
How do you control what the model says?
How do you engineer the context for every request?
What does it cost, and how do you know it works?

(Then we build the whole thing from scratch in Part 5.)

Let’s start with what happens under the hood when you send a message…

Part 1: What Happens When You Send a Message

When you call a model API, a lot happens between your request and the first token appearing on screen.

Even if you never need to use a GPU, this helps you understand the latency, cost, and trade-offs…

From Text to Tokens

Before the model can process your message, it needs to convert it from text into numbers.

Models do not read words. They work with tokens, small chunks of text mapped to numerical IDs. A token might be a whole word (“the”), part of a word (“un”, “believ”, “able”), or a single character.

The process of splitting text into tokens is called tokenization.

As we covered in Part 1, there are different ways to split text into tokens:

You could split by individual characters, but then the model has to process very long sequences just to read a short sentence. You could split by whole words, but then the vocabulary becomes enormous, and the model cannot handle any word it has not seen before.

Subword tokenization is the sweet spot, and virtually all modern models use it.

The most common algorithm is Byte Pair Encoding (BPE). BPE starts with individual characters and iteratively merges the most frequent pairs into a single token. Common words like “the” or “and” become single tokens. Rare or long words get split into meaningful pieces. The word “unbelievable” might be broken into three tokens: “un” + “believ” + “able.” This keeps the vocabulary manageable, typically 30,000 to 100,000 tokens. And if the model encounters a word it has never seen before, it can still process it by breaking it into known subword pieces.

This is why language models struggle with certain tasks that seem trivial to humans…

How tokenization works. The sentence “The cat sat on the mat” becomes six tokens, each mapped to a numerical ID. Below, “strawberry” is split into three tokens (”str”, “aw”, “berry”), showing why the model cannot count the letter r: the r’s are split across token boundaries.

Ask “How many r’s are in strawberry?” and the model might get it wrong because it never sees the individual letters. The word “strawberry” might be tokenized as “str” + “aw” + “berry,” so the model literally cannot count the r’s because they are split across token boundaries. Reversing a word, counting specific characters, and other character-level operations are all hard for the same reason: the model operates on tokens, not letters.

Tokenization has direct practical consequences for building a chat assistant.

Token count does not equal word count: a typical English sentence of 10 words might be 13 to 15 tokens. Code is usually more token-dense than prose. Non-English languages often require more tokens per word because BPE vocabularies are built primarily from English text. Since API pricing, context window limits, and latency all scale with token count, understanding this conversion is essential for cost estimation and context management, both of which we cover later in this newsletter.

Reminder: this is a teaser of the subscriber-only newsletter series, exclusive to my golden members.

When you upgrade, you’ll get:

Simple breakdown of real-world architectures
Frameworks you can plug into your work or business
Proven systems behind ChatGPT, Perplexity, and Copilot

How the Model Got Here: From Pretraining to Chat

Before we look at what happens on each request, it helps to understand what happened before you ever sent a message…

The model you are calling went through a multi-stage pipeline to become a chat assistant, and each stage has direct consequences for the product you are building.

Pretraining is where the model learns language.

As covered in Part 1, the model is trained on a massive corpus, books, websites, code, research papers, and learns to predict the next token given all the tokens before it. This is self-supervised: no one labels the data. The model just reads trillions of tokens and learns patterns. Pretraining is what gives the model its general knowledge, its grammar, its ability to write code, and its tendency to produce plausible-sounding text, whether or not it is true.

When your chat assistant confidently states a wrong fact, that behavior traces back to pretraining: the model learned to produce likely continuations, not verified truths.

Post-training is what turns a text predictor into a chat assistant.

The pretrained model is good at completing text, but it does not know how to follow instructions, hold a conversation, or refuse harmful requests. Post-training fixes this in stages.

First, supervised fine-tuning (SFT) trains the model on curated examples of good conversations: a user asks a question, and a human-written response shows what the ideal answer looks like. This teaches the model the format and style of a helpful assistant.

Then, reinforcement learning from human feedback (RLHF) refines the model further. Human raters compare pairs of model responses and pick the better one. The model learns from those preferences, becoming more helpful, more accurate, and less likely to produce harmful or unhelpful output.

This is why the same base model can feel completely different depending on how it was post-trained. It is also why system prompts work: the model was specifically trained to follow developer-level instructions during post-training.

Scaling laws govern the relationship between model size, training data, and performance.

Research (notably Kaplan et al. at OpenAI and the Chinchilla paper from DeepMind) showed model performance improves predictably as you increase the number of parameters and training tokens, following power-law curves.

This is why the industry keeps building bigger models: doubling model size produces measurable, predictable gains in capability. For you as a builder, scaling laws explain the cost-capability tradeoff you face.

A frontier model (hundreds of billions of parameters) costs more per token but handles harder tasks. A lightweight model (fewer parameters, less training) is cheaper and faster but less capable. The tiered pricing you see from API providers, frontier versus mini versus nano, maps directly to where each model sits on the scaling curve.

Understanding this helps you make the model routing decisions we discuss later: use the big model when you need its capabilities, and the small model when you do not.

Let’s keep going!

Multi-Agent Architectures, Clearly Explained

Neo Kim — Thu, 30 Apr 2026 07:57:47 GMT

Share this post & I'll send you some rewards for the referrals.

A single agent1 has one context window2, one set of tools3, and one running loop4.

When a task outgrows any of those three, you need more than one agent. That’s what a multi-agent system is. Instead of one agent doing everything, you split the work across many agents. Each one has its own role, tools, and context.

Most multi-agent systems fail not because the model is weak but because you chose many agents before you actually need them, or chose the wrong architecture once you have many agents.

So you shouldn’t split anything yet.

Onward.

[Webinar] Stop babysitting your coding agents (Partner)

Agents can generate code. Getting it right for your system, team conventions, and past decisions is the hard part – you end up wasting time and tokens in correction loops.

More MCPs give agents access to information, but not understanding. The teams pulling ahead use a context engine to give agents exactly what they need.

Join Unblocked live on May 6 (FREE) to see:

Where teams get stuck on the AI maturity curve
How a context engine solves for quality, efficiency, and cost
Live demo: the same coding task with and without a context engine

(Thanks to Unblocked for partnering on this post.)

Here’s what’s inside:

Why single agents break. Context overflow, slow serial work, and why one agent can’t always hold every tool, model, or permission.
The six architectures. Orchestrator-worker, pipeline, hierarchical, swarm, mesh, and handoffs. Plus, where each one works and breaks.
How agents coordinate. Run loops, MCP vs A2A, shared state, isolated state, memory, and stopping conditions.
The real cost. Why more agents mean more tokens, more latency, more coordination overhead, and more ways to fail.
Failure and security risks. Bad instructions, misalignment, weak verification, prompt injection, context contamination, and privilege creep.
Case study. How Spotify used an orchestrator-worker system to turn ad planning from a 15-minute workflow into 5 seconds.

Limits of single-agent systems

One agent with good prompts and the right tools handles more than most people expect.

For example, Cognition’s Devin processed 5 million lines of COBOL (Common Business-Oriented Language) across 500GB of repositories with a single agent, raising its pull request merge rate from 34% to 67%.

But a single agent has three HARD limits. When your task runs into any of them, better prompts won’t help:

1. Context overflow

A context window can only hold so much.

Past that limit, the earliest information drops out, and the agent starts losing track of its own plan. When compression5 alone can’t fix the overflow, a second agent with its own context can.

2. Parallelism

Independent tasks shouldn’t wait in line.

If you have four research queries that don’t depend on each other, running them one at a time wastes time. Running them across four separate agents takes roughly as long as the slowest one.

Anthropic’s research system uses this exact pattern and reduced total query time by up to 90%.

3. Specialization

Different parts of a task often need different models6, tools, or access levels:

A code-writing agent needs a sandbox
A research agent needs web search
A customer-facing agent needs user data, but shouldn’t have access to production databases

When one agent can’t hold all the tools and permissions the task needs, you give each role its own agent.

If none of those three conditions apply, stay with one agent.

Better prompts and better tools solve most problems without adding the extra work of coordinating agents. But once you know you need many agents, the next question is what shape the system takes...

Multi-agent architectures

Every multi-agent system makes a different choice about who coordinates work.

Here are 6 architectures that range from tight central control to NO coordinator at all:

1. Orchestrator-worker

One central agent breaks a task into pieces, assigns each piece to a worker agent, and then puts the results together:

Workers don’t talk to each other; all communication goes through a central agent
Orchestrator calls each worker as a tool call7, waits for a result, and decides what to do next

This is like an air traffic control tower that talks to every plane, while no plane talks directly to another.

Anthropic’s Claude Research system works this way:

A central agent running Opus 4 breaks a research query into parts and creates 2 to 10 worker agents on Sonnet 4 (sometimes more), all at the same time. The workers search the web, read documents, and gather evidence in parallel. When they finish, the central agent reads their results and writes a single research report.

This setup beat single-agent Opus 4 by 90.2% on Anthropic’s internal research eval8.

Tradeoffs

Central agent is both coordinator and bottleneck…

It talks to workers one at a time. If each call takes 3 seconds and 20 workers are waiting, the ceiling is about 7 tasks per second. So the central agent becomes the slowest part of the system.

Anthropic’s Claude Research system had this problem as well: workers duplicated each other’s work.

Without specific instructions, many workers run overlapping searches on the same topic, wasting both tokens9 and time. The takeaway is orchestrator-worker depends entirely on the quality of the lead agent’s instructions.

Vague task splitting turns parallelism into duplicated work.

2. Pipeline

Agents run in a fixed order, one after another.

Each agent’s output becomes the next agent’s input, and entire sequence is set in advance. While orchestrator-worker lets the central agent decide what to do as it goes, pipeline removes that choice.

An assembly line works the same way: each station does one job, passes the result forward, and never sees the finished product.

Stripe uses this pattern to check whether new businesses on its platform are legitimate.

Before agents, a human reviewer had to jump between customer databases, legal sources, and support tickets to decide whether a business was safe to approve. Now their engineering team broke that work into a fixed flow of agent stages using a directed acyclic graph (DAG). So work moves forward through stages without looping back.

Order is set at design time, and each stage has a contract: defined output format next stage expects.

Stripe calls these contracts “rails” because they keep any single agent from spending too much time on irrelevant data. This setup cut their average handling time by 26%, and reviewers rated the agent outputs 96% helpful, with a full record of every decision at every step.

Tradeoffs

Latency adds up.

A 5-stage pipeline where each stage takes 2 seconds means a 10-second wait before any output, and adding a stage to improve quality increases the response time.

Yet the upside is predictability.

When every stage has a narrow contract, you can trace any failure back to exactly one step. That’s why regulated workflows like Stripe’s continue to use pipelines.

When the cost of being wrong is a regulator flagging your process, the extra seconds are a small price to pay.

3. Hierarchical

A top-level manager agent gives work to one or more layers of manager agents below it, which then give work to individual workers.

Two levels are the minimum; big systems stack more. The result is a tree…

No single agent needs the full context.

The top-level agent holds the high-level goal and a summary from each branch, while each lower level sees only what its narrower role needs. Picture a military chain of command where orders flow down, reports flow up, and no one skips a level.

IBM watsonx Orchestrate runs on this pattern:

A top-level supervisor agent acts as a router and planner across 80+ pre-built domain agents for HR, sales, and procurement. Let’s say a user tries to “order new laptops for the design team”. The request reaches a Procure Equipment supervisor, who then hands the work to three specialized child agents:

One requests quotes from approved vendors
Another checks responses
A third submits a purchase request

The supervisor decides only who gets called and in which order.

Tradeoffs

Details might get lost at each level.

A worker produces a detailed result. The mid-level manager shortens it to one sentence. By the time it reaches the top, the detail that matters might be gone. Hierarchical structures trade detail for coverage: the higher you go, the wider the scope of each agent and the less it knows about any specific piece.

These three architectures share one thing: a clear chain of command tells you exactly who’s in charge…

The next three architectures drop the chain entirely. So they're harder to debug, but they survive partial failures better…

Reminder: this is a teaser of the subscriber-only newsletter, exclusive to my golden members.

When you upgrade, you’ll get:

High-level architecture of real-world systems.
Deep dive into how popular real-world systems work.
How real-world systems handle scale, reliability, and performance.

4. Swarm

In a swarm, many agents work as equals.

They coordinate through a shared blackboard: a data store (typically Redis cache, database table, or vector store) that every agent can read from and write to. Yet there are NO direct messages between agents…

29 LLM Evaluation Concepts Every Engineer Needs to Know

Anshuman Mishra — Mon, 27 Apr 2026 11:50:18 GMT

Share this post & I'll send you some rewards for the referrals.

You ship an LLM feature.

It passes your manual tests. A day later, a user posts a screenshot of it hallucinating wildly! You tweak the prompt, run it again, and it works fine. Did you fix it, or did you just get lucky?

Welcome to the central frustration of LLM engineering: you can NO longer just run a test and call it done.

This isn’t a debugging problem…

It’s a measurement problem. And measurement has a name: evaluation.

Most articles on LLM evaluation are written for ML researchers. This one is for engineers building real applications. You know how to ship software. You’re just new to the ways LLMs fail.

We’ll cover the vocabulary, methods, and mental models. By the end, you’ll have a framework for building an eval system from scratch. Not just an understanding of why it matters.

Let’s start with why your existing testing instincts don’t work here…

Your team’s second brain. Now in Slack. (Partner)

Your engineers talk on Slack. They code in the terminal. Somewhere between those two things, context goes to die.

A bug was debated in #incidents at 2 AM.
An architectural call was made in a DM.

Every handoff leaks context, and every leak costs you. That’s the context tax - and your team pays it every day.

CodeRabbit Agent for Slack is built for agentic SDLC workflows. One agent for your entire Software Development Lifecycle, living in the channel where the work already happens. It’s built on four things:

Context - your org’s operating picture, pulled from across code, tickets, docs, monitoring and cloud.
Knowledge Base - a living memory of your team. Every run leaves a trace, so yesterday’s decisions don’t become tomorrow’s debates.
Multi-Player - works in shared threads alongside your team. Steerable, resumable and aligned as work evolves.
Governance - scoped access, cost attribution. Every run explainable and attributed.

Your team keeps shipping. Agent keeps the context.

From the team that pioneered AI code reviews. 2M code reviews every week. 6M repos. 15K customers. And now, one agent for your entire SDLC, right in Slack.

Get 20% off a group subscription

(Thanks to CodeRabbit for partnering on this post.)

I want to introduce Anshuman as a guest author.

He leads evals efforts at Zomato, where he built Gavel -- an internal LLM eval platform that started as a handful of scripts, got pitched to a VP, and now serves AI and ops teams across the company. Now he’s on the ground floor of making a large organization AI-native, one eval system at a time.

I highly recommend you checkout his newsletter, AI Proof -- it’ll help you stay relevant in the AI era.

Before we get into solutions, you need to understand something…

LLM evaluation feels nothing like regular software testing. It’s not that it’s harder. It’s that the rules changed.

Let’s dive in!

1. Non-determinism Problem

Write a function.

Call it with the same input twice, and you’ll get the same output twice.

But LLMs don’t work that way.

The same prompt can produce a different response on every run.

Sometimes slightly different. Sometimes completely different. This isn’t a bug. It’s called temperature.

It controls how random the outputs are. (More on this later.)

This breaks something deeper. Your entire mental model for testing assumes determinism. You run a test. It passes or fails. You know what you know.

With LLMs, a passing test is a data point, not a verdict. You’re not testing a function. Instead, you’re sampling from a probability distribution. One sample tells you almost nothing.

This is the first reason your existing instincts mislead you.

2. Fuzzy Correctness Problem

When a regex matches, it matches. Binary. Objective. Done.

But what’s the correct response to “summarize this support ticket empathetically?”

Is the correct answer the one that’s most concise?
The one that uses the warmest language?
The one that captures all the key facts?
The one a human reviewer would score highest?

LLM quality is multi-dimensional and subjective.

There usually isn’t one correct answer. There’s a range of acceptable answers and a range of bad ones. The line between them is a judgment call.

You can’t measure what you haven’t defined.

What does ‘good’ mean for your use case? You need a clear answer to that before you can evaluate anything. Most people skip this step. It comes back to haunt them.

3. Silent Regression Problem

You updated your prompt.

Ran it a few times. The outputs seemed better. You shipped it.

But did quality actually improve, or did you get lucky with the samples you checked?

This is the silent regression problem. Without a systematic evaluation process, every prompt change is a blind bet. You might be making things better. You might be fixing one failure mode while introducing another. And you have no way of knowing.

In traditional software, CI (Continuous Integration) catches regressions before they reach users. In LLM engineering, there’s NO such equivalent. So people rely on their gut feelings and manual spot checks. You find out about regressions when a user complains!

These three problems make LLM evaluation its own discipline…

Now let’s build the vocabulary to actually talk about it…

Share this post & get rewards for the referrals.

Primitives of Eval

These are six terms you’ll hear constantly.

Learn these, and the rest falls into place:

4. Criteria

Criteria are the dimensions of quality that matter for your specific use case.

A customer support bot might care about:

Does it address the user’s actual issue?
Is the tone empathetic?
Does it avoid triggering escalation unnecessarily?

A code generation tool cares about something completely different:

Is the output syntactically valid?
Does it follow the project’s conventions?
Does it actually solve the stated problem?

Same technology, but entirely different criteria.

This is a product decision, NOT a technical one. You can’t outsource it to your model or your eval framework. Someone on your team needs to sit down and answer: What does a good output actually look like here?

Get this wrong, and everything downstream is measuring the wrong thing…

5. Quality Dimensions

Quality dimensions are the standard industry vocabulary for LLM output quality.

Here are five of them that come up constantly:

Relevance: Did the output address what was actually asked? A response can be accurate, well-written, and completely beside the point.
Coherence: Does it hold together logically? No contradictions. No mid-sentence topic shifts. Flows like something a thoughtful person wrote.
Factual Accuracy: Is what it says actually true? Distinct from relevance. A response can be relevant and wrong.
Helpfulness: Does it give the user what they need to move forward? The difference between a technically correct answer and a useful one.
Safety: Does it avoid harmful, biased, or inappropriate content? This dimension matters more in some domains than others -- but it always matters.

Knowing these helps you write rubrics that don’t miss important failure modes.

6. Rubric

Once you have the criteria, you need to operationalize them.

That’s what a rubric does.

Take a vague criterion like “helpfulness.” A rubric breaks it into specific, scorable questions. Does it directly answer the question? Does it avoid unnecessary hedging? Is it under 200 words? Can a non-technical user understand it?

Think of it like a code review checklist.

Instead of “Is this good code?”, a checklist asks: Are there tests? Is the function under 30 lines? Are variable names descriptive? Rubrics do the same thing for LLM outputs.

The rubric is what makes evaluation reproducible.

Two different reviewers, human or AI, should arrive at similar scores. Same output, same rubric, same conclusion.

Without a rubric, every evaluation is just someone’s opinion.

7. Test Cases

A test case is an input/output pair that forms one unit of your evaluation.

Input is a prompt: ideally, one representative of real user traffic.

Output can be one of two things. A reference answer showing what good output looks like. Or the live model output you’re about to score.

Think of test cases like unit tests…

Except that a failing test case doesn’t mean the output was wrong. It means it scored below your defined threshold on your rubric.

That distinction matters.

You need a lot of them. A handful of test cases gives you anecdotes. A few hundred gives you a signal.

8. Golden Set

Everything in your eval gets measured against one thing: your golden set.

A curated collection of high-quality test cases.

Building a good golden set is harder than it sounds:

The instinct is to write examples yourself, covering the use cases you anticipate. That’s a start. But users phrase things differently than you expect. They hit edge cases you didn’t imagine.

And they might misuse features in creative ways.

A golden set built from your imagination reflects your imagination and not your users’. So seed it with real production queries instead. Anonymized, cleaned, representative.

Your golden set is your ground truth.

Treat it like a critical system artifact. Version and update it when you discover new failure modes.

9. Pass/Fail Threshold

Eval scores are rarely binary.

A rubric usually produces a score of 1 to 5, 0 to 10, or a percentage. The pass/fail threshold is what converts that score into a decision.

If your rubric scores range from 1 to 5 and your threshold is 3, any score below 3 is a failure. Simple in theory, but HARD in practice.

Setting the right threshold is a product call, not a technical one.

It depends on a few things: how much imperfection your users can tolerate. How severe are failures in your domain? The cost of false positives.

Set your threshold too low, and you’re shipping garbage. And if you set it too high, you’re shipping nothing.

10. Eval Coverage

Eval coverage is how well your golden set reflects real user inputs.

Most teams have low coverage and don’t know it. They built their golden set from examples they had written. Happy path, a few obvious edge cases. Meanwhile, production traffic is different: weird inputs, unusual phrasings, use cases nobody anticipated.

Low coverage means your eval suite is optimistic…

i.e., you’ll pass your tests but fail your users.

The fix isn’t writing more examples yourself. Instead, sample from production regularly and review failures. Then add the inputs that exposed new weaknesses to your golden set.

Eval coverage is something you build over time.

11. Temperature, Top-p, and Reproducibility

Temperature is the dial that controls how random your model’s outputs are.

Low temperature (close to 0) makes the model nearly deterministic: same input, same output, every time. High temperature makes it more creative. It samples from a wider range of probable tokens, producing more varied responses.

Top-p (nucleus sampling) is a related setting.

Instead of a randomness dial, it sets a probability cutoff. Only the most likely tokens make the cut. Top-p = 0.9 means the model considers only the top 90% most likely next tokens.

Both settings directly affect your eval results…

Run evals at temperature = 1.0 and the same prompt might pass today and fail tomorrow. NOT because your model changed, but because randomness swung against you.

Here’s the standard practice: set temperature to 0 during eval runs. And lock in determinism. If you need creative variance in production, test at your production temperature.

Just know you’re accepting noisier results…

12. Statistical Rigor

Even at temperature = 0, a single eval run isn’t enough.

Your golden set is a sample of your input space. That sample has its own variance. One unlucky set of examples can make a good prompt look bad. And one lucky set can make a bad prompt look good.

So run many evaluations across different samples.

Then report the mean and the variance, not just the score. When comparing two prompt versions, check whether the difference is real. Or if it’s just noise.

In practice: if you change a prompt and your score goes from 4.1 to 4.3, that might be a real improvement.

It might be a random fluctuation. Without variance1 data across many runs, you can’t tell the difference. Most teams run once, report the number, and ship. That’s how confident regressions get deployed…

Share this post & get rewards for the referrals.

How Do You Score Outputs?

You have criteria, a rubric, and test cases2.

Now the question is: who does the scoring? There are three options.

Each has different tradeoffs in cost, speed, and accuracy.

13. Human Evaluation

This is the gold standard.

A human reviews the output using your rubric and gives a score. It’s slow, expensive, and can be inconsistent. But it’s the closest thing to ground truth.

You can’t use humans for every eval run; it doesn’t scale.

But you shouldn’t remove human evaluation completely either. It’s what keeps your system grounded. Everything else: metrics, LLM judges, tests, is just an approximation of human judgment.

So use human evaluation strategically:

To build and validate your golden dataset.
To periodically check the accuracy of your automated evals.
To debug when something breaks and you don’t know why.

This way, you balance cost with reliability.

14. Heuristic/Code Based Evaluation

This is the fastest and cheapest type of evaluation.

It uses simple code checks to validate the output's structural properties. For example:

Is the response valid JSON?
Is it within the character limit?
Does it include all required fields?
Does it avoid banned phrases?
Does it match a specific format (like a regex)?

Heuristic eval is good at catching structural problems.

But they don’t measure quality. They can’t tell if a response is helpful, accurate, or well-written.

Think of heuristics as your first line of defense.

They catch basic issues before you run more expensive evaluations. They’re NOT enough on their own, but they’re an essential part of a complete eval system…

15. Semantic Similarity Evaluation

Sometimes you have a reference answer, a known-good response representing ideal output.

Semantic similarity evaluation measures how close your model’s output is to the reference. i.e., in meaning, and not exact wording.

This is where embeddings come in…

Each piece of text gets converted into a vector: a list of numbers that represents its meaning. Texts with similar meanings have vectors that are close to each other. Cosine similarity measures the angle between two vectors. A score of 1.0 means identical meaning. A score near 0 means unrelated.

This matters because string matching is too strict.

Take two sentences: “API returns a 404 error” and “Endpoint responds with a not found status.” They mean the same thing, but use different words. An exact match would call the second one wrong. Semantic similarity would call them equivalent.

But here’s the limitation: it only measures closeness to your reference.

It can’t catch a response that’s fluent and factually wrong. Not if the wrong answer happens to be semantically similar to the right one. So use it as a fast, scalable layer. And don’t rely on it alone!

16. Task Specific Metrics (BLEU, ROUGE, Execution-based)

For certain tasks, there are established metrics purpose-built for automated evaluation:

BLEU (Bilingual Evaluation Understudy): Originally for machine translation. Measures n-gram overlap between generated text and a reference. Good for tasks where exact phrasing matters.
ROUGE (Recall-Oriented Understudy for Gisting Evaluation): Designed for summarization. Measures recall: how much of the reference answer appears in the output? Useful when you care about coverage: did the model hit all the key points?
Execution-based evaluation: For code generation, one metric matters. Does it run? Does it produce the correct output? Execution-based eval runs the generated code against test cases and checks the results. A function that returns the wrong answer fails. Another function that passes all tests succeeds.

But all three share the same trade-off:

They measure surface-level similarity to a reference, not actual quality. So use these where they fit.

17. LLM as Judge

This is what makes evaluation scalable…

LLM-as-judge uses a more capable model to evaluate your application’s outputs. You give it three things:

Original input,
Output you’re evaluating,
Your rubric.

The judge then returns a score and an explanation.

Think of it like automated load testing.

Nobody clicks through 10,000 user flows to check for regressions. You can instead automate it. LLM-as-judge does the same thing for quality. It lets you run your rubric across thousands of outputs without thousands of human hours.

GPT-5 and Claude Opus are commonly used as judges. They’re capable enough to apply nuanced rubrics reliably. You’re calling them via API, NO custom training required.

But here’s the catch: judges can be wrong.

They have their own biases and blind spots. An LLM judge is an approximation of human judgment, not a replacement for it.

More on how to handle this in a moment…

18. Pointwise vs Pairwise Evaluation

These are 2 flavors of LLM-as-judge evaluation:

Pointwise
- You ask, “Score this output on a scale of 1–5.”
- It’s simple and fast.
- And each output needs one evaluation call.
Pairwise
- You ask: “Here are two outputs. Which one is better?”
- This is usually more reliable because comparing is easier than scoring.
- It’s especially useful when you want to know if a new prompt is actually better.

But the downside of pairwise is the cost…

You’re evaluating two outputs instead of one, so it can be twice as expensive. At a small scale, this doesn’t matter, but at a production scale, it adds up quickly.

Because of this, the industry standard is to use a tiered approach:

Online evaluation (production monitoring)
- Use fast, cheap methods like heuristics or smaller, fine-tuned judge models.
- These can run continuously without high cost or latency.
Offline evaluation (pre-ship testing)
- Use the best LLM judge you have.
- Run pointwise and pairwise comparisons.
- This is acceptable because you’re only running it before deployments.

Think of the powerful judge model as a gate before deployment, and NOT something you run on every request.

19. Judge Calibration

Before you rely on an LLM judge, you need to know how well it matches human judgment…

Judge calibration measures this:

You take a sample of outputs, have both humans and the judge score them, and then check how often they agree.

If agreement is high, your judge is a good proxy for human evaluation.
If agreement is low, the judge may be measuring the wrong thing.

A poorly calibrated judge is risky.

It can give you confidence in results that aren’t actually correct. So calibrate your judge before using it. Then recalibrate periodically, especially after changing models or prompts.

Get 20% savings by getting a group subscription right now:

RAG System Evaluation

Most LLM applications today aren’t just “prompt in, response out.”

They use Retrieval-Augmented Generation (RAG): first fetch relevant documents, inject them into the context, and then generate a grounded response.

This helps reduce hallucinations because the model can rely on real data rather than just its training data. But it doesn’t solve the problem completely…

It also introduces new failure modes that standard LLM evaluation doesn’t catch…

20. RAG Triad

The standard framework for evaluating RAG systems has three dimensions:

1 Faithfulness

Did the answer actually come from the retrieved context?

This often surprises people. You can retrieve the perfect documents, and the model can still ignore them. Faithfulness checks whether the response is grounded in the provided sources.

A failure here looks like a confident answer that isn’t supported by the retrieved documents. So it’s just a hallucination with extra steps.

2 Answer Relevance

Did the response actually answer the user’s question?

A response can be faithful to the context and still miss the point. The model may use the right documents but answer the wrong question.

Answer relevance measures how well the response matches the user’s intent. While Faithfulness checks if the model stayed within the context.

3 Context Precision

Did the retrieval step fetch the right documents?

Even a perfect generator will fail if the input context is poor. If retrieval brings in weak or loosely related documents, the model either guesses or tells the user it doesn’t know.

Context precision evaluates the retrieval stage. It checks how many of the retrieved documents were actually relevant.

Think of a RAG system as three stages: retrieval, augmentation, generation.

Each stage can fail on its own… The RAG triad is your observability layer to isolate where the problem is, so you know what to fix.

21. RAG-Specific Failure Patterns

Understanding the RAG triad is one thing. Spotting failures in practice is another…

Here are the three most common failure patterns:

1 Retrieval returns irrelevant chunks (Context precision failure)

The system retrieves the wrong or loosely related documents. And the model then either:

Hallucinates to fill the gaps,
Or says it doesn’t have enough information.

The fix is in your retrieval layer: embedding quality, chunking strategy, re-ranking.

2 Retrieval is correct, but the model ignores it (Faithfulness failure)

The right documents get retrieved, but the model doesn’t use them.

This is usually a prompting issue. The model isn’t being forced to rely on the provided context. The fix is to strengthen the prompt, so the model stays grounded to the sources.

3 The answer is grounded but doesn’t help the user (Answer relevance failure)

The answer is technically correct and grounded, but doesn’t solve the user’s problem. This usually means your knowledge base is missing the right information. The fix is to improve or expand your data, NOT your prompt.

Get 20% savings by getting a group subscription right now:

Get 20% off a group subscription

Offline vs Online

Scoring a single output is one problem.

Building a system that reliably catches failures (before and after deployment) is another. There are two environments where evaluation happens:

22. Offline Evaluation

Offline evaluation happens before you ship.

You make a change: a new prompt, model version, or retrieval strategy. Before it reaches users, you test it against your golden dataset. Then you score the outputs and compare them to your current system.

Think of this like a continuous integration (CI) pipeline for LLMs.

A change shouldn’t go live unless it passes your evals. If quality drops, you catch it before deployment.

Plus, this is where you use your best (most accurate) judge model. For example, GPT-5, Claude Opus. Cost matters less here because you’re only evaluating a finite set of examples, and NOT each user request.

23. Online Evaluation

Online evaluation occurs in production.

You’re sampling live outputs continuously and scoring them. The goal is to catch issues that offline evals miss: edge cases, unusual inputs, and failures that only appear at scale.

Think of online eval as your monitoring system…It helps you catch problems before users report them.

Yet the main constraint is cost. You can’t run your most expensive models on each request. So online eval relies on heuristics and smaller judge models; it’s cheap enough to run continuously.

This means you trade some accuracy for scale…

Offline and online eval should work together:

Offline eval catches known issues before deployment
Online eval catches unexpected issues in production

So you need both for a reliable system.

24. Prompt Versioning and Regression Testing

Your prompt is code; treat it that way.

Track every prompt change: use version control, keep history, and compare differences. When something breaks, you should know exactly what changed and when.

Without versioning, you’d NOT know what changed.

Regression testing means running your eval suite against each prompt version. If your old prompt scored 4.2 and the new one scores 3.8, you’ve introduced a regression and caught it before it reached users.

This sounds simple, but most people don’t do it. They lack a solid eval setup: no golden dataset, no clear rubric, no infrastructure.

This is what separates people who iterate confidently from people who ship & pray.

25. Benchmark-Based Evaluation

Benchmarks are standard tests used to compare different LLM models:

1 MMLU (Massive Multitask Language Understanding)

Tests knowledge across many subjects like math, science, law, and medicine. Think of it as a general knowledge exam for LLMs.

2 HellaSwag

Tests common sense reasoning. The model is given the start of a scenario and must predict what happens next. (Many earlier models struggled with this.)

3 HumanEval

Test code generation. The model gets a function signature and must write the correct implementation. It’s measured using pass@k: how often the model gets the right answer within k attempts.

Here are two useful leaderboards:

Hugging Face Open LLM Leaderboard for open-weight models
Chatbot Arena (LMSYS), where models get ranked based on human preferences

Remember, these benchmarks are useful for comparing models, but they don’t always reflect real-world performance for your specific use case…

26. Benchmark vs Real-World Tradeoff

Benchmarks don’t tell you if a model will work for your specific use case.

A model that scores 90% on MMLU might still struggle with your domain-specific language. A model that performs well on HumanEval might generate code that doesn’t fit your standards… Benchmarks measure general capability, but your application might need a specific capability.

So use benchmarks to narrow down your options. Then use your own eval on your own golden set to make the final decision.

27. Dataset Contamination / Data Leakage

There’s a problem with benchmarks that most engineers overlook:

Dataset contamination occurs when evaluation data overlaps with the model’s training data. The model has already seen the answers. So its high benchmark score reflects memorization, NOT capability.

This happens because training data and benchmark datasets come from the internet. And the data overlap is often unknown. Over time, widely used benchmarks become less reliable, which is why new ones keep getting created…

To avoid this, don’t rely on public examples for your evals…Plus, use real user queries and create your own datasets instead.

Share this post & get rewards for the referrals.

Failure Modes (What Not to Do)

You can have the right tools and still build a BAD eval system.

Here are the common mistakes:

28. Eval Anti-Patterns

Vibe-based evaluation.

“I tried it a few times, and it looked good.”

This is the most common mistake. Informal spot-checking creates false confidence. It doesn’t catch edge cases, track performance over time, or scale beyond one person looking at a few outputs.

Vibes are a starting point, but not an eval system.

The single-sample trap.

You run your eval suite once and report the results.

But LLMs are nondeterministic…

A bad prompt might look good
A good prompt might look bad

So run many samples and aggregate results. Remember, report variance, not just average scores.

Goodhart’s Law in disguise.

“When a measure becomes a target, it stops being a good measure.”

If you optimize for a metric too aggressively, the metric stops measuring what you care about…

Reward confidence: you’ll get confident hallucinations
Reward length: you’ll get long, low-quality answers

Metrics are only a proxy for quality. So don’t mistake them for the actual goal.

Eval-production mismatch.

Your golden dataset reflects what you expect.

Yet real users behave differently. They’re vague, unpredictable, and use your system in ways you didn’t anticipate. If your evals don’t reflect real usage, your scores are misleading. A high pass rate on unrealistic data doesn’t mean your system works.

Ignoring tail failures.

A 92% pass rate sounds good. But what about the 8%?

LLM failures are often catastrophic, not graceful. One harmful or incorrect response can matter more than hundreds of correct ones. So always review failure cases, not just averages.

The worst outputs tell you the most…

Get 20% savings by getting a group subscription right now:

Get 20% off a group subscription

Decision Framework

There’s no single tool that solves LLM evaluation.

What works is a layered system… Each layer catches problems the previous one misses…

29. Eval Stack

Layer 1: Heuristics.

Fast, deterministic, and cheap. These run on every output and catch structural issues such as incorrect format, missing fields, or banned content.

Layer 2: Semantic similarity and task-specific metrics.

These run automatically against reference answers. They help catch meaning-level failures without the cost of an LLM judge.

Layer 3: LLM-as-judge (offline).

This runs before deployment on your golden dataset using your best judge model. Its job is to catch quality regressions before they reach users.

Layer 4: LLM-as-judge (online, smaller model).

This runs in production on a sample of live outputs using a smaller, cheaper judge model. It helps catch failures at scale that offline evals missed.

Layer 5: Human spot checks.

These run periodically. And keep you calibrated, verify automated evals scoring still match human judgment, and add new failure cases back into your golden dataset.

Each layer has a different cost profile and catches a different failure type…

Eventually, you want all five. But you do NOT need all of them on day one.

So if you are starting from scratch, here is a simple three-step MVP:

Step 1: Build a golden set of 50 examples.

Don’t write all of them yourself.

Use real user queries if you have them. If not, create 30 representative examples and 20 edge cases. Also, include normal cases, adversarial inputs, and strange phrasings you expect users to try.

This dataset becomes the foundation of your eval system.

Step 2: Add one deterministic heuristic.

Pick the single most important structural requirement for your output.

That might be length, format, a required field, or a banned phrase. Then write a simple code check for it. This is quick to build and catches more failures than most people expect.

Step 3: Add one LLM judge prompt.

Take your rubric and turn it into a judge prompt.

Then use a strong model to score outputs on one important quality dimension, and run it on your golden dataset. Read the scores and the explanations carefully.

You will find something surprising…that’s the point.

That is your MVP eval system.

Everything else builds on top of it: pairwise comparisons, online monitoring, RAG scoring, regression testing, and more.

Start here…

Closing Thoughts

Evaluation isn’t something you do at the end.

Instead, it’s what makes iteration possible. Without it, you’re not doing LLM engineering. But guessing and shipping changes, only to find out from users that they didn’t.

Traditional software engineering learned this lesson the hard way: CI, automated tests, production monitoring…these aren’t optional. They’re what make fast, reliable development possible.

LLM systems need the same discipline. The difference is that the outputs are nondeterministic and hard to measure.

Now you have the vocabulary to deal with that…

Start simple and build your eval stack step by step. Over time, you’ll know whether a change improved quality before users ever see it.

👋 I’d like to thank Anshuman for writing this newsletter!

If you’re building LLM applications and want to go deeper, follow him on LinkedIn and X.

Don’t forget to check out his newsletter, AI Proof -- it’ll help you stay relevant in the AI era.

Louis and I launched the GENERATIVE AI MASTERCLASS (newsletter series exclusive to PAID subscribers) this month.

When you upgrade, you’ll get:

Simple breakdown of real-world architectures
Frameworks you can plug into your work or business
Proven systems behind ChatGPT, Perplexity, and Copilot

👉 CLICK HERE TO JOIN THE GENERATIVE AI MASTERCLASS

(Golden members will get the next Generative AI newsletter in the first week of May.)

If you find this newsletter valuable, share it with a friend, and subscribe if you haven’t already. There are group discounts, gift options, and referral rewards available.

👋 Find me on LinkedIn | Twitter | Threads | Instagram

Want to reach 200K+ tech professionals at scale? 📰

If your company wants to reach 200K+ tech professionals, advertise with me.

Thank you for supporting this newsletter.

You are now 210,001+ readers strong, very close to 210k. Let’s try to get 211k readers by 29 April. Consider sharing this post with your friends and get rewards.

Y’all are the best.

Variance is how much your results change when you repeat the same evaluation. It tells you how spread out or inconsistent your scores are.

Criteria: what you care about when judging the output (e.g., accuracy, clarity, helpfulness)
Rubric: how you score those criteria consistently (e.g., a 1–5 scale with clear definitions)
Test cases: specific inputs or examples that you use to evaluate the model

Vector Database - A Deep Dive

Maxine Meurer — Sat, 25 Apr 2026 09:11:55 GMT

Share this post & I'll send you some rewards for the referrals.

You’ve built a search feature.

Then a user types a query, your application hits the database, and results come back ranked by relevance.

Simple enough, it works perfectly.

But you thought about how search needs to actually understand what users meant, not just match the exact words they typed.

A user searching for “how do I log in” should find results about “account access” and “sign in” even if those exact words never appear in their query.

This kind of search, where the database understands meaning rather than just matching words, is called semantic search. It requires a completely different approach to how you store and search your data.

Yet if you try adding this in as a feature, everything that made your traditional relational database fast will stop working.

The tools you relied on: indexes, WHERE clauses, keyword matching were all built for finding exact values.

Semantic search does NOT work with exact values.

Instead, it works with meaning, and meaning is harder to search efficiently. A search that took milliseconds before now takes seconds, and the bigger your dataset, the worse it gets.

Your database is NOT broken.

It’s doing exactly what you told it to do: comparing your search against every single item in your database, one by one, because there is no other way to find “similar” when there is NO exact match.

This is where vector databases come in.

Not because they are trendy, but because traditional databases were never designed to handle this kind of search at scale.

If you’re a software engineer encountering the term “vector database” for the first time, whether your team is adding AI features or you keep seeing it mentioned everywhere and want to understand what it actually means, this newsletter explains it from the ground up…

[Webinar] Stop babysitting your coding agents (Partner)

Agents can generate code. Getting it right for your system, team conventions, and past decisions is the hard part – you end up wasting time and tokens in correction loops.

More MCPs give agents access to information, but not understanding. The teams pulling ahead use a context engine to give agents exactly what they need.

Join Unblocked live on May 6 (FREE) to see:

Where teams get stuck on the AI maturity curve
How a context engine solves for quality, efficiency, and cost
Live demo: the same coding task with and without a context engine

(Thanks to Unblocked for partnering on this post.)

I want to introduce Maxine as the guest author.

She’s a cloud infrastructure engineer who spends her days scaling databases, debugging production incidents at 2 am, and writing about what actually works in production.

You can get a copy of her LLMs for Humans: From Prompts to Production (at 30% off right now). It’s 20 chapters of practical applied AI with real production context, not theory. And it’ll help you get smarter about using AI tools in infrastructure workflows.

Checkout her work:

Plus, if you’re thinking about making a career move into cloud or DevOps and want a structured path to get there, get a copy of her The DevOps Career Switch Blueprint.

Inside this newsletter, you’ll get:

Why traditional search breaks. How exact-match databases fail with semantic meaning and why O(n) similarity search doesn’t scale.
How vector search works. Embeddings, high-dimensional space, and why similarity is measured by distance instead of exact matches.
The core architecture. Storage, indexing (HNSW), and query layers, with key tradeoffs in cost, latency, and accuracy.
Query execution in practice. ANN search, metadata filtering (pre vs post), and ranking, including how real systems return results.
Production realities. Scaling, replication, cost structure, monitoring, and failure modes teams hit in production.
Decision framework. When pgvector is enough, when to use a dedicated vector DB, and when distributed systems become necessary.

Why Traditional Databases Struggle with Embeddings

Traditional relational databases are good at exact matches and range queries.

They use indexes and data structures so the database can jump directly to relevant rows without scanning the entire table.

When you query WHERE user_id = 12345, database uses an index to find that row instantly, the same way you’d look up a word in a dictionary: you don’t read every page, you jump straight to the right section. Also when you query, WHERE created_at > ‘2026-01-01’, it works the same way for dates.

But embeddings break this model because there is no exact value to look up!

An embedding is a list of numbers that represents the meaning of a text.

Words, sentences, or entire paragraphs get converted into these numbers by an AI model, and similar meanings produce similar numbers.

Instead of finding a specific row, you find rows closer in meaning, and “close” requires measuring distance in high-dimensional space rather than matching values. Think of it like a regular map with latitude and longitude, except you’re working with 1,536 dimensions instead of just 2.

Let’s walk through a concrete example to see why this creates a problem…

Imagine you have a document database where each document gets split into chunks.

And think of chunks like cutting a book into individual paragraphs so you can search each one separately. Each chunk gets converted into a 1,536-dimensional vector using an embedding model like OpenAI’s text-embedding-3-small:

A user submits a search query
App converts the query into a vector using the same embedding model
Now you have to find the “most similar” document chunks

Without specialized indexing, the only way to find “most similar” ones is to calculate the distance between your query vector and every chunk vector in your database, sort by distance, and return the top results.

Let’s break down what it means…

A vector is simply a list of numbers. In AI, those numbers represent meaning.

When you convert text into an embedding, you’re assigning it a location in a very high-dimensional space (imagine a map with 1,536 dimensions instead of the normal 2, latitude and longitude). Similar pieces of text end up close to each other on this map, while unrelated text ends up far apart.

When a user searches for something, their query also gets converted into coordinates on this map. To find the most relevant results, you need to measure the distance from the query’s location to every document chunk’s location in your database, then return the closest ones.

But here’s the problem…

Without an index, there’s no shortcut. You must measure the distance to each item.

Now imagine you’re standing in Times Square in New York City, and you need to find the 10 closest coffee shops. Without a map or any organized system, your only option is to:

Walk to every building
Check if it’s a coffee shop
Measure how far it is from Times Square
Keep a list of the 10 closest ones you’ve found

If there are 10k buildings, you check all 10k. And if there are 10 million buildings, you check all 10 million.

The more buildings that exist, the longer it takes…

Put simply, it takes O(n) time, and it grows linearly with the size of your dataset.

In practical terms, if you have 10,000 document chunks and each distance calculation takes 0.0001 seconds, one query takes 1 second. That’s not terrible.

But what if you have 10 million chunks?

That same query now takes 1,000 seconds, i.e., 16 minutes. And if your application serves 100 queries per second from different users? Your system immediately grinds to a halt!

This is why traditional databases struggle with embeddings…

Their indexes are built for exact matches.

They can jump straight to user_id = 12345 without checking every row. But with vectors, there are no exact matches, only “similar” matches, and similarity requires measuring distances.

Let’s now imagine finding a similar song to the one you already like…

You cannot just look it up by name; you have to compare it against every other song across hundreds of characteristics: tempo, key, energy, mood, and so on. The more songs in your library, the longer it takes.

Vector search works the same way, except instead of songs, you compare text, and instead of a few characteristics, you are comparing across 1,536 dimensions at the same time.

Without specialized techniques to avoid checking everything, the math simply doesn’t scale.

This is where people often bring up pgvector, a PostgreSQL extension that adds vector search capabilities to Postgres. It’s a good option for smaller datasets and lets you avoid adding another database to your stack. This keeps operational efforts very low.

For datasets with fewer than 100,000 vectors and moderate query volumes, pgvector works well and keeps your infrastructure simpler.

But as your dataset grows into the millions and your query latency requirements tighten, you hit the limits of what a general-purpose database can do for this specific workload.

You don’t need a specialized vector database just because you’re working with embeddings, but you need to understand when the operational trade-offs tip in favor of dedicated infrastructure.

This brings us to the core challenge that vector databases solve…

How do you find approximate nearest neighbors in high-dimensional space without comparing against every vector, while still returning results that are “good enough” in terms of accuracy?

The answer involves building specialized index structures that trade accuracy for massive speed improvements, and understanding those trade-offs is essential for operating these systems in production.

Architecture: How It Works Under the Hood

Vector databases have three key layers: storage, index, and query.

Each layer involves trade-offs that directly affect your operational concerns: cost, latency, accuracy, and scalability.

Let’s break down what matters from an infrastructure perspective…

Storage Layer: Where Vectors Live

Vectors are massive.

A single 1,536-dimensional vector using 32-bit floats takes up 6KB of storage. If you multiply that across millions of vectors, storage costs add up fast. This is a problem every vector database must solve at its foundation: the storage layer determines how vectors are physically stored, compressed, and retrieved from disk. This layer focuses on compression techniques that reduce both storage costs and the memory footprint while preserving as much accuracy as possible.

The most common approach is quantization.

Quantization reduces the precision of each dimension to use fewer bits.

Instead of storing each dimension as a 32-bit float, you quantize to 8-bit integers (scalar quantization) or binary representations (binary quantization).

This can reduce storage and memory requirements by 4x or more, but it introduces approximation error. Approximation error is the gap between the compressed version of data and the original.

When you round numbers to save space, your calculations become slightly less precise. In vector search, this means you might occasionally miss a result that was truly the closest match, but in practice, the difference is small enough that most applications never notice it.

In production, you’re constantly trading off storage costs for search accuracy, and the right balance depends on your use case.

Another critical decision is the trade-off between memory and disk…

For maximum query speed, you want your vector index in memory, but that’s expensive at scale. Some vector databases support tiered storage, where frequently accessed vectors stay in memory while cold data lives on disk, or hybrid approaches in which the index structure stays in memory while raw vectors get pulled from disk when needed.

These architectural choices directly affect your cost structure and query latency percentiles.

From an operational perspective, you need to understand how your chosen vector database handles durability and recovery:

What happens if a node crashes mid-write?
How long does it take to rebuild indexes after a restart?
Are vectors written to disk synchronously or asynchronously?

These aren’t theoretical questions. They’re the difference between acceptable downtime and a major incident when something goes wrong.

Indexing Layer: Graph Structure

This is where the “magic” happens, and by magic I mean carefully engineered data structures that let you find approximate neighbors without exhaustive search.

The most popular algorithm in production vector databases is HNSW (Hierarchical Navigable Small World), and it’s worth understanding at a conceptual level even if you’re not implementing it yourself.

The core idea behind HNSW is based on “small world” networks, the same principle that says you’re only a few connections away from any other person on the planet.

Instead of comparing your query vector against every vector in your database, HNSW builds a multi-layer graph in which similar vectors are connected to each other. When you query, the algorithm starts at the top layer and navigates toward your target by following edges to similar neighbors, dropping through layers until it reaches the most similar vectors at the bottom layer.

Remember our map analogy?

Your vectors are locations on a high-dimensional map, and you’re trying to find the closest ones to your query without checking every single location. HNSW solves this by building a navigation system with different zoom levels.

Think of it like Google Maps navigating from New York to a specific coffee shop in San Francisco:

Top layer (zoomed out): From New York, HNSW identifies the general direction of your target, the West Coast, and follows connections to get you to California. You’re not considering every city in America, just the major waypoints that get you closer to where you need to be.
Middle layers (zooming in): Now you’re looking at California. The algorithm narrows down to the San Francisco Bay Area, following connections between nearby locations. You’re still not checking every street, just the neighborhoods that are directionally correct toward the coffee shop you want to visit.
Bottom layer (street level): Finally, you’re at the specific neighborhood and can see individual addresses. The algorithm follows the last few connections to find the exact coffee shop location (or, in vector terms, the most similar document chunks).

At each zoom level, you’re following connections between “nearby” locations without checking everywhere.

You only need to follow the connections that get you closer to your target. This is how HNSW turns an O(n) problem into something much faster: it builds a navigation structure that lets you skip large portions of the map in the wrong direction.

The key trade-off in HNSW is speed versus accuracy, controlled by two parameters: the number of connections each vector has (called “M” or “max connections”) and the number of neighbors to check during search (called “ef” or “exploration factor”).

Higher values mean better accuracy, but slower queries and larger indexes…

Lower values mean faster queries, but you might miss some relevant results. In production, you tune these parameters based on your latency requirements and tolerance for accuracy.

Building the HNSW index is expensive.

Inserting vectors requires calculating distances and updating graph connections, which is why bulk index building often happens offline or during low-traffic periods. Index build time matters because it affects how quickly you can recover from failures, deploy updates, or onboard new data.

You do not need to understand the math behind HNSW to work with vector databases.

Just like driving a car, you might not know how the engine works, and that doesn’t affect how you drive. But knowing that the engine exists and what it does helps you figure out what is wrong when the car breaks down.

What you need to know is that HNSW produces approximate results, not perfect ones, and it’s possible to adjust its approximation. When search results seem wrong, queries suddenly slow down, or the index fails to build, this understanding helps you identify the problem.

Get a copy of LLMs for Humans: From Prompts to Production right now and save 30%.

Reminder: this is a teaser of the subscriber-only newsletter, exclusive to my golden members.

When you upgrade, you’ll get:

High-level architecture of real-world systems.
Deep dive into how popular real-world systems work.
How real-world systems handle scale, reliability, and performance.

And much more!

Get the AWS Data & AI Book

Query Layer: How Requests Move Through System

When a query hits your vector database, it follows a specific path:

9 Agentic Patterns, Simply Explained

Neo Kim — Wed, 22 Apr 2026 11:50:43 GMT

Share this post & I'll send you some rewards for the referrals.

When you build software with LLMs1, you need to decide how much control to give the model.

Does your code run every step, or does the model figure out the steps on its own?

Agentic patterns are design patterns for making that decision. They’re the same architecture choices you already make in any system: who controls what happens next, what happens on failure, and how data moves between components.

The difference is that some of those components are now language models.

This newsletter covers nine of those patterns…

On one end, workflow patterns where your code controls every step. On the other hand, agent patterns where the Large Language Model (LLM) decides what to do next.

That boundary is a decision about how much control to hand to the model: at some point, you stop telling the model what to do and start letting it figure that out.

The first question isn’t which pattern to use. It’s whether you need one at all.

Let’s start there...

Find out why enterprise leaders are doubling down on data foundations for AI (Partner)

AI is moving fast, but your data foundation isn’t keeping up?

That’s exactly why leaders from JPMorgan, Mercedes-Benz, Siemens, and Roche contributed to this AWS book on agentic analytics.

Here’s what you get:

Real strategies from enterprise leaders: Learn how top companies are building data foundations for AI systems that actually scale.
Practical frameworks you can apply: From data strategy to data products, ML, and agentic AI.
Perspectives from 15+ enterprise leaders: Each chapter brings a unique view from senior leaders across industries.
Complex topics broken down clearly: Understand what matters without getting lost in theory.

An AWS book designed to help you build data foundations for intelligent agents that scale.

Download it now and learn how to build systems ready for intelligent agents.

(Thanks to AWS for partnering on this post.)

Inside this newsletter, you’ll get:

The shift from prompts to systems. Why simple LLM calls break down, and how workflows and agents introduce structured decision-making.
A practical escalation ladder. When to stay simple, when to use workflows, and when agents are actually justified.
Workflow patterns. chaining, routing, parallelization, and orchestrator-workers, with clear tradeoffs in cost, latency, and failure modes.
Agent patterns. Reflection, tool use, ReAct, and planning, focusing on how the model makes decisions and where things go wrong.
Evaluator-optimizer loop. How to add a reliability layer with evaluation, iteration limits, and cost control.
Real-world case study. Combining multiple patterns in an AI code review pipeline, showing how to choose the simplest architecture that works.

Not everything needs an agent

Most tasks don’t need an agent2…

Most tasks don’t even need a workflow…The default should be the simplest setup that works, and you escalate only when that setup breaks.

Start with a direct API call to an LLM. You can do more with this than most people think:

Summarization
Classification
Extraction
Rewriting
Translation
Code generation with clear specs

Most people skip this level too quickly!

Workflow patterns are the next step up.

Escalate when a task has many steps, and when focused attention at each step improves the output. Between those steps, you define validation gates: checks that verify the output before passing it forward. The defining feature: you can write every step down before the system runs. Your code owns the control flow.

And the LLM handles the details.

Agent patterns are for when the number and type of steps are unknown until the system runs.

The model needs to look at results, decide what to do next, and choose its own path. For example, say you built a support agent that handles order cancellations. A customer asks to cancel, but the agent doesn’t know upfront whether it needs to check the refund policy, look up the shipping status, or pass the issue to a human.

The steps depend on what it finds…

Here’s how you know it’s time to switch: you write more code handling errors and exceptions than doing the actual work. Or you keep adding special cases for situations the LLM encounters you didn’t plan for.

If you can still write down all the steps before the system runs, stick with a workflow.

Agent patterns sit within multi-agent architectures, i.e., many agents cooperate on the same problem, each with its own role.

One common mistake: you get your system to 70-80% of a prototype and assume the architecture needs upgrading. It usually doesn’t…The real issue is usually prompt quality or missing validation gates. Move to a more complex pattern only after the simpler one has actually failed, not when it feels limiting.

With that baseline set, here are the four workflow patterns:

Workflow patterns: you control the flow

With workflow patterns, your code controls what happens.

You decide the steps, the order, and the checks between them. The LLM handles each step, but your code is in charge…

1. Prompt chaining

Prompt chaining breaks a task into a series of LLM calls.

Each call takes the output from the previous step, and a check runs between them to ensure the result is correct before moving on.

If you’ve ever set up a CI/CD (Continuous Integration/Continuous Deployment) pipeline, this is the same idea. A build pipeline compiles, lints, tests, and packages. Each step must pass before the next one runs.

If tests fail, the pipeline stops.

For example, in legal contract review, this looks like: extract all clauses from a contract, classify each clause by risk level, then generate a summary of high-risk items for human review. Products like Thomson Reuters CoCounsel and Robin AI run variations of this pipeline.

Each step gets a focused prompt with a narrow task, and a validation gate checks the output before passing it forward.

Tradeoffs

Latency grows linearly…

Five steps mean five round-trip. Errors carry forward: if step two misclassifies a clause, steps three through five process bad input. The gates between steps catch these failures early, but they only catch what the conditions you write can catch.

2. Routing

A routing pattern classifies the input and sends it to the right handler: a different prompt, a different model, or a different sub-workflow.

Each handler is built for one type of input.

This is like a hospital front desk. A patient walks in, the front desk checks their symptoms, and then sends them to the right doctor. The front desk doesn’t treat anyone. And the doctor doesn’t check in patients.

Sierra AI routes across 15+ models this way.

Classification tasks go to one model, tool calling to another, and response generation to a third. Customer support platforms like Intercom Fin route each customer request based on what the customer wants and how they feel about it, sending requests to either an AI handler or a human agent.

The cost argument is built into the pattern…

Simple queries hit cheap, fast models. While complex ones hit expensive, capable ones. You’re not paying the highest prices for work that a smaller model handles well.

Tradeoffs

In 1 word: misclassification.

A complex query routed to the cheaper model produces a poor answer. A simple query routed to the expensive model is just a waste of money.

Either way, the router is a single point of failure, and its accuracy sets the upper limit for the entire system.

3. Parallelization

In software, parallelization means running many tasks simultaneously rather than one after another. With LLMs, it has two variants that solve different problems…

Sectioning splits a task into independent parts, runs them in parallel, and merges the results. This is like a construction crew. Electricians, plumbers, and carpenters work in the same building at the same time. Their work gets combined at the end.

GitHub Advanced Security works this way: CodeQL and third-party tools like Snyk and Semgrep scan the same pull request in parallel, each producing independent security alerts.

Voting runs the same task many times with different prompts and combines the answers. This is like a jury. Each juror looks at the same evidence and gives their own opinion, and the group decides together.

You’d build this for something like a security review: three separate prompts analyze the same code change for vulnerabilities, and you flag an issue only if two or more agree. Running the task many times is more expensive, but it catches fewer false alarms.

In short, sectioning gives each worker a different job; voting gives every worker the same job.

Tradeoffs

Cost multiplies with every parallel branch.

Partial failures are a design decision you need to plan for up front: if one branch fails, do you retry it, proceed with the others, or fail the whole operation? There’s no universal answer, and a wrong choice is expensive.

4. Orchestrator-workers

The orchestrator-workers pattern uses one central LLM to break a task into subtasks, assign each subtask to a worker LLM, and combine the results.

The difference from the earlier patterns is that the subtasks aren’t known in advance. Orchestrator figures them out as it goes. This is like a general contractor building a house:

They hire the right people, but don’t know every subcontractor they’ll need upfront
Plumber finds a structural problem, so the contractor brings in a structural engineer
The work plan changes as the project reveals new problems

Cursor’s agent mode works this way. It starts with different agents for different parts of the codebase:

One adding tests
Another updating documentation
A third refactoring shared utilities

You’re trusting the LLM to figure out what work needs to be done, not just to do the work you’ve assigned.

This uses several LLMs, but it’s NOT a multi-agent architecture: a single central LLM remains in control of the entire process. In multi-agent systems, agents can call each other directly without a central coordinator, and control can transfer between them.

Tradeoffs

The orchestrator can lose track of the original goal.

It breaks simple tasks into too many subtasks. And when workers finish faster than it can combine results, the orchestrator itself becomes the bottleneck.

These four patterns get gradually more powerful and more expensive…

Chaining and routing are cheap and predictable: you know what will happen before it runs. Parallelization saves time but costs more. Orchestrator-workers handle problems you can’t predict upfront, but it introduces failure modes you can’t predict either. The first three patterns are simple to understand.

Most production systems shouldn’t need to go further than parallelization. When they do, the next four patterns hand control to the model itself...

Reminder: this is a teaser of the subscriber-only newsletter, exclusive to my golden members.

When you upgrade, you’ll get:

High-level architecture of real-world systems.
Deep dive into how popular real-world systems work.
How real-world systems handle scale, reliability, and performance.

Agent patterns: LLM controls the flow

With agent patterns, you STOP defining what happens and in what order.

Instead of defining steps, you define constraints: what tools are available, how much the model can spend, and when to stop. The model observes, reasons, and chooses what happens next.

That trust decision appears differently in each pattern: how much autonomy, over what actions, with what limits.

5. Reflection

Reflection is when the LLM generates output, reviews its own work, and then fixes the problems it found.

I rejected 1000s of resumes at Meta

Austen McDonald — Wed, 15 Apr 2026 07:40:44 GMT

Share this post & I'll send you some rewards for the referrals.

Most engineering resume advice you come across is junk.

At best, it comes from a tech recruiter who’s looked at 100s of resumes. At worst, it’s from yet another senior engineer cobbling together a blog post based on their experience applying to MAANGO.

This post is not like that.

I’ve made thousands of hiring and leveling decisions as mobile Hiring Committee Chair at Meta. I also led Meta’s engineering team responsible for finding and ingesting over 2M engineers’ resumes into their internal ATS (Applicant Tracking System).

I’ve seen a lot of resumes :)

And because of that, I know that you should treat your resume like a product. Think of it as a landing page for your career, designed to drive each target user who touches it to take action. And yes, your resume has multiple target users—four, in fact—and you should learn what each one is looking for.

(Oh, and btw, a lot of this applies to your LinkedIn profile too, since that is literally a landing page.)

Onward.

The tax you pay to run multiple agents (Partner)

If you’ve spent any time with coding agents, you know the feeling.

You start the morning with a clean plan. Spin up a few agents. One is refactoring the auth module. Another is writing tests. A third is scaffolding a new API endpoint. You’re flying.

Then, around 10:30 AM, you look up and realize you have 20 terminal windows open. One agent is blocked waiting for a decision you forgot to make. Another finished 40 minutes ago, and you never noticed. A third went sideways three commits back. You’re no longer flying. You’re drowning.

You’ve shifted from human as driver to human as director.

When running coding agents in parallel, the bottleneck isn’t just context. It’s your own attention trying to manage 10 agents across 10 terminals. You’re losing your mind to terminal chaos.

Meet Cline Kanban, a CLI-agnostic visual orchestration layer that makes multi-agent workflows usable across providers. Multiple agents, one UI. It’s the air traffic controller for the agents you’re already running, regardless of where they live.

Interoperable: Claude Code and Codex compatible, with more coming soon.
Full Visibility: Confidently run multiple agents and work through your backlog faster.
Smart Triage: See which agents are blocked or in review and jump in to unblock them.
Chain Tasks: Set dependencies so Agent B won’t start until Agent A is complete.
Familiar UI: Everything in a single Kanban view.

Stop tracking agents and start directing them. Get a meaningful edge with the beta release.

Install Cline Kanban Today: npm i -g cline

Get Started Today

(Thanks to Cline for partnering on this post.)

I want to introduce you to Austen McDonald as a guest author.

Austen is a former Senior Engineering Manager and Hiring Committee Chair at Meta, where he spent almost 10 years and conducted over 1,000+ interviews. He’s the industry’s resident expert on behavioral interviews and writes about them on his Substack and in his book, Mastering Behavioral Interviews. As AI raises the bar for ownership and independent judgment, behavioral interviews matter more than ever.

For a few days only, use code SDN26 for 20% off Premium Access, which includes a print copy of his book and an AI copilot that helps you find your best career stories, shape them into high-signal answers, and deliver them well.

Part 1: Your Resume is a Product

Your resume is a landing page for your career search.

It’s the first thing these four personas see when they consider you for an engineering job:

Machines (Applicant Tracking System),
Recruiters (we’ll include sourcers for all you sticklers),
Interviewers,
And Hiring Managers.

Each of these personas looks progressively deeper into the content in the resume. Each one also extracts different, but overlapping, information.

Let’s consider the needs of each of these personas, what they look for in a resume, how they read it, and the decisions they make along the way…

Machines (ATS)

Unless your profile is handed directly to a recruiter, your hiring process starts with software. Whether you apply online, get referred, or turn up in a sourcer’s keyword search, you need to pass this automated filter first…

How ATS systems work:

When you submit a resume, the ATS parses it into structured data: your name, contact info, work history, education, and skills.

It then scores you against the job description on keyword relevance, years of experience, and other recruiter-configured criteria. High-scoring candidates float to the top of the pile; low-scoring ones may never be seen by a human.

Of course, companies use AI-powered search that goes beyond keyword matching. These systems understand that “built ML pipelines” and “machine learning infrastructure” mean the same thing. This is good news—it means you don’t need to stuff your resume with every keyword—but it also means your accomplishments need to be substantively relevant to the role and described as such.

What they need:

Clean, parseable formatting. Fancy designs can break text extraction: tables, some multi-column layouts, text boxes, icons, or images. Even creative section titles like “My Journey” instead of “Professional Experience” are risky.
Easy to find minimum qualifications since an automated tool is probably filtering first on whatever is stated in the job description.
Keywords matching the job posting and role requirements.
Technologies named where they can be easily indexed.

Hot Take: I have no data for this, but numerous companies have buttons on their career sites that say, “Upload your resume to see if you’re a fit.” Seems like it would be valuable to iterate on your resume until the site tells you you’re a good fit, lol.

Recruiters

Recruiters (or, more properly, sourcers) are speed readers scanning hundreds of resumes daily.

After basic machine filtering for experience, education, and past companies, they’ll skim your summary and a few accomplishment bullets, then make an in/out call in 5–7 seconds.

They’re looking for two things: candidates who meet the stated requirements, and candidates who fit the hiring manager’s mental model of a great hire. Those who clear both bars move forward to the next stage.

This is either a direct screening process, such as a phone call, or a final call to the hiring manager or committee before they begin screening process.

What they need:

To immediately see that your profile fits the minimum requirements. They will reject you immediately if you don’t have the years of experience or the proper degrees, or have worked in similar roles (based mostly on title) that are identified in the job description.
A quick visual path to something that excites them about you and convinces them you fit the idea of what the hiring manager is looking for. This could be a name-brand company, a project relevant to the hiring team, or frequent use of technology related to the role. Give them a reason to take action and move you to the next step.
Brevity. Obviously, with 100s of these to review, the shorter the time all this takes, the better. So keep it to one page unless you’re a CEO.

Interviewers

There are a couple of places where interviewers or members of the engineering team read your resume.

Sometimes there is a layer of reviewers who approve the resume before they green-light a technical screen. It could be the hiring manager or a committee of ICs. You can see them more as a technical version of a recruiter, as they have the same needs.

Technical screeners are a different type of reader. They’re looking for conversation starters and signals that you’d contribute to the team.

Onsite interviewers (especially behavioral interviewers) are gauging your level based on the business impact you’ve driven. They’re also looking for red flags, such as performance challenges: short stints at a workplace or career gaps.

What they need:

Easy to find accomplishments that look like they fit with the work of the company/team
Evidence you can defend your claims under scrutiny: those accomplishments need technical details
Signals of technical depth and leadership
A minimum of reasons to be suspicious

Hiring Managers

Hiring managers are sometimes the first, as we talked about above in “Interviewers,” but almost always the last person to touch your resume.

At this stage, they’re reviewing a handful of finalists, which makes them the most thorough readers of your resume. They know what their team needs and look for specific evidence of alignment with those needs.

What they need:

Evidence that your past work resembles what they’re hiring for
Measurable impact with appropriate scope
They want ownership and growth signals: can you self-direct, drive work to completion, and adapt fast, i.e., you’re ready for agentic coding.
Clear visual hierarchy so they can find what matters

Now that you know who’s reading your resume and what they’re scanning for, there’s a natural question: what do you actually put on the page?

The answer isn’t “everything you’ve ever done.” Instead, curate a story of the impact you’ve made in your career. The next part walks you through that step by step...

Share this post & get rewards for the referrals.

Part 2: Resume Content that Converts

Now that we understand the users of your resume, let’s build a product that converts, moving you through the recruiting pipeline to a hire!

The Basics: What Goes In and What Stays Out of Your Resume

Before diving into content, we should get the basics out of the way.

Here’s the standard structure that readers expect, roughly in this order:

Contact Information – name, email, LinkedIn (maybe GitHub, if it’s good; more on that later)
Summary – Who you are and why you’re right for the role in 2-3 sentences
Work Experience – Your professional history with accomplishments
Education – Degrees, relevant coursework, honors
Projects – Side projects, open source, portfolio work
Skills – The keyword soup for machines, as we’ll see

This order should vary depending on who you are: experienced candidates can de-emphasize education; new grads should surface projects higher.

But readers expect to find things in roughly this order, so don’t make them hunt.

Contact Information

Of course, you should include basics like name and email.

If you want a phone call, include your phone number. Recruiters will ask for it later if you don’t give it now.

I also believe you should include a LinkedIn URL, linkified in the PDF form of the resume. Companies expect engineers to have LinkedIn profiles and may ingest that profile data automatically (don’t tell the LinkedIn lawyers). Much of the advice I give in this newsletter applies to LinkedIn profiles as well, especially the summary and bullet-point writing.

And please get a decent profile photo and cover photo for your LinkedIn profile.

Anything else you put in this contact information section should be additive to your candidacy. Like, don’t include a GitHub or personal website link if that’s there, it’s unimpressive. GitHub profiles need at least some recruiting-friendly repos that are deployed somewhere and well-documented. Personal websites should work (lol) and be a high-quality representation of yourself. Otherwise, don’t include them.

If you’re a US citizen applying to US roles and your citizenship isn’t obvious from your resume, note it in your contact section.

Write a Summary

Most resume advice is subjective.

Here’s my most opinionated take in this newsletter: always include a summary and put it at the top.

Two or three sentences max. And tailor it to the role. After reading it, I should immediately come away with the idea that you’d be a good fit for the role. Again, recruiters will heavily depend on this summary to help them decide.

⚠️ Warning: Don’t let AI write your summary. AI-generated summaries are an epidemic. They all sound the same—vague, buzzword-laden, and interchangeable. Recruiters and hiring managers can spot them instantly, and they leave readers with zero understanding of who you actually are.

These all say nothing:

Results-driven Front End Engineer with 4+ years of experience building scalable, user-centric web applications. Proven track record of delivering high-quality code in fast-paced environments. Passionate about creating seamless user experiences and collaborating with cross-functional teams.
Dynamic and detail-oriented Front End Engineer with expertise in modern JavaScript frameworks. Adept at translating complex requirements into elegant, performant solutions. Strong communicator with a passion for continuous learning and staying current with industry trends.
Innovative Front End Engineer bringing 4 years of hands-on experience crafting responsive, accessible interfaces. Committed to writing clean, maintainable code and driving technical excellence. Thrives in collaborative environments and passionate about mentoring junior developers.

My mind just numbs over reading these.

Phrases like “results-driven,” “passionate,” and “cross-functional teams” are filler. They fit any candidate, which means they describe none of them.

What kind of products have you built? (B2B dashboards? Consumer apps? E-commerce?)
What’s your technical specialty? (Performance optimization? Design systems? Accessibility?)
What makes you different from the other 500 front-end engineers applying?

A Better Summary

Let’s build something better, remembering to tailor the summary to the role.

Suppose you were applying to some B2B SaaS company as this same front-end engineer who’s looking to replace their AI-built summary.

What about this one, that I typed with my own meat sticks:

I make complex data feel simple. I’m a front-end engineer with 4 years experience building B2B decision platforms at multiple fast-moving startups.

First, it’s shorter, and therefore more likely to be read.

My technical writing instructor in college told me, “Every line decreases your readership by half.” I don’t know if that’s true, but it’s a good principle to live by.

Next, it’s interesting and attractive. Who doesn’t want someone who makes complex data feel simple?

It also immediately communicates who you are: a front-end engineer with 4 years of experience at startups. Now I know whether your profile is relevant to me or not.

Finally, it connects to the role.

Note: Let’s talk about the “cute summary”. I actually prefer something like “Assistant to my local Claude Code instance” over a buzzwordy AI summary. At least it communicates some creativity and humanity. But I think you can do better and should do better unless it’s really obvious your resume matches the target role—like you’re applying for a senior ML safety engineering role at Big Tech X and just came from Big Tech Y with the same title.

Writing Bullets That Get Read

Resume readers are hiring you to deliver business value using technology.

That’s why they need an engineer. So give them confidence that you can do that with the bullets you list under your experience.

Here’s how you can do this:

Lead with results, not activities. Start with what happened, not what you did. The activity is the supporting detail. Use a formula like: “Delivered [business value] using [technology/approach].”
Quantify and show scope. Percentages, dollar amounts, user counts, latency improvements, and length of the project. Let the numbers imply how hard the work was.
Your bullets become interview topics. The accomplishments you list will get asked about. Make them interesting enough to discuss and defensible under scrutiny.
Include leadership signals. Highlight moments where you took initiative, managed others, or drove a project forward—even if you weren’t a manager.
Lead with your best. Don’t sort accomplishments chronologically within a role. Place the most relevant and impactful ones first in each section.
Embed tech in your accomplishments. Mentioning technologies within achievement bullets improves your ranking for those keywords, not just in a separate skills section.

Here are some examples of weak vs strong bullets:

Weak: Worked on the checkout system and fixed bugs.
Strong: Reduced checkout abandonment by 12% by identifying and resolving a race condition in the payment flow using React and Stripe’s API.
Weak: Responsible for migrating the database to a new system.
Strong: Led a 6-month migration of 50M records from MySQL to PostgreSQL, coordinating across 3 teams and achieving zero downtime during cutover.
Weak: Helped improve the CI/CD pipeline.
Strong: Cut deploy times from 45 minutes to 8 minutes by parallelizing test suites and implementing Docker layer caching, adopted by 12 engineering teams.
Weak: Mentored junior engineers on the team.
Strong: Mentored 3 junior engineers through their first production launches, with one promoted to mid-level within 8 months.

Projects & Proof of Work

I don’t think projects or other non-paid work are super valuable on a resume, unless you are in one of two positions:

You’re a new grad with little paid experience, and projects are actually what you’ve done most of in your life,
Or you’re trying to land a role in a new technical area where you haven’t technically worked (*cough* AI).

Outside of those two cases, projects are marginal. They might tip the scales if a recruiter is on the fence, but they rarely move the needle on their own.

Generally, though, prioritize contributions from paid work and don’t spend too much vertical space on projects.

GitHub can be useful. If your GitHub profile has a lot of green squares and/or a few polished repos deployed somewhere, with good READMEs, tests, and documentation, then great. Be sure to pin repos that showcase your best work. But if your contribution chart doesn’t look like a Christmas tree and/or you don’t have meaningful projects, just leave it off.

Side projects have some value. I don’t think side projects do a ton for you, since working by yourself with no expectation of production-level code is just a different environment than a real job. But they do show initiative and genuine interest. Skip the generic description (”AI-powered todo app in React Native”). Write something that shows personality: “Maximalist AI approach to running my life — a case study in build-fast-with-Claude-Code development.”

Technical blogs or writing are worth considering. Publishing shows communication skills and depth of understanding. If you provide anything, make sure it’s easy for me to find the 2-3 best pieces.

The Skills Section

If there’s one part of the resume that’s most often overlooked by readers, it’s the skills section. Even though that’s true, it still serves a purpose.

Include a skill soup section. A ranked list of technologies ensures you show up in keyword searches and reassures recruiters when hiring managers say, “They must know React.”

Keep it limited to what you actually know. Don’t list everything you’ve ever touched—only what’s relevant and what you can actually discuss. Some interviewers will probe for depth if you include an unholy number of skills. Also, I feel like I should charge you for my AI tokens if you include things like “HTML.”

Tailoring Your Resume Per Job

Honestly, most people don’t tailor their resume to roles because it’s just too much work. That’s probably true, especially if you’re applying to dozens of jobs.

Rather than tailoring per job, tailor for big-name companies, which are worth the extra effort, and keep a small set of versions for different role types. Perhaps you have one focused on your front-end work for tech-heavy senior IC roles, and another on your leadership skills for tech lead roles. Maybe one that highlights your AI personal projects and one that focuses on your full-time data engineering experience.

If you’re going to tailor, you need to know what to tailor to. Leverage the job description for this:

Look for repeated terms. If “distributed systems” appears three times, it matters more than “familiarity with GraphQL,” which only appears once.
Separate requirements from wish list. Most job postings have a “must have” and “nice to have” section—or they bury the real requirements in the first few bullets and pad the rest. Focus your tailoring energy on what appears at the top.
Identify the core problem they’re hiring for. “Looking for someone to own our data pipeline” means they want reliability and ownership signals. “Help us scale to 10x users” means they want performance and system design experience. Mirror this kind of language back to them.
Note specific technologies vs general skills vs filler. “Experience with Kafka” is a keyword you either have or don’t. “Experience leading teams” means they’re looking for some kind of people leadership and cross-functional signals. “Strong communication skills” is a filler that every job post includes. Don’t waste resume space on filler.

A quick trick: paste the job description into an LLM and ask, “What are the top 10 concepts in this job description and what are they looking for in this candidate?” Upload your resume and ask it to compare.

Now that you’ve identified what to focus on, here are some specific tactics to apply:

Adjust your summary. Emphasize the aspects of your experience that align with the needs of that role or company that we just discussed, identifying.
Reorder the sections. We’ll talk more about formatting below, but if you’re applying to AI roles but don’t have AI experience other than personal projects, leaving those projects at the bottom is a sure way to get ignored. Lift them up, even above the professional experience section if needed.
Reorder your bullets. Lead with accomplishments most relevant to the target role.
Expand or condense roles. Similar to reordering bullets, you can expand or condense on certain past roles that are most relevant, or even completely remove some experience if you think it’s distracting.
Match the skills. If different skills are needed, then include those or reorder them to the front of the skills listing.

Now you have the raw materials: a summary, impact-driven bullets, and content tailored to your target roles. But even the best content fails if it’s buried in a cluttered design.

The next section is about making your resume visually effortless for each type of reader…

Share this post & get rewards for the referrals.

Part 3: Designing a Resume for Ease of Use

Your resume doesn’t have to be pretty, but it needs to fulfill the visual needs of the four readers. Aim for a simple, easy-to-use formatting:

Most important stuff at the top: top of the page and the top of each new section
Optimize for a clear visual flow, top to bottom
Draw the reader’s eye to the most important parts with text size and weight. Be careful with color.
Make it easy for machines to read

Formatting for Both Machines and Humans

Use simple formatting. Avoid overly creative designs. Distribute it as a PDF that exports cleanly to text. You can test this by uploading to Google Docs—if formatting breaks, the ATS may misread it too.

Keep it to one page. Admittedly, part of this is convention, but having everything on one page makes it easy for your readers to review. It’s like having the key elements “above the fold” on a product page. Here are some space-saving tips:

Adjust margins, even all the way down to 1/2” on every side.
Think about what you can fit on one line, such as the company name, title, years.
Older or shorter experiences can sometimes be summarized in one line instead of a series of bullets. The farther the experience is down your resume, the less likely a reader is to actually read it, so feel free to reduce fidelity for those roles.
Consider a two-column format, with the lesser-read parts like contact information, skills, and education in a smaller, side column.

Use formatting to guide the eye. Sub-bullets, bolding, and visual hierarchy. If every bullet looks equally weighted, eyes glaze over.

Don’t overthink templates. You’re not going to get hired because of your amazing resume template. Don’t spend too much energy on this. Find something that matches the principles here and then put your energy into the content.

Suggested Templates

I almost didn’t include templates at all, since there are so many good ones and the final formatting choices don’t matter that much. But how can this newsletter about resumes be complete without a couple of resume templates?

Here are a couple of templates that follow our guidelines: a single-column example and a two-column example. Notice how much more vertical space you save with the two-column format.

Single Column Template

Two-Column Template

Making Your Experience Scannable

Make recognizable names visible. Make sure the eye jumps directly to household-name companies and high-impact projects. Recruiters notice brand recognition quickly, and readers only read the first part of each section.

Surface minimum qualifications. Many companies require recruiters to verify basic criteria, such as education or years of experience. Make this information prominent, as in the summary.

Prioritize your recent or flagship experience. Give more visual weight and spend more vertical space on the recent experience. Similarly, for experience at a name-brand company. Both of these will be the most-read content by reviewers. To accomplish this, compress older experience by removing bullets or shortening them.

Handling sticky situations: gaps and layoffs. Minimize these if you can. You don’t want to get screened out b/c of these before you can talk to someone:

Remove the month indicators on roles that hide an intra-year gap.
If you were recently let go, don’t update the current role with an end date yet. You can probably get away with this for 6-12 months since people expect resumes to be somewhat out of date.
If you have recent stints at startups that folded after a year, add a brief one-liner explaining what happened. Otherwise, reviewers may assume it reflects on you.

At this point, you know what to write and how to format it.

But there are a lot of common mistakes I see in resumes. The good news is they’re fixable in five minutes.

Part 4: Common Mistakes & What to Avoid

Writing resumes can be tricky, so let’s consider some common advice I give coaching clients when we’re talking about resumes…

Common Resume Mistakes

Too many buzzwords. “Results-driven,” “synergy,” “leverage” all mean nothing really. If you can swap your bullet with any other candidate’s and it still makes sense, it’s too generic.

Zero impact. Describing activities without outcomes. “Worked on the payments team” tells me nothing. What shipped? What changed because you were there?

Hard-to-read templates. Fancy designs with unusual typefaces, lots of white space, etc., that prioritize aesthetics over scannability are pretty risky unless you’re applying to a design-heavy role. A reader should be able to make a decision about your resume in about 3 seconds.

Irrelevant information. Remove that summer lifeguarding job from 2015 when you’re a senior engineer in 2026. Every line that doesn’t serve the reader is a line they might stop reading at.

Wall of text. If I can’t immediately see where the key information is and there‘s no hierarchy, no bolding, and no visual breaks, then I just get overwhelmed and close your profile.

What NOT to Include

Unnecessary personal info. Age, photo (in the US), full address. These create opportunities for bias and waste space.

Irrelevant jobs. Unless they demonstrate transferable skills or fill a gap, cut them. Nobody hiring a backend engineer cares about your retail experience from a decade ago.

Social media links. Exception: LinkedIn. Your Twitter, Instagram, or TikTok is irrelevant noise unless you’re applying to a social media company and they showcase relevant work.

References available upon request. Everyone assumes this. It’s a filler.

Hobbies and interests. Unless directly relevant (you’re applying to a gaming company and list game development as a hobby), these just take up space.

ATS tricks. Don’t try things like packing your resume with white text to rock the ATS layer. Recruiters and engineers who write ATS systems know about these tricks and despise them.

Share this post & get rewards for the referrals.

Closing Thoughts

Your resume is never done.

That might sound exhausting, but it’s actually liberating. You don’t need to craft the perfect resume before you start applying. You need a good enough resume that you improve over time.

Treat your resume like a product. We started this newsletter with that idea, and it’s worth restating here. Products ship, get feedback, and iterate; your resume should too. If an interviewer or recruiter asks you about something on your resume, consider adding it. Did you bomb a question about an accomplishment you listed? Either cut it or prepare better.

Get human feedback early and often. Have friends, mentors, or peers review your resume. If you can, get feedback from someone who’s actually hired engineers.

Iterate ruthlessly. When I’ve worked with coaching clients on their resumes, it usually takes 3-4 revisions before it’s good.

The takeaway: Your resume has four readers—machines, recruiters, interviewers, and hiring managers—and each one needs something different. Build for all of them. Make it easy to scan, with the information each persona needs. Quantify your impact. Make it scannable. And then iterate.

Now go get that interview.

👋 I’d like to thank Austen for writing this newsletter!

Also, subscribe to his free newsletter, Mastering Behavioral Interviews, and check out his book by the same name.

It will help you prepare for the interview round you’re underinvesting in, the one that separates Senior engineers from Staff and above. He’s got a lot of practical advice similar to what he shared in this post.

Remember, you can claim 20% off Premium Access to his interview prep content with code SDN26, but only for a few days!

If you find this newsletter valuable, share it with a friend, and subscribe if you haven’t already. There are group discounts, gift options, and referral rewards available.