• Share this post & I'll send you some rewards for the referrals.

• Block diagrams created using Eraser.

A payment backend is one of the few systems where a bug doesn’t just cause a bad user experience but real money to vanish from user accounts.

This makes payment system design fundamentally different from most backend engineering…

The core challenge here is NOT performance or feature richness, but correctness under failure. Every design decision, from database choice to retry policy, must answer one question: what happens when this component fails mid-transaction?

This is a complete system design blueprint for building a payment backend that integrates Stripe as the payment service provider.

Onward.

The agent harness wasn't supposed to be the black box (Partner)

Agent loop is the most important piece of infrastructure in your workflow right now, and for most developers, it’s the one piece they can’t open up.

Agent builders have to jump through all the hoops themselves, crafting the infrastructure and tools, testing the harness, while fighting to maintain what they’ve built.

Meet Cline SDK: agent harness behind Cline 2.0, fully open-sourced. The same runtime that powers Cline across VS Code, JetBrains, and the CLI is now an npm install away: npm i @cline/sdk. Inspect it, fork it, extend it, ship on it.

• Best-in-class harness: 74.2% on Terminal-Bench 2.0 with Claude Opus 4.7 ahead of Claude Code (69.4%) and strongest numbers published on open-weight models.

• Open model & provider choice: Anthropic, OpenAI, Google, Bedrock, Mistral, or any OpenAI-compatible endpoint.

• Real plugin system: Register tools, hooks, commands, providers, message builders. Prototype as a local file, harden into a package. Extend it freely for any of your agent use cases.

• Scheduled + event-driven agents: Cron and event specs for PR reviews, dependency checks, coverage audits, changelogs no separate orchestration layer.

Stop building around your agent. Start building on it.

Install Cline SDK today: npm i @cline/sdk Or try the rebuilt harness directly: npm i -g @cline

Get Started Today

(Thanks to Cline for partnering on this post.)

I want to reintroduce Hayk Simonyan as a guest author.

He’s a senior software engineer specializing in helping developers break through their career plateaus and secure senior roles.

If you want to master the essential system design skills and land senior developer roles, I highly recommend checking out Hayk’s YouTube channel.

His approach focuses on what top employers actually care about: system design expertise, advanced project experience, and elite-level interview performance.

Inside this newsletter, you’ll get:

• Three approaches to accepting payments. Why no-code checkout hits limits fast, when building your own processor makes sense, and why integrating a PSP like Stripe is the right call for most companies.

• How money actually moves. The full card transaction lifecycle from authorization to settlement, including the six entities involved and the engineering risks most developers miss.

• High-level architecture. The seven components of a production payment backend, how they split the synchronous and async paths, and why each boundary exists.

• Idempotency and exactly-once processing. How to design a system that never double-charges, even when servers crash mid-transaction, using idempotency keys, recovery points, and the atomic phases pattern.

• Webhook handling and the payment state machine. How to process Stripe webhooks reliably when duplicates and out-of-order delivery are expected, and how to enforce valid state transitions at the database level.

• Designing for high availability. How to approach 99.999% uptime across your API layer, database, and async workers, and why a circuit breaker in front of Stripe protects your own system more than it protects Stripe.

Three Approaches to Accepting Payments

Before diving into the design, it’s worth understanding the landscape…

When a company needs to accept payments, there are three broad approaches, and choosing the wrong one for your scale wastes either years of engineering time or millions in unnecessary fees.

Option 1: Build your own payment processor

This means connecting directly to card networks (Visa, Mastercard), acquiring banking licenses, handling PCI DSS1 (Payment Card Industry Data Security Standard, which is the set of security requirements every company that touches card data must comply with), Level 1 compliance2, building your own fraud detection, and managing relationships with issuing banks.

The upside is lower transaction costs.

At a massive scale, even saving 0.5% off each transaction adds up to hundreds of millions per year. Amazon, Uber, and Airbnb have all moved in this direction for parts of their payment stack.

The downside is the cost to get there: acquiring necessary licenses alone takes 12-24 months, costs millions, and comes with ongoing regulatory obligations. So this approach only makes sense once you are processing billions in annual payment volume.

For anyone else, it is premature optimization at its extreme…

Option 2: Use a payment service provider (PSP) like Stripe, PayPal, Adyen, or Braintree

A PSP is a company that handles the entire payment processing chain on your behalf: card network integrations, banking relationships, fraud tooling, PCI DSS compliance, and global payment method support.

You pay a per-transaction fee, typically around 2.9% + $0.30 with Stripe, in exchange for not having to build or maintain any of that infrastructure. This is the right approach for most companies, from early-stage startups to large-scale platforms.

Even Shopify, which processes billions in payments annually, built its own payment product (Shopify Payments) on top of Stripe’s infrastructure rather than connecting directly to card networks.

Option 3: Use a no-code payment flow, such as Stripe Checkout

Stripe offers hosted checkout pages and payment links that require almost no backend integration.

You create a checkout session, redirect the user to Stripe’s hosted page, and Stripe handles everything: UI, payment method selection, and confirmation.

This is the fastest way to accept payments, and it works well for simple use cases: selling a single product, collecting donations, or building a quick prototype.

The trade-off is limited customization and less control over the payment experience.

You cannot deeply embed it in your own UI, easily build custom subscription logic, or react to payment events in real time with full control over the flow.

How Money Moves Through Card Network

Understanding the payment flow end-to-end is a prerequisite to designing a backend that handles it correctly…

Six entities participate in each card transaction:

1. Cardholder

2. Merchant (Business or individual selling the product or service)

3. PSP (Payment Service Provider, in this case, Stripe)

4. Acquiring bank (Merchant’s bank)

5. Card network (Visa/Mastercard)

6. Issuing bank (Cardholder’s bank)

Money flows through 3 distinct phases, each with different timing and failure modes…

Phase 1: Authorization (1-3 seconds)

The customer submits card details through Stripe Elements.

These are technically iframes hosted by Stripe, even if they look like part of your site. We use Stripe.js, which is a JavaScript library that Stripe requires you to load directly from their servers (for security reasons).

The process starts on the backend, where a PaymentIntent3 gets created via Stripe’s API to track the transaction lifecycle. This returns a client_secret to the frontend.

The customer then enters their card details into Stripe Elements - secure iframes hosted by Stripe that look like part of your site. Using Stripe.js, the client-side code tokenizes the card details so the PAN (Primary Account Number: a 14 to 16-digit number on a credit or debit card) never reaches the merchant’s server.

What is PaymentIntent?

A PaymentIntent is a stateful object Stripe uses to track a single payment from start to finish.

Think of it as the source of truth for a transaction…

In older systems, you’d just send a “charge” request and hope it worked. With modern payments, things are more complex because of 2FA (e.g., 3D Secure4) or fraud checks.

PaymentIntent manages this by moving through different states:

• Requires payment method: You created the intent, but the user hasn’t typed their card yet.

• Requires action: User must verify the payment in their bank app.

• Processing: Stripe is talking to the banks.

• Succeeded: Money is authorized or captured.

Stripe formats an ISO 8583 authorization message5 and sends it to the acquiring bank6, which identifies the card network via the BIN7 (first 6-8 digits) and forwards the request.

The card network routes it to the issuing bank, which checks card validity, available funds, CVV/AVS8 match, and fraud risk models.

• The issuer returns a two-digit response code, approve or decline, back through the entire chain.

• If approved, a hold gets placed on the cardholder’s available balance. But no money moves yet.

Mastercard reports an average network response time of 130 milliseconds; the full round-trip, including all hops, completes in 1-3 seconds.

Visa’s network handles a peak of 56,000+ messages per second.

Phase 2: Capture (immediate/delayed)

Capture9 is when the merchant tells acquiring bank (the merchant’s bank) to finalize the authorized amount.

For digital goods and subscriptions, capture occurs immediately. Stripe’s default is capture_method: automatic.

For physical goods, hotels, or ride-hailing, capture is delayed until fulfillment. You can capture an amount less than or equal to what was authorized, but never more. Authorization holds10 typically last 5 to 10 days.

Visa allows 10 days for standard e-commerce and up to 31 days for lodging.

Here’s the Engineering Risk:

If you don’t capture within 7 days, the issuing bank might cancel the hold. This is a common source of bugs. Your state machine must treat AUTHORIZED and CAPTURED as “distinct” states. If your system thinks a payment is authorized but the bank has dropped the hold, the capture call will fail.

So you need a background job to detect these “stuck” payments and mark them as expired.

Phase 3: Clearing and settlement (T+1 to T+3)

At the end of the business day, captured transactions get batched and sent to the card network for clearing.

The network calculates fees and exchanges transaction files overnight…

Settlement is when the actual money moves. The issuing bank sends the funds (minus fees) to the card network, which then passes them to the acquiring bank.

The acquiring bank finally deposits them into the merchant’s account.

In banking, “T” stands for the Transaction Date. T+2 means the money arrives two business days after the transaction. Stripe’s US default is T+2, though new accounts usually have an initial hold of 7 to 14 days.

Fee Breakdown

On a $100 US online transaction, the 2.9% + $0.30 fee gets split three ways:

• Interchange11 (~$2.05): This goes to the Issuing Bank (the customer’s bank). This is the largest cut. It covers the bank’s risk and pays for the customer’s credit card rewards.

• Assessment (~$0.16): This goes to the Card Network (Visa or Mastercard) for using their rails.

• Markup (~$0.70): This is what Stripe keeps for providing the API, security, and infrastructure.

Functional and Non-Functional Requirements

Functional requirements for a payment backend serving a mid-to-large platform fall into 6 categories:

1. Customers must be able to make one-time payments and save payment methods for reuse.

2. Merchants or the platform must be able to accept payments, with support for marketplace-style split payments if needed.

3. Refunds, both full and partial, must flow through a dedicated refund state machine.

4. System must handle subscriptions and recurring billing, including proration on plan changes, trial periods, and Stripe’s Smart Retries for failed renewal payments.

5. Backend must process Stripe webhooks reliably, as webhooks are the source of truth for payment status.

6. The entire payment lifecycle must be modeled as a finite-state machine with enforced transitions.

Non-functional requirements are where payment systems diverge sharply from typical backend services:

• Exactly-once payment processing is the key requirement. A double charge on a $1,000 purchase means money taken from a person twice. True exactly-once delivery is impossible in distributed systems (Two Generals Problem12), so the practical implementation is at-least-once delivery combined with idempotent processing and reconciliation.

• High availability targets are extreme. Stripe itself maintains 99.999% API uptime.

• Idempotency must be enforced at every layer: client-to-backend, backend-to-Stripe, and webhook processing. Every retry must produce the same result as the original request.

• Consistency over availability. Payment systems are one of the few domains where the CAP theorem should tilt decisively toward consistency. A stale read that shows an incorrect balance or a lost write that drops a payment is far worse than brief unavailability. This is why every major payment platform, including Shopify, Uber, and Airbnb, uses SQL databases with ACID guarantees for core payment data.

Scale estimates for a mid-to-large platform

A platform processing 100,000 payments per day generates roughly 500,000-1,000,000 webhook events daily (each payment triggers 5-10 events across creation, authorization, capture, and related objects).

Stripe’s default API rate limit is 100 requests/second per account, with individual endpoints limited to 25 requests/second; higher limits are available by arrangement.

Payment API latency should be under 5 seconds end-to-end, including the Stripe round-trip, with internal service-to-service calls under 100ms.

High-Level Architecture

The architecture splits into seven components, each with a distinct responsibility.

It isolates the synchronous customer-facing path from asynchronous processing and keeps the webhook ingestion pipeline decoupled from business logic.

1 Payment API Service

It’s the synchronous entry point.

It receives payment requests from checkout, validates input, checks idempotency keys13 against the database, creates or retrieves the payment record, and returns an immediate response to the client.

For Stripe integrations, this service creates a PaymentIntent and returns the client_secret to the frontend, which uses Stripe.js to complete the payment (including 3D Secure challenges).

The API service should never block on downstream processing. It creates the payment record, dispatches work, and returns.

2 Stripe Integration Layer

It abstracts all Stripe-specific API calls behind a uniform interface.

This adapter handles Stripe’s error types, maps them to internal error codes, attaches idempotency keys to every POST request, and manages timeouts. Shopify and Airbnb both use this pattern; Airbnb calls it the “PSP adapter,” which isolates provider-specific logic from the core payment domain.

If you ever need to support a second PSP (Adyen, Braintree), only this layer changes.

3 Payment Database

This is the system of record.

It stores the current state of every payment, idempotency keys, and the immutable audit log. The schema design is covered in the next section, but the critical principle is: payments table holds a mutable current state (optimized for queries), while the payment events table holds an append-only immutable log of every state change (optimized for audit, debugging, and reconciliation).

Both exist in the same SQL database and get updated within the same transaction.

4 Webhook Receiver

It’s a lightweight HTTP endpoint that does exactly three things: verify the Stripe signature (HMAC-SHA25614 using the Stripe-Signature header), store the raw event, and return 200 OK.

It must respond within seconds, since Stripe has an approximately 20-second timeout. All business logic happens asynchronously. The receiver enqueues the event onto the message queue for processing by workers.

5 Message Queue

It (Kafka, SQS, or RabbitMQ) decouples webhook receipt from processing and provides at-least-once delivery guarantees.

Uber uses Apache Kafka as the backbone of their payment platform’s async stream processing. The queue also supports the transactional outbox pattern15: when a payment state change is committed to the database, an outbox record is written in the same transaction, then relayed to the queue by a separate process.

6 Background Job Workers

They handle 4 categories of async work:

• Webhook event processing: dequeuing events and applying state transitions idempotently,

• Retry workers: retrying failed PSP calls with exponential backoff,

• A reconciliation worker: daily comparison of internal records against Stripe’s records,

• And a stuck-payment detector: alerting on payments in intermediate states beyond a configurable threshold.

7 Ledger

It provides double-entry bookkeeping16.

Every money movement is recorded as a balanced pair of debit and credit entries. Uber explicitly built its next-generation payment platform on double-entry bookkeeping for auditability, and Stripe’s own ledger system logs approximately 5 billion money-movement events daily.

The ledger is append-only.

If a mistake needs correction, a new reversing entry is inserted, never an update or delete.

The high-level architecture tells us what to build, but database design is where correctness is either enforced or lost. Every pattern we covered above, idempotency, exactly-once processing, the state machine, all of them depend on the database schema doing the right thing.

If we get this wrong, no amount of application-level logic will matter.

So let's see what the database design looks like…

Database Schema to Enforce Correctness

Read more

• Share this post & I'll send you some rewards for the referrals.

You ask ChatGPT to write an email.

It drafts it… But you still have to copy the text, paste it into Gmail, and hit send. AI did the thinking,,,YOU did the task.

OpenClaw changes this…

It’s an open-source autonomous AI agent1 that runs on your machine, connects to your messaging apps, and does the work itself. You could message it on Telegram: “Check my inbox, pull any invoices, save the attachments.” Then it connects to your email, finds invoices, downloads the attachments, saves them, and messages you back with a summary.

You don’t have to open a single app…

OpenClaw bridges the gap between the thinking layer and the act layer.

Large language models (LLMs), such as Claude, ChatGPT, and Gemini, provide reasoning. While OpenClaw2 provides hands.

Onward.

Run OpenClaw with MyClaw

The best AI assistants aren’t the ones waiting for your next prompt—they’re the ones that send you the answer before you think to ask.

MyClaw runs OpenClaw in the cloud, schedules recurring tasks, and delivers updates directly to Telegram or WhatsApp, so it feels less like another chatbot and more like an assistant that’s actually on the job.

If this newsletter got you interested in OpenClaw, MyClaw is an easy way to see what an always-on AI assistant feels like.

Try MyClaw

(Thanks to MyClaw for partnering on this post.)

Here’s what you’ll find inside this newsletter:

• What OpenClaw actually does. Why it’s different from every AI tool you’ve already tried.

• How the Gateway works. The always-on server running on your machine that makes everything else possible.

• Skill system. How to install the right ones first, and what most people get wrong.

• What breaks and why. Most common failure modes, with fixes for each.

• The real security risks. Documented, avoidable, and worth reading before you touch a single setting.

• How to build your own. Full architecture in five layers, from the ground up.

By the end of this newsletter, you will understand OpenClaw well enough to run it.

What Is OpenClaw

OpenClaw is the orchestration layer, the connector between an LLM (like Claude, GPT, or Gemini) and your real-world tools.

The intelligence comes from the LLM you connect to it.

Think of the LLM as the brain: it reasons, plans, and decides what to do. OpenClaw is the hand: it executes those decisions across your files, messaging apps, shell, and APIs.

Here’s what OpenClaw offers:

• A gateway3 that runs on your machine (Mac, Linux)

  This is the always-on server running on your machine that coordinates everything: your messaging apps, LLM, tools4, and memory.

• A multi-channel messaging interface: WhatsApp, Telegram, and so on.

  You talk to the same assistant from all these apps, with shared context and memory, instead of managing a different bot per platform.

• A skills platform with 5,400+ modular capabilities on ClawHub5.

  Skills6 are plain Markdown files that tell the agent how to perform a specific task, like checking your GitHub PRs or sending a Slack message.

• A persistent memory system in plain Markdown files you own.

  The agent’s knowledge, preferences, and logs are stored as editable Markdown on disk, so you can inspect, version-control, and move its LLM context like code.

• Proactive scheduling engine (Heartbeat7 + cron jobs) that acts without being prompted.

  OpenClaw can wake itself up on a schedule to check servers, triage email, or send summaries, instead of waiting for you to type a command.

• Model-agnostic: plug in Claude, GPT, DeepSeek, Gemini, Grok, or local models via Ollama8.

  You can mix and match models (cheap vs powerful, cloud vs local) and route different tasks to different providers to optimize cost and performance.

How It Differs From Other AI Tools

Here’s how:

• ChatGPT, Claude, Gemini: You prompt them, and they respond. They have no access to your email, no ability to run scripts, and no memory once you close the tab9.

• Siri, Google Assistant: Polished products, but closed ones. You can’t add custom skills, connect arbitrary APIs, or self-host them.

• LangChain, CrewAI, AutoGen: Developer frameworks. You write the Python, define the chains, and build the interfaces from scratch. OpenClaw ships all of that out of the box.

• Claude Code, Codex CLI: Terminal tools for coding sessions. Reactive and session-based. OpenClaw runs 24/7 for everything, not just code.

So OpenClaw is NOT just a product you use in a browser tab.

It is a service that runs on your machine and continues to work while you’re away.

Share

How OpenClaw Processes Messages

Imagine you send a single Telegram message at 9 pm:

“Check my open PRs and let me know if anything needs a response tonight.”

You lock your phone and go to sleep…

By morning, you will have a reply:

Three PRs are open, and one has a teammate's review comment asking about error handling. OpenClaw drafted a response and flagged it for your approval, leaving the other two untouched because they were still in CI. It also remembered that last week you said you prefer to handle auth-related reviews yourself, so it left the one in the auth module alone.

None of this required you to open GitHub, write a prompt, or copy anything between tabs…

Why OpenClaw Runs 24/7 on Your Machine

When you run the OpenClaw gateway, you start a Node.js server that binds to port 18789 and stays running.

This is the fundamental difference between OpenClaw and a browser-based AI tool…

Every mainstream AI assistant, including ChatGPT, Claude, and even DeepResearch mode, only exists while you have a tab open. (DeepResearch can run a long session, but once it’s done, it stops.)

i.e., if you close the tab, everything disappears.

But OpenClaw never stops:

It runs as a background daemon on your local server and stays alive whether you are at your desk or asleep. That persistence is the foundation of everything else: scheduled jobs, proactive alerts, multi-channel messaging, and long-term memory. None of it works without a process that is always on.

This central server is called the Gateway…

Gateway: Control Center of Your Agent

The Gateway orchestrates everything.

It sits between your messaging apps, LLM, and tools, coordinating every request from the moment it arrives until a reply goes back out.

When a message arrives from Telegram, WhatsApp, or Slack, the Gateway authenticates the sender and creates or retrieves a session for the conversation.

It then assembles the full context10, passes it to the LLM, handles any tool calls that come back, and routes the final reply to the correct channel.

Say you ask the agent to clean up old log files:

Before it runs anything, the Gateway intercepts the shell command and sends you a confirmation request through Telegram: “I’m about to run $ rm -rf ~/logs/*.log, approve this?”

Nothing executes until you say yes… If you don’t respond, it waits…

The Gateway also serves the Control UI at http://localhost:18789.

It’s a browser-based dashboard that lets you watch the agent’s reasoning in real time, approve pending commands, and inspect active sessions. Every tool call, model response, and approval request shows up here live. Since the Gateway runs on your Mac Mini or VPS, you can access the Control UI from any browser on the same network. If you use Tailscale11, you can reach it from your phone or laptop without being on the same network.

The Gateway also supports hot configuration reloads…

You can change your model, update permissions, or add a new channel without restarting the process. The running server picks up the changes and applies them immediately.

Share

Channel Layer: How OpenClaw Talks to Messaging Apps

OpenClaw connects to Telegram, WhatsApp, Slack.

Each of these works completely differently at the protocol level:

• Telegram uses a bot token with polling or webhook delivery.

• WhatsApp uses a reverse-engineered web protocol; it requires QR-code authentication and stores session state locally.

• Slack uses Socket Mode with two separate token types.

• Discord uses a gateway WebSocket with its own heartbeat requirements.

Without a dedicated abstraction layer, each new platform requires touching the routing logic, session management, and message pipeline.

OpenClaw solves this with a Channel Layer:

It treats each platform as a separate plugin acting as a translation adapter. Each adapter translates incoming messages into a standard internal format: a stable identity key, content, metadata, and attachments. Only then does it pass anything to the Gateway. Everything after that boundary is platform-agnostic. The agent has NO idea whether your message came from Telegram or Slack.

It sees a normalized message and responds accordingly…

This also means security policies like allowlists, pairing workflows, and mention-gating apply uniformly across all platforms rather than being reimplemented per channel.

LLM Layer: Where Reasoning Happens

OpenClaw is a brain socket.

You provide the intelligence by connecting an API key. And it provides the infrastructure for everything else.

It means you can use Claude, GPT-4, Gemini, DeepSeek, or a locally run model via Ollama. Plus, you can change models at any time without any reconfiguration.

When a message arrives at this layer, OpenClaw assembles a system prompt by combining the agent’s configured personality, list of available skills, relevant memory retrieved from previous sessions, and current conversation history.

It then calls the AI model with the full context.

If the model decides it needs to take an action, run a shell command, search the web, read a file, or call an API, it signals that as a tool call. OpenClaw executes the tool, captures the output, and feeds the result back into the conversation. The model reads the result and decides what to do next: call another tool, or produce a final answer.

This loop continues until the model generates a response with no further tool requests.

You can watch this entire loop unfold in real time in the Control UI at every tool call, every model response, every intermediate step…

And when a conversation reaches the model’s context limit, compaction12 triggers automatically. The agent summarizes the conversation so far and replaces the raw message history with that summary, preserving continuity without the cost of loading thousands of tokens on every reply.

Share

Plugin System: How OpenClaw Stays Modular

Almost everything in OpenClaw that is not part of the core engine is implemented as a plugin13:

• Telegram integration is a plugin.

• GitHub access is a plugin.

• The memory system is a plugin.

This is what makes OpenClaw easy to extend and customize…

When OpenClaw starts, it reads your configuration and loads only the plugins you have enabled. Disabling a capability means changing one line in your config, while adding a community plugin is as simple as placing it in the right directory.

If you want to write your own, you just implement a defined interface and never touch the core engine.

Plugins can hook into lifecycle events at every meaningful point: before and after tool calls, before and after messages, at session start and end.

This is how you add things like audit logging, rate limiting, or custom approval flows without touching the core agent at all…

Memory System: How Agent Remembers You

The memory solution (loading the entire conversation history into every prompt) fails quickly.

A week of active conversation will overflow any context window… Even if it fits,,, loading all of it on every reply is expensive and slow…

OpenClaw treats short-term continuity and long-term recall as separate problems that need different solutions:

• Short-term memory keeps the current conversation on track. Every session is saved as a file on disk. When you send a message, OpenClaw loads only the recent part of that session into context, just enough to keep things flowing without loading everything at once.

• Long-term memory handles recall across days and weeks. All past sessions are indexed into a local database. Before the agent answers any message, the memory indexer searches the database and pulls in any relevant information automatically. If a conversation from five days ago is relevant, it gets included. You never have to remind the agent of something it already knows.

• Standing notes are a third layer. These are Markdown files you write yourself, stored in ~/.openclaw/workspace/memory/. Think of them as a permanent briefing document. Your preferences, your projects, your context. The agent reads them every time.

How OpenClaw Handles Tasks That Take Hours

Most interactions with OpenClaw are fast… You send a message; the agent calls a tool, and replies in seconds.

But some tasks take much longer: scraping hundreds of pages, running a full test suite, processing a large dataset, or coordinating a multi-step workflow across several APIs.

OpenClaw handles these differently:

When a task is going to take a while, the agent does NOT hold the conversation open and make you wait. It acknowledges the task, starts working, and messages you when it’s done. You can lock your phone and walk away.

The Gateway keeps running and messages you when the task is done.

For tasks that span multiple sessions or require human input mid-way, the agent completes what it can, writes the current state to MEMORY.md or a workspace file, and then pauses. When you return and pick up the conversation, it resumes from the saved state.

NO context gets lost between sessions because important state was written to disk, not held in memory.

For truly long-running background jobs, cron jobs in isolated session mode are the right tool. Each run gets its own session, completes its work, delivers output to your channel, and exits cleanly. The next scheduled run starts fresh.

This keeps long-running automation from accumulating stale context over time.

One practical limit worth knowing: if a single tool call takes too long, Gateway will time out waiting for a response.

For tasks like large web scrapes or slow API calls, break the work into smaller chunks. Instruct the agent to process in batches and report progress between each batch, rather than attempting everything in a single tool call.

Ready for the best part?

How a Message Travels End to End

Read more

• Share this post & I'll send you some rewards for the referrals.

If you ask questions about your company’s policies, a contract you have open in another tab, or anything outside training data, an AI personal chat assistant will either say “I don’t know” or invent something that sounds right.

The most direct fix is to include the relevant documents in the prompt.

But this only works if the documents fit within the context window, and most real knowledge bases do not. Company policies, contracts, and internal wikis are lengthy, and loading them in full for every query can be impractical because of latency, cost, and context window limitations.

Retrieval-Augmented Generation (RAG) is a more practical approach.

Instead of loading the entire knowledge base into every prompt, the system retrieves only the most relevant passages at query time, includes them alongside the question, and the model generates a cited answer from that material.

It is the architecture behind Perplexity, ChatGPT with Search, GitHub Copilot’s codebase search, and most internal AI tools inside large companies.

Onward.

Code review for AI-native engineering teams — at 1M+ LOC (Partner)

Pull request review was built for human authors.

AI-written code fails differently, and most reviewer tools never caught up.

AgentField just open-sourced a multi-agent code reviewer built for AI-native teams:

• Runs on open or closed models — Kimi, DeepSeek, Claude — so it scales to whatever a team can afford.

• Drops into GitHub Actions in minutes.

• Built to make deep code review economically viable at scale.

The architecture rationale — the four jobs of code review, and which three became load-bearing once AI took the first — is in AgentField’s latest post.

Star & Deploy

I want to reintroduce Louis-François Bouchard as the author of this newsletter.

He’s a best-selling author (Building LLMs for Production), the co-founder of Towards AI, and the creator of the YouTube Channel, What’s AI, where he helps people understand AI and learn how to apply it in the real world.

Through his development work with clients and his content, teaching, and AI training programs on the Towards AI Academy, Louis focuses on making AI practical for builders, engineers, and curious learners alike.

At Towards AI, he and his team train AI engineers through courses built for every stage, from beginner to advanced. That educational mission and the real-world experience building for his clients are exactly why I wanted him in this newsletter series.

Here’s what’s inside this newsletter:

• The full RAG architecture. How a question becomes an embedding, how the system finds the right chunks across a large knowledge base, and how the model generates an answer grounded in retrieved sources.

• Document ingestion and chunking. What makes a chunk useful, three strategies for splitting documents, and the metadata that makes citations possible.

• Retrieval and generation in production. Hybrid search, reranking, query transformation, system prompts, citation strategies, and how to make the system decline questions it cannot answer rather than guess.

• Failure modes and evaluation. The three ways RAG breaks in practice, the four RAGAS metrics that surface them, and how to build a test set that catches regressions.

• Production concerns. Caching, access control, monitoring, and cost at scale.

• A practical eight-step build. A folder of PDFs turned into a working Q&A system with hybrid retrieval, reranking, structured citations, and a quality scorecard.

Why Build Your Own Knowledge Q&A System?

Perplexity, ChatGPT with browsing, and Gemini with Search already answer questions with cited sources.

They are capable and improving fast.

But the limitation is that they are built for public knowledge. The moment your questions involve internal documents, regulated data, or a knowledge base within your infrastructure, off-the-shelf tools hit hard constraints.

Here is where they fall short:

• Off-the-shelf tools cannot access regulated data. You can upload personal files into a ChatGPT project and get useful answers. You cannot do that with customer records, contracts, medical files, or anything covered by HIPAA, GDPR, or SOC 2. That data has to stay inside your own perimeter, behind your own access controls, with your own audit trail.

• Your knowledge is not in a folder you can drag and drop. It lives in Confluence, in a Drive folder with thousands of contracts, in a support tool, or in an internal wiki. Off-the-shelf assistants can connect to some of these, but the data flows out of your infrastructure and into theirs, and is processed under their privacy policy. A system built on your infrastructure reads directly from those sources, using your auth, and ensures data never leaves your perimeter.

• Retrieval happens at query time, so answers stay current. A model’s training data has a cutoff, but your documents keep changing. Because RAG retrieves at query time, a document updated after the model was trained is still used to answer the next question, with no retraining required.

• You control what the system knows. When a document is added to the knowledge base, the system uses it for the next query. When it is removed, it is no longer used. When it is updated, the next answer reflects the new version. This control is what lets you curate, version, and audit the knowledge base, and it is what regulated industries require.

• Focused retrieval is more accurate and cheaper. A model answering a question from five selected passages is working with a much smaller, more relevant input than one drawing on its full training data. More relevant context produces more accurate answers, and shorter prompts cost less per call.

These five constraints all point to the same architecture.

The rest of this newsletter builds on it…

Part 1: Turning Documents into Searchable Knowledge

A RAG system has two stages:

• First runs once, or whenever new documents are added: it processes your documents, converts them into vector representations, and stores them in a database that can be searched by meaning.

• Second runs on every user query. This part focuses on the first: ingestion, chunking, embedding, and storage.

Document Ingestion and Preprocessing

Before anything can be retrieved, your documents need to be readable by the system.

That sounds straightforward, but documents come in many formats: PDFs, HTML pages, Word files, Markdown exports from Notion, Confluence pages, and scanned forms. Each format stores text differently and needs its own parser.

A poor extraction at this stage degrades everything downstream, so it is worth getting right.

There are five things to handle during ingestion:

1. Text extraction. As text is extracted, the structural information alongside it is preserved: headings, page numbers, and table boundaries. This is what allows the system to later cite not just a document, but the specific section and page a passage came from.

2. Tables, code, and images. These elements break when treated as regular text. A table flattened into a paragraph loses its row-column relationships. Code blocks lose meaning when indentation and line breaks are stripped. Images contain no text at all unless a caption is present or OCR is run on them. Each is typically handled through a separate extraction path.

3. Scanned PDFs. A digital PDF stores text as text. A scanned PDF stores it as an image, which a regular parser cannot read. Optical Character Recognition (OCR) is used for converting scanned documents into readable text. But OCR takes several seconds per page, which adds time to the ingestion stage. Tesseract is the standard library for clean scans. Vision-language models work better on low-quality images and complex layouts.

4. Duplicates. When the same document exists in multiple folders, it can be indexed multiple times, which causes retrieval to return identical chunks. A common approach is to hash each document’s content and skip any document whose hash has already been indexed. For near-duplicates such as different versions of the same contract, fuzzy text matching is used instead.

5. Boilerplate. Headers, footers, navigation menus, and copyright notices add noise without adding information. These are typically identified by how frequently they repeat across pages and stripped before the text is passed downstream.

Clean text is necessary but not sufficient.

Before it can be embedded and searched, it has to be cut into pieces small enough for the embedding model to represent accurately.

Chunking

A long document cannot be embedded as a single vector.

An embedding that represents 50 pages of mixed content does not align well with a specific question. Instead, each document is split into smaller pieces called chunks, and each chunk is embedded individually.

Different chunk sizes have different tradeoffs…

Larger chunks carry more context but produce less specific embeddings. Smaller chunks produce more specific embeddings but may lack the surrounding context needed to fully answer a question. A well-formed chunk is self-contained, covers a single topic, and includes its section heading and document title.

Here are three common strategies for splitting documents into chunks:

1. Fixed-size chunking splits the text into chunks of N tokens, with a fixed overlap, usually 50 to 100 tokens. The overlap ensures that a sentence cut at a chunk boundary still appears in full in one of the two adjacent chunks. It is the simplest approach and works well on uniform text such as articles and documentation.

2. Recursive chunking splits on paragraph boundaries first, then sentences, then words, going to a finer level only when a chunk is still too large. It follows the structure already present in the document rather than imposing an arbitrary boundary. Most popular RAG libraries, including LangChain and LlamaIndex, use this as their default.

3. Semantic chunking splits a document into sentences, then forms small groups of consecutive sentences and creates an embedding for each group. The embedding of each group is compared with that of the previous group, and a chunk boundary is placed where the difference between the two embeddings is large. The assumption is that a large change in the embedding indicates a change in topic. Semantic chunking performs better than the other methods when topics shift within paragraphs rather than across them, but it requires an embedding call for each sentence group at ingestion time, making it the most expensive of the three.

A common starting point is recursive chunking at 500 to 800 tokens with a 50- to 100-token overlap.

Each chunk is tagged with metadata: source document, page number, section heading, and last-modified date. This metadata is what the system uses to produce citations later.

The chunks are now ready to be searched. Searching by exact word match would miss most relevant results, since users rarely use the same words that documents do.

Each chunk is therefore converted into a numeric vector that captures its meaning…

Embeddings

These vectors are called embeddings: fixed-length vectors of numbers that represent the meaning of a piece of text.

Two chunks with similar meaning produce vectors that are close to each other; two chunks on different topics produce vectors that are far apart.

This is what allows a query like “reset my password” to retrieve a document titled “account credential recovery,” even though the two share no words in common.

Some commonly used embedding models in 2026 include OpenAI’s text-embedding-3-large, Cohere’s embed-v4, and open-source options such as bge-large and e5-mistral.

The typical starting point is an API model, with a move to self-hosted open-source if data residency, privacy, or cost becomes a constraint. For specialized domains such as legal, biomedical, or financial text, a domain-specific model generally outperforms a general-purpose one.

The MTEB benchmark is a useful reference for comparing models.

Embeddings have three limitations worth understanding before choosing a retrieval strategy:

• Negation. “Drugs that don’t interact with warfarin” and “drugs that interact with warfarin” produce nearly identical embeddings. The word “don’t” has very little effect on the resulting vector, even though it inverts the meaning.

• Exact identifiers. Order numbers, error codes, and SKUs such as E-4012 carry no semantic meaning, so similarity searches cannot reliably match them. Hybrid search (covered in Part 2) is used in these cases.

• Numeric filters. A query like “customers with revenue above $5M” is a structured filter, not a similarity match. Structured retrieval (covered in Part 2) is used for these queries.

Another operational constraint is that the model used to embed the chunks must also be used to embed queries during retrieval.

The two share a single vector space, and switching models means re-embedding the entire corpus.

The choice of embedding model is effectively a long-term commitment…

Vector Databases

Once the chunks are embedded, they need to be stored in a system that supports fast similarity search.

A vector database stores each chunk’s embedding alongside its metadata and returns the closest matches to a query vector. The metadata, source document, page number, section heading, and last-modified date are returned with each result, so the system can point the user back to the source.

The basic operation is similarity search: given a query vector, return the K vectors in the database that are closest to it.

Closeness is usually measured using cosine similarity, which compares two vectors based on the angle between them.

A direct implementation of this compares the query vector against every vector in the database. This works for small databases of a few thousand vectors, but the cost grows linearly with the database size and becomes impractical at scales of millions of vectors.

Production systems use approximate nearest neighbor (ANN) search, a family of algorithms that returns close-to-exact results at a fraction of the cost. ANN trades a small amount of recall for a large reduction in latency, often by orders of magnitude, with the exact tradeoff depending on the dataset, the index, and the recall target.

Hierarchical Navigable Small World (HNSW) and Inverted File (IVF) are two of the most commonly used ANN index types. Some managed vector databases abstract this choice entirely, while others let you pick the index and tune its parameters.

Here are three deployment options, each suited to different constraints:

• Managed services such as Pinecone, Turbopuffer, and Weaviate Cloud. The service runs the infrastructure and exposes an API. This is appropriate when the team does not want to operate the database directly.

• Self-hosted open-source options such as Weaviate, Qdrant, and Milvus. These run on your own infrastructure and offer full control over configuration, scaling, and metadata filtering.

• Database extensions such as pgvector for Postgres. These add vector search to an existing relational database. A practical option when the stack already runs Postgres, and the corpus stays under approximately one million chunks.

With the index built and the chunks stored, the offline pipeline is complete. The next question is how the system uses it when a user actually asks a question.

Reminder: this is a teaser of the subscriber-only newsletter series, exclusive to my golden members.

When you upgrade, you’ll get:

• Simple breakdown of real-world architectures

• Frameworks you can plug into your work or business

• Proven systems behind ChatGPT, Perplexity, and Copilot

Unlock Full Access

Part 2: Finding the Right Information

This part covers how the system returns the chunks that contain the answer…

Basic Retrieval

The default retrieval approach has three steps: embed the question using the same model used to embed the chunks, run a similarity search against the vector database, and return the top K results.

K is a tradeoff parameter.

Too small, and the system misses chunks that contain the answer. Too large, and the prompt fills with irrelevant material, which reduces answer quality and increases cost. A reasonable starting value is between 5 and 10.

The right value for your system is determined over time by testing on real queries. It depends on the expected answer length and how much room the LLM’s context window leaves for the retrieved chunks.

Basic retrieval works well on direct questions over a uniform corpus, but it has three common failure modes:

• Vocabulary gap. User uses one word, and the document uses another. For example, the user types “downtime” and the document uses “service interruption.” Acronyms, internal jargon, and exact identifiers often fall outside the scope of what a general-purpose embedding model has been trained on.

• Multi-hop questions. Answer is spread across many documents that are not semantically similar. For example, the question “what did the CFO say about the product launch?” requires information from both an earnings transcript and a product roadmap. The two documents do not embed near each other in vector space, so similarity search alone is unlikely to retrieve both.

• Ambiguous queries. Query has more than one valid interpretation. For example, “tell me about Apple” could refer to the fruit or the company. Without disambiguation, similarity search returns results for one interpretation without indicating that another was possible.

The vocabulary gap and the exact-identifier problem both come from the same source: similarity search alone cannot match words it does not understand semantically. Hybrid search adds a second retrieval path that can.

The third failure mode, ambiguous queries, is handled by a different technique covered later in this newsletter…

Let’s keep going!

Hybrid Search

Read more

• Share this post & I'll send you some rewards for the referrals.

You hit an error you can’t solve…

So you paste it into Claude Code, and the agent tries a few fixes, but NONE of them work. The bug is affecting checkout, which makes it a money problem, so you drop it in Slack and explain the whole thing again to the senior responsible for that flow.

She asks good questions… You answer them…

Then someone says, “Make a ticket so we don’t lose this,” and you type the same context into Jira a third time. Same problem, three tools, and you wrote it out fresh for everyone.

But none of them remembered what you’d already said in the last…

The bug happened the way most bugs appear these days: you just told a coding agent the feature you had in mind. This is how the job works now,,, and most of us don’t know where agents fit into the traditional software development lifecycle…

So let’s learn about it in this newsletter…

We start with what an agent actually is, because the word gets stretched over everything from autocomplete to a bot that opens its own pull requests1. Then, we’ll cover eight stages every team moves software through, from planning a feature to keeping it alive in production.

For each stage, we answer one question: can you hand this to an agent today, and what happens when you do?

In some stages, agents already carry real weight… In others, they make things worse…

By the end of this newsletter, you’ll know which is which, and the harder problem hiding underneath all of it.

Onward.

I’m happy to partner with CodeRabbit on this newsletter.

One thing I believe, after researching this newsletter, is that the biggest problem with AI agents in software development is NOT code generation. It’s the context loss between tools, tickets, reviews, and Slack threads.

CodeRabbit is building around exactly this problem…

Their new Slack agent maintains shared context across the software development lifecycle, so agents don’t have to start from scratch every time work moves between coding, testing, incidents, and reviews.

They already run AI code reviews across 6M repositories for 15,000+ teams.

And I genuinely think the idea of shared memory across the SDLC is one of the more important directions in agentic engineering right now.

Try CodeRabbit's Agent Today

What an AI agent is

An agent takes a goal, picks its own next step, uses tools2, and keeps going until the job is done or it gets stuck.

That’s the whole idea…

When you hit the checkout bug and pasted it into Claude Code3, you didn’t tell it which files to open or what to run. Claude Code worked out the rest on its own:

• Read files,

• Grep4 for functions,

• Run tests,

• Read errors,

• Try again.

…and you watched.

“Agent” gets used loosely, so it helps to line up the tools by how much they do on their own.

The simplest is AI AUTOCOMPLETE, like the earlier versions of GitHub Copilot5. It finishes the line you’re typing with a Tab.

A step up is a CHATBOT: you ask, it answers, and you copy what’s useful back into your editor. That’s your ChatGPT.

An AGENT is the one who does the work… You give it a bug, and it reads files, runs tests, edits code until the job is done.

To do this, it has to keep track of what it has already tried and what it learned along the way. That running context is what lets it pick a sensible next step instead of starting over each time.

One agent manages its own context fine…

But it becomes difficult when you have to run several agents at once, and that difficulty is largely what makes agents hard to use well.

Software gets built in eight stages, and every team moves through them with or without agents:

• Plan: decide what to build,

• Design: decide how it’s structured,

• Code: write it,

• Test: prove it does what it should,

• Review: check the work before it ships,

• Deploy: ship it to production,

• Operate: keep it running and watch it,

• Maintain: fix, update, and clean up over time.

In theory, you can put an agent to work in any of these stages. Yet it behaves differently in each…

Software lifecycle: where agents fit

The same agent that looks brilliant in a demo can be useless on your actual codebase.

What changes between the two is how much it costs you when the agent is wrong, and this comes down to 4 things:

• How fast you find out. A failing test tells you in seconds. While a bad design decision can go unnoticed for months.

• How easily you can check the answer. You can read a git diff and judge it. “Is this the right thing to build” has no such test.

• How much breaks. A wrong function fails one test. A wrong deployment takes down system for everyone.

• Whether you can undo it. You can revert a commit. But you can’t uncorrupt the data a bad database migration6 already wrote.

A demo is set up to do well on all four…Small task, instant feedback, nothing real at stake… Yet your work usually isn’t.

So, for each stage below, the real question is the same: when the agent gets it wrong, how much does that cost you?

1. Plan

Planning is deciding what your team builds next.

Which feature out of the ten on the list, why that one, and when it ships. It happens in meetings, in docs, in back-and-forth between engineers and the people who run the product…

An agent does useful work here.

Give it a feature spec, and it reads your codebase, maps what the change will touch, flags the parts nobody pinned down, and writes the tickets faster than you would by hand. Hand it the boring write-ups, and it does them well.

But decision-making is something an agent CAN’T do:

Choosing what to build means weighing customer urgency, revenue, and risk against three people who each want something different, then standing behind the choice when it goes wrong. An agent has no stake in the outcome and doesn’t understand your business well enough to make that call.

The people with the most reason to claim otherwise say the same thing.

Boris Cherny, who created Claude Code, was asked why Anthropic keeps hiring engineers when the company says Claude now writes most of its code. His answer:

“Someone has to prompt the Claudes, talk to customers, coordinate with other teams, and decide what to build next. Engineering is changing, and great engineers are more important than ever.”

i.e., company shipping the most capable coding agent on the market still needs actual humans for planning.

2. Design

Plan is which problem you solve…

Design is the shape of the solution: one service or three, which database, where the boundaries sit, and what you can change later without a rewrite.

Ask an agent, and you will get a real answer.

It will compare microservices against a monolith7, lay out the tradeoffs, and recommend one. Yet the answer is textbook, and this is the problem. A textbook answer doesn’t know your team is five people, your traffic triples every December, or the last person who touched the payments service quit and left no documentation. So the right design for your situation lives in facts that were never written down, and an agent can’t read what nobody wrote.

There’s a second reason this stage stays with humans…

A bad design is slow to show itself. The code passes review, and tests go green, so nothing indicates the structure is wrong. But it’s only until you try to build the next three features on top of it, and each one takes twice as long and breaks something that used to work.

By the time you see the pattern, fix is no longer a code change. It’s a complete rewrite…

The root of it is agent works on the task in front of it without a picture of the entire system or any reason to keep the system simple. This blindness shows up in two ways in design.

The first is duplication:

GitClear analyzed code changes across thousands of repositories, found a sharp rise in duplicated code after teams adopted AI coding tools. In 2024 alone, the number of code blocks with five or more duplicated lines increased 8x.

The agent writes a new block that does the same job as one already sitting three files over, because it never checked what the codebase already had. Every copy is one more place you have to remember to fix later, and this is a structural cost.

And second is the opposite problem: over-engineering.

Ask an agent to design something, and it tends to overdesign.

It adds layers, patterns, and configuration for cases that will never come up because more structure looks more thorough.

Andrej Karpathy, who co-founded OpenAI, put it plainly: models like to overcomplicate, bloat their abstractions, and leave dead code behind.

This is the textbook answer showing up again…

The agent designs for the general problem because it doesn’t have the full real-world picture. You will inherit a system harder to read and change than the thing you asked for.

3. Code

This is the stage where agents are doing their best work…

At least the MONEY says so. Cursor went from $100M to $1B in annual revenue in about a year and has more than a million people using it every day. In Stack Overflow’s 2025 survey, 19% of professional developers said they use Cursor, and 10% use Claude Code, two tools that weren’t even on the survey the year before. These tools write real code that ships, and the people paying for them are not running demos.

What you get out of them depends almost entirely on how you set them up…

Rules files

The first thing to set up is a rules file.

Most coding tools now read a file in your repo when they start, a CLAUDE.md8 or an AGENTS.md, and treat whatever is in it as standing orders. You write down the things you would otherwise have to repeat in every chat:

• Commands to run tests and linter9,

• Conventions the codebase follows,

• Traps you learned the hard way and want the agent to avoid.

Without this file, every session starts from zero…

You explain the same setup, correct the same wrong assumption, and watch the agent make the same mistake it made yesterday, just because it has NO memory of yesterday.

Writing correct rule files is already somewhat of a science in coding, and there are many excellent sources that can help you.

(I recommend starting with the official Anthropic guide on CLAUDE.md files.)

Spec-driven development

The second thing is to match the size of the task to how much you plan first…

A small, clear job gets done well from a one-line prompt. “Add input validation to this endpoint” needs nothing fancy. While a large task like “add photo sharing to my app” falls apart because that prompt forces the agent to guess at thousands of decisions you never specified.

So for anything real, you write a short spec first…

A spec documents non-technical product decisions, high-level architecture, and what the changes should do in plain language before any code gets written. The agent turns it into an implementation plan, noting exactly which files and lines of code will change.

Then it executes the code one task at a time, and you review between every step…

This workflow is called spec-driven development10, and it is how people ship real features with agents instead of demos.

PROOF: GitHub repo Superpowers (200k+ stars) works with all major coding agents. It encapsulates the process into a series of automatic slash commands, making it trivial to adopt in your daily workflow.

But spec-driven development isn’t new,,, it is an old engineering tradition.

The only thing changed is how fast you move from a vision in your head to a written spec to a working implementation. All with the help of coding agents…

Current limitations

There are many, but the first one you’ll hit is the context window11…

An agent holds everything it’s working on in a fixed amount of memory, measured in the number of tokens12.

The longer a single task runs, the more memory it fills with its own earlier steps, and at some point, the oldest facts drop out of the window. It forgets a decision it made twenty minutes ago and starts contradicting itself.

This is why the spec-and-plan loop matters beyond just planning…

The spec and plan documents can hold massive context on disk agent can refer to at any time. This frees up its window to focus only on the task at hand. It doesn’t matter whether your agent has a 1M-token window.

Quality drops as the context fills, well before the window is full.

Anthropic documents this for its own models, and a study of 18 models found every one got worse as the input grew. The decline is steady, and a model with a 1M-token window can already be slipping at 50k.

4. Test

Testing is where you find out whether the code does what you said it should, before anyone else depends on it.

This is the stage agents are ‘built’ for…

A test runs in a few seconds and tells the agent one thing: pass or fail. So the agent can write a test, run it, see what broke, and try again, as many times as it needs, without you in the room.

The products doing this are real and busy…

• Momentic, a tool that tests apps by clicking through them the way a user would, ran over 200 million test steps in a single month and caught 390,000+ bugs.

• Diffblue Cover, which writes tests for large-scale Java systems at banks like Goldman Sachs and JPMorgan, writes them 250 times faster than a person and adds enough tests to raise test coverage by 50 to 70%.

This does not mean you can hand testing to an agent and walk away… The failures here are easy to miss, and there are three of them:

The first one affects everybody.

Ask an agent to write tests, and a lot of them pass without checking anything real.

The test runs the function, and function returns an answer. The check is technically true, and still nothing useful got tested. Agents write these empty tests all the time.

This is why, when Claude Code writes a plan for me, I almost always send it back with one note: make the tests mean something, not just pass. The agent then usually throws out its first set and writes much better ones.

The second failure is a direct follow-up to the first.

The agent writes so many tests that keeping them all working becomes a job in itself.

Large corporations like Meta are already running into this. Their testing setup, built up over decades, could not keep up with the number of tests the agents were writing, so they moved to tests that are made fresh for each pull request and deleted as soon as they run.

The third one is the hardest to fix.

Agents are slow and expensive at testing anything you can see on screen…

A logic test is quick because the agent reads pass or fail straight from the code. A test of a button or a page gives no such answer, so for every step, the agent has to:

1. Take a screenshot,

2. Work out where the button sits on screen,

3. Click it,

4. Take another screenshot to check what changed.

A person clicks a button in a fraction of a second... The agent spends time and tokens on all four steps to do the same thing…

Did you know?

CodeRabbit’s new Slack agent keeps shared context across coding, reviews, incidents, and tickets, so your agents stop starting from zero every time work moves between tools.

Use it to carry decisions, debugging history, and architectural context across your entire software development lifecycle:

Try CodeRabbit's Agent Today

5. Review

Before any code ships, someone has to read it and sign off on it.

For years, someone was always a person. You open a pull request, and one or more teammates read your changes line by line. They look for bugs, for code that does not match the rest of the project, and for anything that does not do what the request says it does.

Nothing merges until one of them approves it…

But this process is slow:

Reviews pile up in a queue while people are busy with their own work. One senior engineer often ends up reviewing everyone else’s code, becoming the bottleneck. And a reviewer reading a 600-line change late on a Friday is NOT going to catch much.

This is the part agents handle well, and the WIN is speed and coverage.

An agent reviews every pull request the moment it opens. It never sits in a queue, never has a bad day, and never skips the boring change nobody wanted to read. This speed and coverage matter. CodeRabbit, one of the biggest AI review tools, now runs on more than 6 million repositories for over 15,000 teams.

So every change gets a look. This is the real value,,, and it is worth a lot.

What an agent can’t do is decide which of its own comments matter.

It flags a lot, and only some of it is worth your time. A 2025 study of 278,790 real review comments found developers acted on 16.6% of suggestions from AI reviewers, compared with 56.5% from human reviewers.

i.e., people accept a human reviewer’s note 3x as often.

The danger is what it does to your own attention…

When most of the comments are not worth acting on, you stop reading them closely. You skim, you click approve, and the one comment in twenty that caught a real bug slides past with the rest. The agent reviewed every line, and you still missed the bug, because it buried the signal in noise.

So treat the agent as a first pass that never misses a file, NOT as the reviewer who gets the final say…

6. Deploy

This is the first stage where the answer is a clear NO. Don’t hand it to an agent…

Two things make deploy different from everything before it:

• First, you can undo a bad deploy, but not in the clean way you undo a bad commit. You can roll back, sure. By then, the broken version already reached real users, money may have moved, and the rollback is its own risky operation; you’re running under pressure.

• Second, it breaks everything at once. A bad function fails one test. A bad deploy takes down the entire system for every user at once.

So the deploy button is one of the few places where the SAFE move is still a human hand…

Let the agent prepare the release, run the checks, and stage everything it can. Keep a person in front of the final push to production, where a wrong call costs the most and undoes the least.

7. Operate

Once the code is live and real users are on it, someone has to keep it healthy…

You watch the dashboards for trouble. When something breaks, you find out why and fix it fast. A lot of these land at bad hours, because production doesn’t break on a schedule. You’re on call, and your phone can buzz at 2 a.m.

There is more of this work now than there used to be…

All the AI-written code from the earlier stages ships, and it breaks more often as measured by two separate studies:

• Google’s DORA13 report tracks delivery stability across thousands of teams and found greater AI use correlates with lower stability in both its 2024 and 2025 editions.

• Faros AI analyzed telemetry14 from 22,000 developers and found the number of production incidents per merged change increased by 242% as AI use climbed. More code ships, more of it breaks, and someone is on call for all of it…

Ironically, agents are really good at analyzing incidents caused by bad agent code.

The first part of any incident is gathering. You pull the recent deploys, search the logs, check the dashboards, and walk through the runbook15 step by step. It’s slow, and it’s almost the same process every time.

An agent does it fast…

Datadog’s Bits AI SRE16 investigates an alert the moment it fires and has the findings ready before you’ve opened your laptop.

One of its customers said: investigation is done before the engineer sits down. At 2 a.m., this head start is worth a lot.

What you can’t hand over is the decision on what’s actually wrong…

The agent will tell you the cause, and it says it with the same confidence whether it’s right or not. An alert fires; the agent says it’s the database connection pool. It sounds right, so you spend 30 minutes there. But the actual cause can be completely different. A human might have said “I’m not sure yet” and kept looking. The agent’s speed means you commit to the wrong fix sooner.

So let the agent gather the facts and keep the diagnosis YOURS…

8. Maintain

After a feature ships, the work doesn’t stop…The code still needs “babysitting”.

A library releases a security patch, so you bump to the new version. A messy function needs cleaning up; a small bug turns up; docs stopped matching the code three releases ago. None of it is glamorous, and all of it piles up. Ignore it long enough and the codebase rots.

Of all eight stages, this is where agents earn the clearest yes…

The reason is maintenance work grades itself. A dependency bump either passes the tests or it doesn’t. A refactor is meant to leave the behavior the same, so if the tests still pass after the agent rewrites the code, the refactor worked. The test suite is the judge, and it’s one the agent can run on its own, as many times as it needs, until the change is green.

This is the exact condition agents are built for…

The products doing this run at a scale no other tool mentioned today matches.

Dependabot (GitHub’s tool) watches your dependencies and opens a pull request when one falls behind, and is used across millions of repositories. GitHub says repositories using automated security updates fix critical vulnerabilities significantly faster because the update is waiting for you instead of sitting on a to-do list that nobody gets to.

But there is still a subtle way this can go WRONG:

Each change is small and passes its tests, so each one clears review without much thought. The agent doesn’t see the entire system, so it ships the refactor that works while leaving the code a little worse than a careful person would.

A bit more duplication here, an odd structure there…

No single change is worth stopping for. Over time, they stack up into a codebase, making it harder to read and change than before, and nobody can point to the commit where it went wrong. So this is the same blindness from the ‘Design’ stage, showing up slowly instead of all at once.

Let the agent handle the maintenance, and keep a human to read what it actually changed…

Across the lifecycle

One pattern runs through all eight stages:

Agents do well wherever a machine can tell them they’re wrong. A failing test, a red build, a type error: agent gets an answer in seconds and tries again on its own until the work is green. This includes code, tests, dependency bumps, first pass at an incident.

But they struggle wherever the only judge is a person…

Whether this is right feature, right architecture, or right moment to ship to production. No test comes back red on those. And the answer depends on what your business needs and how much it costs you when you’re wrong, and none of it is written down for the agent to check.

These are also the stages where a mistake is slow to find and hard to undo, so the agent runs furthest before anyone catches it.

Every verdict here looked at one agent doing one job, though. Real teams run a dozen at once, and it changes the problem…

Why more agents do NOT make your team faster

So far, we have looked at one agent doing one job. Real teams run a dozen at once…

A coding agent, a test agent, a review agent, and an on-call agent, all working at the same time. The trouble is none of them shares memory with the others. So the coding agent doesn’t know what the test agent already tried. And the review agent doesn’t know what the on-call agent learned last night.

Each one starts blank about every other one.

That gap costs you more than you’d expect…

Nicholas Carlini at Anthropic ran 16 Claude agents in parallel to build a C compiler. Over 2,000 sessions and about $20,000 of compute, and the project worked. But here is what he found running them together:

“Every agent would hit the same bug, fix that bug, and then overwrite each other’s changes. Having 16 agents running didn’t help because each was stuck solving the same task.”

So the agents duplicate and overwrite each other, and a person has to review it all and resolve the collisions before any of it merges.

The context that would stop the repetitive work already exists… It is just the agent can’t usually read it…

It’s in a Slack thread, in the reasoning on a closed pull request, in an old ticket, or in what the on-call agent worked out last night. The next agent can read your code, but it can’t read the conversation where you decided why the code looks a specific way.

This decision history is the part NO agent has…

The fix is to give the agents one shared memory which survives between runs, running where the team already talks, so a person can interrupt and redirect.

CodeRabbit launched a Slack agent that works this way, and the way they describe the problem is the subtle patterns we have been hinting at throughout the newsletter: “Each phase runs on a different tool and uses a different agent. None of them talks to each other. What one engineer figures out in coding doesn’t show up in testing.”

Their agent runs in Slack, so it reads the same threads your team does and carries what it learns from one conversation to the next. Ask it Friday about something it saw Monday, and it still knows…

Shared memory is the real problem under all eight stages, and nobody has fully solved it yet…

What to expect when an agent is on your team

Say you decide to put agents to work anyway, before that problem is solved.

Here is what the job actually feels like once you do it:

The first thing to know is the speed you feel may NOT be the speed you get…

METR, an independent nonprofit running studies on what AI can and can’t do, tested this in 2025. (They aren’t selling an AI tool, so they had no reason to tilt the result either way.)

They took 16 experienced developers and gave them 246 real tasks on their own projects, the repositories they had maintained for years. The developers did half the tasks with AI tools and half without, and a timer recorded both.

They finished the AI tasks 19% slower. The surprising part is same developers thought the AI had made them 20% faster. They were slower and felt quicker at the same time.

This gap is what matters here…

These were experts on code they already knew well, where they were fast to begin with, so it won’t match every job. But your own sense of whether an agent is helping is not reliable, so check it against something real.

With more agentic work, your week changes a lot…

An agent opens more pull requests than you would, and each one is bigger than the last. So you spend less of your day writing code and more of it reading what the agent wrote and deciding if it’s right. That reading has a different cognitive cost. Do it all day, and you end up tired in a way that writing the code yourself didn’t make you, even when the agent saved time.

The other thing to set up before you “trust” an agent is a limit on what it can “break”…

An agent running commands can do real damage when it gets something wrong:

In July 2025, Replit’s coding agent deleted a production database after it was told, in plain words, to change nothing. It wiped records for 1,200+ business leaders and 1,000+ companies. The agent’s own summary afterward read: “I destroyed months of work in seconds.”

Three controls keep that from being you…

• Run the agent in a sandbox so it can’t reach production in the first place.

• Give it scoped permissions, so it only gets access to the one thing the task needs and nothing else.

• And keep an audit log of every command it ran, so when something breaks, you can see exactly what it did.

So pick your first agent where a wrong answer is cheap to undo…

Coding, review, and maintenance are good places to start, because a bad suggestion gets caught before it ships, and a bad refactor fails the tests. Design and deploy are the worst places to start, for every reason this newsletter has already covered.

Begin where the agent can be wrong without costing you much, learn how it behaves, and only then move it somewhere that matters more.

Your engineers talk on Slack. They code in the terminal. Somewhere between those two things, context dies…

• A bug was debated in #incidents at 2 AM.

• An architectural call was made in a DM.

Every handoff leaks context, and every leak costs you. That’s the context tax and your team pays it every day.

CodeRabbit Agent for Slack is built for agentic SDLC workflows. One agent for your entire Software Development Lifecycle, living in the channel where the work already happens. It’s built on four things:

• Context: your org’s operating picture, pulled from across code, tickets, docs, monitoring, and cloud.

• Knowledge Base: a living memory of your team. Every run leaves a trace, so yesterday’s decisions don’t become tomorrow’s debates.

• Multi-Player: works in shared threads alongside your team. Steerable, resumable, and aligned as work evolves.

• Governance: scoped access, cost attribution. Every run explainable and attributed.

Your team keeps shipping. Agent keeps the context.

From the team that pioneered AI code reviews. 2M code reviews every week. 6M repos. 15K customers. And now, one agent for your entire SDLC, right in Slack:

Try CodeRabbit's Agent Today

(Thanks again to CodeRabbit for partnering on this post.)

Louis and I launched the GENERATIVE AI MASTERCLASS (newsletter series exclusive to PAID subscribers).

When you upgrade, you’ll get:

• Simple breakdown of real-world architectures

• Frameworks you can plug into your work or business

• Proven systems behind ChatGPT, Perplexity, and Copilot

👉 CLICK HERE TO JOIN THE GENERATIVE AI MASTERCLASS

(Golden members will get the next Generative AI newsletter in the first week of June.)

If you find this newsletter valuable, share it with a friend, and subscribe if you haven’t already. There are group discounts, gift options, and referral rewards available.

Subscribe now

Want to reach 210K+ tech professionals at scale? 📰

If your company wants to reach 210K+ tech professionals, advertise with me.

Thank you for supporting this newsletter.

You are now 210,001+ readers strong, very close to 210k. Let’s try to get 211k readers by 29 May. Consider sharing this post with your friends and get rewards.

Y’all are the best.

Share

References

• Boris Cherny on why Anthropic keeps hiring engineers: x.com/bcherny (mirror)

• Duplicated code rose 8x after AI adoption: GitClear, AI Copilot Code Quality 2025

• Andrej Karpathy on models bloating abstractions and leaving dead code: x.com/karpathy

• Cursor revenue and daily users: Cursor Series D announcement

• Cursor and Claude Code developer adoption: Stack Overflow 2025 Developer Survey

• Context quality drops as input grows, across 18 models: Chroma, Context Rot

• Anthropic on CLAUDE.md memory files: Claude Code memory docs

• Spec-driven development with Superpowers: github.com/obra/superpowers

• Anthropic on context windows: Claude context windows docs

• Momentic, 200 million test steps and 390,000 bugs in a month: Momentic Series A

• Diffblue Cover, 250x faster test writing: Diffblue Cover

• Diffblue Cover, 50 to 70% coverage lift: Uplift Java test coverage

• Diffblue Cover at Goldman Sachs: Goldman Sachs case study

• Meta’s just-in-time tests, generated per pull request: The death of traditional testing

• GitHub Copilot Code Review, 60 million reviews and 1 in 5 of all reviews: 60 million Copilot code reviews and counting

• Developers act on 16.6% of AI review comments vs 56.5% from humans (278,790 comments): Human-AI Synergy in Agentic Code Review

• Amazon Kiro and the 13-hour AWS outage: Amazon’s response (originally reported by the Financial Times)

• DORA, AI adoption and delivery stability: DORA 2024 report, DORA 2025 report

• Faros AI, incidents per change up 242% across 22,000 developers: The AI Acceleration Whiplash

• Datadog Bits AI SRE, generally available December 2025: Datadog launches Bits AI SRE

• Dependabot, 2.7 million repositories and 30% faster fixes: GitHub Octoverse 2025

• Nicholas Carlini, 16 agents building a C compiler: Building a C compiler with Claude

• CodeRabbit’s Slack agent launch and framing: CodeRabbit launches Slack agent

• Mem0, open-source memory layer for agents: github.com/mem0ai/mem0

• METR, developers 19% slower with AI while feeling faster: METR study (paper)

• Replit’s agent deleting a production database: Fortune

• Share this post & I'll send you some rewards for the referrals.

I have a huge huge huge announcement for everyone…

INTRODUCING: DevOps Mastermind

This newsletter series will help you become a fast-growing and high-paying engineer.

If you’ve ever thought:

“I use Docker & Kubernetes, but I still don’t fully understand how they work.”

“I want to master DevOps fundamentals instead of just memorizing commands.”

“I should learn how modern infrastructure & real-world systems actually work.”

Then this is for you.

Here’s what you’ll get inside DevOps Mastermind:

• Architecture breakdown of core DevOps systems.

• Deep dives into Docker, Kubernetes, Terraform, and so on.

• Practical insights into how real-world systems achieve scalability, automation, and reliability.

Onward.

Your laptop is already running an operating system.

Now imagine starting another one on the same machine. A completely separate system with its own files, memory, and processes. You can run Linux inside Windows, or Windows inside a Mac. It boots independently, runs programs, and behaves as if it has full control over the hardware.

But it doesn’t.

Underneath, there is still only one physical machine. One CPU, one pool of memory, one disk. Yet multiple such systems can run at the same time, each isolated from the others and each functioning as if it were the only system present.

This idea changed computing…

Instead of buying a new server each time, you could split one powerful machine into many smaller ones. Each workload gets its own space. Nothing interferes with anything else.

That shift is what made cloud computing possible…

When you run an app on Amazon Web Services, Microsoft Azure, or Google Cloud Platform, it runs inside a virtual machine. The entire cloud infrastructure is built on this foundation.

But how does it actually work? How can one machine present itself as many? How are resources shared without breaking isolation?

Build strong data foundations for agentic AI at scale (Partner)

AI experiments are easy.

But building trusted AI systems that scale across an enterprise is hard. That’s exactly what leaders from Mercedes-Benz, Yahoo, Regeneron, and AWS discuss in this on-demand virtual panel on agentic AI.

Here’s what you’ll learn inside:

• Practical strategies from enterprise leaders: See how teams move from isolated AI experiments to production-scale systems.

• Real-world guidance on governance and architecture: Learn how industry experts approach analytics, unified governance, and continuous learning.

• Frameworks for scaling intelligent systems: Discover how to build AI systems that drive business value.

• Pragmatic infrastructure decisions: Learn how to select the right databases and data foundations for AI-based applications.

Watch the AWS panel and see how enterprise teams are building trusted AI systems with speed, governance, and control.

Watch the AWS Virtual Panel

(Thanks to AWS for partnering on this post.)

I want to introduce Ayaan as a guest author.

He’s a self-taught DevOps engineer focused on Kubernetes & cloud infrastructure. He builds in public and writes about cloud-native systems, sharing daily technical tips and projects.

Check out his work and socials:

• Twitter

• Github

Here’s what’s inside this newsletter:

• The difference between emulation and virtualization, and why one simulates hardware while the other shares it.

• What a virtual machine actually is, how it presents itself as a full computer, and why the guest operating system believes it owns the hardware.

• How the hypervisor works, the different types it comes in, and how it controls access to CPU, memory, storage, and networking.

• What happens under the hood when a virtual machine boots, from resource allocation and virtual hardware setup to kernel startup and runtime execution.

• How features like live migration, isolation, and resource controls make virtual machines practical for modern infrastructure.

• Where virtual machines fit today, why cloud platforms depend on them, and how they differ from containers.

Emulation vs Virtualization

Before we get into how virtual machines work, there is one distinction worth getting clear early.

Emulation is when software pretends to be completely different hardware. Retro gaming emulators work this way. Your laptop takes each instruction from the old console and turns it into something your modern chip understands.

It works, but it is slow…

Virtualization is different. Instead of pretending to be different hardware, it creates a version of the same hardware your machine already has. Most instructions run directly on the real processor with no translation needed.

The result is near-native speed.

Think of it this way:

• Emulation is like hiring a translator for every sentence in a conversation.

• Virtualization is like realizing that both people already speak the same language.

This distinction matters because speed is what made virtualization practical at scale. You cannot run thousands of emulated systems on a single server. With virtualization, you can.

And that is exactly what cloud providers do…

What a Virtual Machine Actually Is

A virtual machine is a complete computer abstraction built on top of a physical machine.

It’s presented with a CPU, memory, storage, and a network interface, just like a physical system. The operating system inside it interacts with these components as if they were real hardware. It has no awareness that it is sharing the underlying machine with anything else.

From its perspective, it owns the system.

This behavior is made possible by a layered design.

At the bottom is the physical hardware, which provides the actual compute, memory, and storage resources. Above it sits the hypervisor, which controls how those resources are allocated and accessed. On top of that, each virtual machine runs its own operating system in isolation.

The hypervisor is the layer that makes the illusion possible.

Because the operating system never interacts with the hardware directly.

Instead, every request for CPU time, memory access, disk I/O, or network communication is handled through the hypervisor. It decides how resources are shared and ensures that each virtual machine remains isolated from the others.

This abstraction enables three key capabilities:

• Isolation: Each virtual machine runs independently, so a failure in one doesn’t affect others.

• Portability: Virtual machines can move between physical systems with minimal changes.

• Efficient resource use: Multiple workloads share the same hardware instead of leaving it idle.

These properties are what made modern infrastructure possible.

Next, let’s look at the hypervisor: what it is, the different types, and how it powers virtual machines…

Hypervisor: Brain Behind the Illusion

The hypervisor is the software layer that enables virtualization.

It sits between the physical hardware and the virtual machines running above it. Every request a virtual machine makes, whether for CPU time, memory, or storage, goes through the hypervisor. It’s responsible for mapping those requests to the underlying hardware resources.

This layer controls all access to the hardware.

Type 1 vs Type 2

Not all hypervisors follow the same structure. The main difference comes from where they run.

• Type 1 (bare-metal): It runs on the physical hardware, with no host operating system in between. Examples include VMware ESXi, Microsoft Hyper-V, and Kernel-based Virtual Machine (KVM). Cloud environments and data centers use these because they provide direct hardware access and minimal overhead.

• Type 2 (hosted): Runs on top of an existing operating system, like a regular application. Tools such as VirtualBox and VMware Workstation fall into this category. They are easier to use but introduce additional overhead.

The difference lies in how the hypervisor accesses hardware and the number of extra layers it adds…

Full Virtualization vs Paravirtualization

There are also different ways a hypervisor presents hardware to a virtual machine.

In full virtualization, guest operating system runs without modification and behaves as if it were on real hardware. The hypervisor handles all the work required to maintain that abstraction.

In paravirtualization, the guest operating system is aware that it is running in a virtual environment. Instead of relying only on emulated hardware, it asks the hypervisor to handle some tasks, which reduces overhead.

The trade-off is between compatibility and performance.

Full virtualization runs any unmodified operating system. Paravirtualization is faster because the guest works with the hypervisor.

Hardware-Assisted Virtualization

Early hypervisors relied on software, making virtualization complex and inefficient. This changed when processor manufacturers introduced hardware support such as Intel VT-x (Intel Virtualization Technology for x86) and AMD-V (AMD Virtualization).

With these features, CPU itself can differentiate between virtual machines and the hypervisor. Sensitive operations pass control to the hypervisor when needed. It manages access and keeps everything secure.

This made virtualization practical at scale.

Nested Virtualization

It is also possible to run a virtual machine inside another virtual machine, a setup known as nested virtualization. You use this for testing, development, and running tools like Kubernetes locally in virtual machines (VMs).

But each additional layer introduces overhead and increases complexity. As you add more virtualization layers, performance drops. This makes the approach unsuitable for most production workloads.

It is useful in controlled scenarios, but not a general-purpose design.

Now you know what the hypervisor does. The next question is how it shares the CPU across multiple virtual machines. The answer is CPU virtualization.

Let’s look at how it works…

CPU Virtualization

The CPU is the brain of any computer.

Every instruction, process, and calculation goes through the CPU. So when you create a virtual machine, the question becomes: how do different operating systems share one CPU without getting in each other’s way?

That is what CPU virtualization solves…

The Privilege Model

Operating systems are designed to be in charge.

When your OS needs to access hardware, manage memory, or control processes, it does so with full authority. The processor has a concept called privilege levels to enforce this.

Think of it like levels of trust, where the most trusted software gets the most access.

In a normal system, the operating system sits at the highest privilege level.

But in a virtual environment, the hypervisor needs to sit above the operating system. It needs to intercept certain instructions and control what the guest OS can and cannot do. So the guest OS gets moved down to a lower privilege level, even though it still believes it is fully in charge.

The guest OS never knows the hypervisor demoted it.

This is the fundamental trick behind CPU virtualization. The guest OS continues to behave as if it owns the processor. But when it tries to do something that needs high-level access, the processor steps in and hands control to the hypervisor.

Everything continues smoothly, just one level deeper…

vCPU and Scheduling

When you create a virtual machine, you assign it CPUs, but these are not physical cores. They’re virtual CPUs (vCPUs) that represent processing capacity managed by the hypervisor.

The hypervisor schedules vCPUs onto physical cores in the same way an operating system schedules processes.

At any given moment, a physical core can execute only one thread of work. The hypervisor switches between virtual machines, giving each one a slice of CPU time. This switching happens fast enough that each virtual machine appears to have continuous access to the processor.

Modern CPUs may expose multiple logical threads per core through technologies like Intel Hyper-Threading or AMD SMT (Simultaneous Multithreading), but CPU time is still shared and scheduled across workloads.

In reality, all virtual machines share it.

This is called time-slicing. It’s the same idea your operating system uses to run multiple applications at once.

Hypervisors can also assign more vCPUs than there are physical cores, a practice known as CPU overcommit. For example, a machine with 16 cores might run virtual machines totaling 32 or more vCPUs.

This works as long as not all virtual machines are busy at the same time.

When demand rises across multiple VMs, they compete for CPU time. This causes contention and reduces performance.

Overcommit is a useful tool, but it depends on workload behavior.

VM-exit and VM-entry

Let’s go one level deeper into what actually happens when a guest OS tries to do something it is not allowed to do on its own.

When a virtual machine tries to run a privileged instruction, the processor stops it. Control is immediately transferred to the hypervisor. This event is called a VM-exit.

The hypervisor then figures out what the guest was trying to do and handles it safely.

Once the hypervisor has dealt with the request, it hands control back to the virtual machine exactly where it left off. This is called a VM-entry. From the guest OS’s perspective, nothing unusual happened. The CPU executed the instruction and continued running.

The entire handoff takes microseconds.

This mechanism is what makes isolation possible. The guest OS cannot access hardware it shouldn’t. Every sensitive operation triggers a VM-exit, which the hypervisor controls. But it also introduces overhead. Every VM-exit and VM-entry takes time, and if they happen too frequently, performance suffers.

Keeping VM exits to a minimum is one of the core challenges of CPU virtualization.

NUMA Awareness and CPU Topology

This matters more in large systems and performance-sensitive workloads, but it’s still worth understanding at a high level…

Modern servers often have more than one physical processor. Each processor has its own local memory. Accessing memory that belongs to a different processor is slower than accessing your own. This architecture is called Non-Uniform Memory Access (NUMA).

The name describes the problem exactly. Not all memory accesses are equal.

In a virtual environment, this creates a subtle issue. A virtual machine might run on cores from one processor but end up accessing memory that belongs to another. The guest OS does not know this is happening. But the performance hit is real.

The solution is NUMA-aware scheduling.

The hypervisor tries to keep a virtual machine’s vCPUs and its memory on the same NUMA node. When it succeeds, the VM runs faster. When workloads grow and resources get spread across nodes, the performance gap becomes noticeable.

For most small deployments, this never comes up. For high-performance databases or real-time systems running on large servers, it matters a lot.

Putting it together, CPU virtualization is not a single mechanism but a combination of coordinated techniques.

The privilege model restricts direct hardware access. vCPU scheduling creates the illusion of dedicated processing power. VM-exits and VM-entries enforce safe boundaries for sensitive operations. NUMA awareness ensures that physical hardware layout does not introduce hidden performance penalties.

Each part addresses a specific challenge.

Together, they allow multiple virtual machines to share a single physical CPU while operating independently and efficiently.

Next, let’s look at how memory is virtualized and how virtual machines manage their view of system memory…

Memory Virtualization

Read more

• Share this post & I’ll send you some rewards for the referrals.

The hardest part of building an AI agent1 has nothing to do with the model…

Calling a language model is the EASY part…

Keeping the agent coherent across a workflow that spans hours or days is where everything actually gets difficult: remembering what was said, tracking what’s already been done, and staying steady when the user changes their mind.

Even with stronger models each year, agents break on those exact failures…

They forget earlier turns, repeat questions they already asked, and behave inconsistently across long tasks. The model is rarely the cause. The cause is a missing or badly designed state and memory, plus the consistency rules that keep the two in line.

This newsletter walks through all three…

100+ Claude Code hacks to ship code 10X faster (Partner)

Top engineers at Anthropic and OpenAI say AI now writes 100% of their code.

If you’re not using AI, you’re spending 40 hours doing what they do in 4.

These 100+ Claude Code hacks fix that and help you ship 10x faster.

Sign up for The Code and get:

• 100+ Claude Code hacks used by top engineers — free

• The Code newsletter — learn the latest AI tools, tips, and skills to code faster with AI in 5 minutes a day

Claim your free playbook

(Thanks to the Code for partnering on this post.)

I want to introduce Sivasankar as a guest author.

He is a Technical Director and GenAI practitioner with over 20+ years of experience in architecting and building Big Data, Cloud, and GenAI solutions.

His work focuses on helping organizations design practical and scalable intelligent systems. He enjoys exploring new ideas through hands-on DIY projects and practical experimentation, often turning real-world use cases into working prototypes.

You can connect with Sivasankar (Shiva) on LinkedIn.

Here’s what you’ll find inside this newsletter:

• State vs memory. What each one is for, how they differ, and why conflating them is the root cause of most agent bugs.

• Memory lifecycle. How an agent creates, updates, summarizes, and deletes memory so it doesn’t rot over time.

• Consistency under change. How state reacts to new instructions right away, and how long-term memory updates only after a change holds up.

• A reference architecture. The four layers (brain, state, memory, external systems) and what each one owns.

• Scaling and failure modes. Where stateful agents break at scale, and the three ways memory itself goes wrong.

By the end, you’ll know how to design agents that track the right things, recover from failure, and stay truthful with the data they don’t control.

Let’s start with the state...

Tracking what the agent is doing with the state

State is the information an agent keeps about the current task: the plan, active constraints, what’s already been done, and what comes next.

It’s the agent’s picture of NOW…

The easiest way to see why it matters is to compare an agent with it to one without it.

Stateless vs Stateful

A stateless agent keeps nothing between requests…

Each input is a fresh start, which is fine for one-shot jobs like “translate this sentence to French, summarize this paragraph, or explain what an API is”.

A stateful agent tracks the current task, session, or workflow…

At each decision point, it checks the state to answer three questions: what step am I on, what has already been done, and what should I do next? Those answers drive the next action, check constraints, and stop the agent from repeating itself.

State matters the moment a workflow has more than one step…

The user books a flight (step one done, constraints locked), asks for a hotel (step two, new constraints layered on), then changes the return date (earlier step invalidated, replan downstream).

Without a state, none of those three questions has an answer.

Where state lives

State is maintained on the agent side (backend server that runs the reasoning loop), and NOT on the client side (chat UI, mobile app, or API caller the user interacts with).

Clients disconnect, refresh, and retry. And agent still needs a steady view of progress.

So where that view lives depends on how long the workflow runs…

Short-lived workflows that end in a single session can remain in an in-memory store. Anything that has to survive restarts needs external storage: a database, a KV store (Redis-style lookups by id), or a serialized state object managed by an orchestration framework2.

LangGraph3, for example, comes with SqliteSaver and PostgresSaver as ready-made checkpoint backends.

At each step’s start, agent loads the latest state snapshot and rebuilds its context. That load makes a chain of steps feel like one chain rather than a set of separate calls.

Checkpointing and recovery

Loading state on each step only works if something wrote it in the first place…

That’s what checkpointing4 does: it saves state at the milestones you pick as meaningful, not every tiny step. If the agent crashes, times out, or gets redeployed mid-workflow, it reloads the last checkpoint and resumes from there.

Picture the travel agent picking a return flight when the API call fails…

Without checkpointing, the agent would have to re-ask the user for dates, budget, and outbound flight details. With checkpointing, the saved state already contains all that, so on retry, the agent picks up at the failed step and continues.

The same setup covers internal errors in the agent’s own code and redeploys that restart the infrastructure underneath it.

This pattern runs in production at real scale…

LinkedIn’s internal SQL Bot5 is a hierarchical multi-agent system built on LangGraph, and its long-running query workflows rely on the framework’s checkpoint layer to pause, inspect state, and resume without losing context mid-task.

State versioning

Long-running agents change over time…

New fields appear, steps get reordered, logic shifts. Without versioning, in-progress workflows break the moment the schema changes, and the stored state written yesterday becomes unreadable tomorrow.

Versioning keeps yesterday’s state readable…

It migrates old formats forward and blocks execution paths that no longer make sense under the new schema. That way, workflow changes roll out without killing ongoing tasks.

This is most evident in production systems where workflows span minutes, hours, or even days…

Travel agent: state in action

Storage, checkpointing, and versioning click together once you watch a concrete plan move through them…

The user says, “Plan a complete trip for me.” The agent then walks a sequence:

1. Pick the date and destination.

2. Select an outbound flight.

3. Then select a return flight.

4. Choose accommodation.

5. Confirm the itinerary.

After each step, the state moves forward…

Once the outbound flight is selected, the state says select the return flight next. And checkpoints occur after preferences are confirmed, outbound flight is selected, return flight is selected, and the hotel is booked.

If the hotel step ‘fails‘ because of a network error, the agent restarts, reloads the state, sees that the dates and flights are already locked in, and retries only the hotel step.

NO re-asking and out-of-order suggestions…

That’s what the state gives you.

Next up is memory: information the agent carries across tasks...

Carrying information across tasks with memory

If the state is the agent’s picture of now, memory is what spans across: past decisions already made, preferences built up over time, conversations it can refer back to, and reference data it queries from other systems (flight schedules, hotel listings).

Agents store memory in three flavors, separated by how long the memory persists and whether the agent owns the data or just reads it from elsewhere.

Short-term memory

Short-term memory holds the current context for the active task…

It’s short-lived and cleared when the task ends. Most implementations keep it in in-memory structures, session variables, or within the LLM6 prompt. The lifecycle is simple: create it when a task begins, update it as the agent reasons and uses tools, and clear it once the task wraps up.

Use it for one-task decisions that don’t need to outlive the current workflow.

Long-term memory

Long-term memory stores information across interactions: user preferences, past bookings, and recurring patterns.

The storage options are vector databases7, KV stores, document stores, or relational databases. A write only happens when the information is actually worth keeping, and retrieval kicks in at the start of new tasks or when a relevant signal shows up in the current one.

Every so often, the agent summarizes or merges old entries to prevent the store from growing out of control…

Use this layer when a preference is steady, behavior repeats, or a piece of information should carry into future interactions.

The big win of long-term memory is NOT loading the entire history into the prompt each turn. The agent stores what’s worth keeping and pulls in only what the current decision needs.

Mem08 is one production system built on this idea:

Tested on LOCOMO9, a benchmark for long conversations, the Mem0 layer reduced p95 retrieval latency10 by 91% against a full-context baseline11. Token cost dropped by over 90% on the same test.

On an LLM-as-judge evaluation12, it scored 26% higher than OpenAI’s memory system.

External memory

External memory is a large reference data that the agent queries on demand instead of storing: flight databases, hotel listings, knowledge graphs, and internal systems of record.

These are external sources that hold the authoritative version of a piece of data, like a booking system or a pricing API…

These live in cloud databases, APIs, knowledge graphs, or external RAG13 indexes. Nothing gets loaded into the prompt all at once. When a decision depends on data that must remain official, auditable, and shared across applications, the agent queries the system of record14 directly.

Memory lifecycle

Even a good memory goes bad if you never manage it. Managing it means walking every entry through four stages:

1. Create when new information arrives (user picks a flight).

2. Update when details change (user adds a second city).

3. Summarize to shrink detail (only the times and locations that matter, not every API response).

4. Delete when the information is no longer needed or the retention window ends (clear past trip details once the booking is complete).

Skip this cycle, and long-term memory grows without limit.

Old preferences start contradicting newer ones, and the agent makes decisions based on a pile of half-truths…

Travel agent: memory in action

One trip exercises all three memory types at once:

• Short-term memory holds the current trip (cities, dates, flight search results).

• Long-term memory holds repeat preferences (airline, seat class, meal choice).

• External memory provides live data that the agent doesn’t own (current flight schedules, hotel availability, seat prices).

The hard part for the agent isn’t storing any of this…

It’s deciding when the state should override memory, when memory should update from what just happened, and when neither should move because the real answer lives in an external system.

That’s the next problem...

100+ Claude Code hacks to ship code 10X faster (Partner)

Top engineers at Anthropic say AI now writes 100% of their code.

Are you using AI to write yours?

These 100+ Claude Code hacks show you exactly how. Sign up for The Code and get:

• 100+ Claude Code hacks — free

• The Code newsletter — learn the latest AI tools and skills to code faster in 5 mins a day

Claim your free playbook

Staying consistent when preferences change

Real users change their minds mid-task.

They try a new option, correct themselves, adjust what they want, and sometimes say one thing on Monday and the opposite on Tuesday. An agent that can’t handle this feels wrong: repetitive, stuck on old instructions, or too eager to rewrite its long-term knowledge on every passing comment.

The core rule is simple: react fast with state, learn slowly with memory.

Reacting fast, learning slowly

State updates as soon as a new instruction arrives…

The current task reflects it right away. But memory updates only after the change looks steady and reusable.

The question is when memory writes happen…

LangChain’s team frames two options: the agent can write “in hot path” (save the new fact before replying to the user) or “in the background” (a separate process updates memory during or after the conversation).

Background writes are what make “learn slowly” work, because they give the system time to check whether the change holds up over a few more turns before it gets saved.

If memory learned too fast, it would rot15…

A rare exception becomes the rule, a short-term constraint becomes permanent, and future tasks go wrong with no clear cause.

Three roles keep this clean:

• State tracks what the agent is doing now,

• Memory stores what it has learned over time,

• Consistency rules keep the two in check when they disagree.

Before any of these rules run, the agent has to pull the right memory for the current decision.

Retrieving only what’s relevant

Agents don’t load all memory…

Instead, they query it like a database, using metadata on each entry:

• topic: flights [seats, airlines, class, timing]

• topic: hotels [room type, location, amenities]

• topic: food [dietary restrictions, cuisine]

• topic: budget [price sensitivity, limits]

• duration: short-term | long-term

• confidence: confirmed | inferred

The current state picks which slices to pull.

Booking a flight pulls flight and budget memory, and leaves hotel memory alone. This keeps the working context focused and stops old or unrelated memory from leaking into the decision.

Rolling back on corrections

When a user corrects the agent, treat it as a constraint change, NOT an error.

The agent always plans under a set of constraints:

seat type = aisle
budget <= 800
departure = morning
hotel near the city center

When the user says, “I want a window seat”, they aren’t saying you made a mistake. They’re saying update the constraints. The agent’s job is to apply the new constraint without throwing away the work that’s still valid.

On a correction, the agent traces which steps depended on the old constraint…

It rolls back to the earliest affected step, marks everything downstream as invalid, and leaves the earlier valid steps alone.

But memory doesn’t update on the first correction…

The agent first checks intent: whether the change is a one-time request, task-specific, or a lasting preference (indicated by phrases like “always a window seat”). Only after the change looks steady does long-term memory update.

When the system of record wins

NOT all memory is truth…

Prices, availability, and inventory come from external systems, not from the agent’s stored preferences. External systems define ground truth; memory shapes decisions without overriding real data, and when the two disagree, the system of record always wins.

Anthropic names this pattern directly in its agent design guidance:

Agents must “gain ‘ground truth’ from the environment at each step (such as tool call results or code execution) to assess its progress.” That keeps the agent in sync with reality, and hallucinations drop because it isn’t guessing at live data it could have just fetched.

Travel agent: consistency in action

The agent already knows 2 things:

• Long-term memory: user usually prefers aisle seats.

• Short-term memory: user wants the cheapest flight for this trip.

Right now, the agent is comparing flights and choosing seats…

Then the user says:

“I want a window seat this time.”

Here’s what the agent does:

1. Reads the new request.

2. Pulls only seat-related preferences (memory).

3. Understands “this time” means a temporary change and labels it accordingly.

4. Rolls state back to the flight selection step because seat preferences affect flight choices.

5. Removes earlier seat and pricing decisions that depended on aisle seats.

6. Recalculates the best flight options using the new window-seat preference.

7. Updates short-term memory for this trip ONLY.

8. Keeps long-term memory unchanged because preference may NOT be permanent.

Later, the user says:

“I prefer window seats going forward.”

Now the agent understands this is a lasting preference…

So it:

1. Read this as a lasting change.

2. Saves window seats as the new default preference.

3. Updates long-term memory.

4. Uses this preference for future trips.

Now, imagine the airline system says there are no window seats available…

The agent tells the user that window seats are unavailable and follows the airline’s live data. It does not remove the preference from memory. Instead, real-world data overrides stored preferences.

So far, the travel agent has shown up in pieces. Now it’s time to run it end-to-end...

Build travel agent from three pieces

Travel Agent AI helps users plan and book trips…

It talks to the user, tracks the current task, remembers useful preferences, and calls external tools to check flights, hotels, and prices.

Every request flows through the same loop…

Workflow loop

1. User input. The user asks for a plan or a specific booking.

2. Intent parsing and reasoning. The agent figures out the goal and picks the next step.

3. State update. Records which step is active and the constraints in play (dates, airline preference, budget).

4. Memory access. Pulls short-term, long-term, or external memory based on what this step needs.

5. Tool execution. Calls APIs for flights, hotels, or pricing.

6. Response generation. Gives a suggestion or confirmation from the current state and retrieved memory.

7. State/memory update. Writes the new selections and any session context back to storage.

8. Next step or loop. Continues to the next task in the plan.

Example prompts

A typical conversation might run across four turns:

1. “Help me plan a trip to Paris next month.”

2. “Book the outbound flight first.”

3. “I prefer window seats and morning flights.”

4. “Now add a hotel near the city center.”

Across these turns:

• State tracks the active step.

• Short-term memory stores choices for this trip.

• Long-term memory stores recurring preferences.

• External data provides live prices and availability.

• Tools connect the agent to flight, hotel, and booking systems.

Agent’s tools

The agent reaches out through flight search APIs, hotel availability services, and pricing and booking systems…

Each step feeds the next, and the saved state keeps the plan aligned with live data even when something fails partway through.

The loop shows when things happen…The next question is what each piece is responsible for...

Reference architecture for a stateful agent

Zoom out from the runtime loop, and a stateful agent splits into four layers:

Each one has a single job, and the lines between them are what keep the agent steady.

• Agent brain (reasoning engine). An LLM sits at the center, planning multi-step workflows, reading user intent, and picking the next action from the current state and retrieved memory.

• State layer. This is where the current workflow lives: which steps are done, which are next, and what constraints are active. It also holds short-term overrides and session-specific choices, and lets the agent roll back when a user changes a preference without restarting the entire plan.

• Memory layer. Short-term memory carries this-task context; long-term memory carries lasting user preferences and habits. Together, they feed the brain with context that matches past decisions and patterns.

• External systems (system of record). APIs, databases, and live data sources provide official information such as seat availability, current prices, and cancellation policies. Memory can suggest, but the system of record always wins in a conflict.

The brain decides, the state tracks, memory remembers, and external systems check.

Mix up those responsibilities, and the agent guesses when it should check a live source, or keeps memory that should have been dropped. Keep them separate, and every layer pulls its own weight.

Deciding what fits in the context window

The four layers only work if they fit…

Every LLM runs inside a finite context window16, and state, memory, tool schemas, tool outputs, and user messages all fight for that space.

What takes up the context window

Every step’s context is the sum of what you send the model in that step…

This includes system and developer instructions, current state, retrieved memory snippets (short-term, long-term, and external), tool schemas and instructions, tool outputs from API responses and search results, and the user’s messages themselves.

A useful mental model for the budget (not a formula to measure):

Context size = instructions + state + memory + tools + data + user input

Visibility is what matters here, not precision…

Tool definitions and raw API responses often consume more context than the model’s actual reasoning.

Anthropic ran into this in their own Claude Code setup:

The full tool definitions17 alone used about 134,000 tokens of context. To fix this, Anthropic introduced Tool Search, a lazy-loading system that loads a tool’s full schema only when the agent actually needs it.

This reduced tool-definition overhead by 85%, increased usable context per session from 122,800 tokens to 191,300, and improved MCP tool-use accuracy for Claude Opus 4 from 49% to 74%.

If the agent loads everything all the time, the context window fills up quickly…

When that happens, older messages drop out first. This usually removes the earliest and most important information, like the user’s original goals, constraints, or instructions.

Retrieve only what’s relevant

Use tags, scope, and recency to pull what matches the current step, not everything the agent has ever learned.

Match retrieval to the current state: booking a flight pulls flight and budget memory, nothing else.

Relevance is really a question of how much history you want to pay for right now…

Summarize and remove old checkpoints

In long workflows, the agent’s state keeps growing after every step…

Keeping every raw log and tool response makes the context large, expensive, and noisy. Instead, the agent can periodically create a short summary of completed work using an LLM.

The summary should keep only the important information:

• decisions already made

• constraints that are now fixed

• tasks that still need to be completed

The agent should remove detailed logs that no longer affect future decisions.

And summaries should not stay forever either.

If an old summary is no longer useful for future tasks, the agent should archive it or delete it. Outdated summaries can confuse the agent just as outdated memory does.

Keep memory outside the prompt

The agent should not keep all its memory in the prompt at all times…

Instead, memory should remain in external storage and be retrieved only when needed. Adding too much memory directly into the prompt quickly fills the context window, increases cost and latency, and makes reasoning less focused.

Most stored memory is only occasionally useful and unnecessary for every step…

Before adding memory to the context, the agent should ask:

• Does this change what I should do next?

• Is this information still valid?

• Would removing it break the current plan?

If the answer to all three questions is “no”, the memory should stay out of the context.

Good agents do NOT succeed by remembering everything…Instead, they retrieve only the information that matters right now.

Cost, latency, and reliability trade-offs

Every reliability improvement comes with a cost…

Retrieving more memory increases latency. Extra tool calls increase costs. Additional consistency checks slow the response down.

At scale, fast agents are usually cheaper but less reliable. More careful agents are safer but slower. You cannot optimize for speed, cost, and reliability simultaneously.

In practice, teams focus on accuracy where mistakes are expensive, such as bookings or payments. Also, they cache tool results when possible to avoid repeated API calls.

Plus, batching API requests reduces overhead. Teams also monitor context size and tune memory retrieval using real usage data instead of guesses.

Now, imagine the travel agent after handling hundreds of trips…

Its long-term memory now contains many user preferences, budgets, airlines, and booking patterns. If the agent retrieves too much memory for every request, the context becomes large and inefficient.

A scalable travel agent:

• retrieves only trip-related preferences

• summarizes old planning steps

• trusts live pricing APIs over stored prices

• keeps prompts small and focused as history grows

Without this discipline, the agent gradually becomes slower, more expensive, and less reliable…

Efficiently managing context is only part of scaling an agent system.

The other challenge is handling memory failures correctly…

100+ Claude Code hacks to ship code 10X faster (Partner)

Top engineers at Anthropic say AI now writes 100% of their code.

Are you using AI to write yours?

These 100+ Claude Code hacks show you exactly how. Sign up for The Code and get:

• 100+ Claude Code hacks — free

• The Code newsletter — learn the latest AI tools and skills to code faster in 5 mins a day

Claim your free playbook

How memory goes wrong

Even if an agent manages context well, its memory system can still fail…

Memory makes agents more capable, but it also creates new ways for them to behave incorrectly. In real systems, three common memory failures occur repeatedly.

These failures happen at different stages of the memory pipeline:

• What gets stored

• What stays over time

• What gets retrieved later

1 Stale memory

The agent relies on outdated information…

Example: The agent remembers that the user prefers window seats, but the user recently switched to aisle seats. Without an update, the agent keeps suggesting window seats.

Why it matters: Old preferences can override newer user intent if the system does not update or expire memory correctly.

2 Wrong information captured

The agent stores incorrect information…

Example: During a conversation, the agent mistakenly saves the user’s phone number as their airline loyalty number.

Why it matters: Once incorrect data enters long-term memory, it can affect many future tasks and become difficult to trace back to the original mistake.

3 Retrieval failures

The correct memory exists, but the agent fails to retrieve it when needed…

Example: The user almost always books business class, but the agent fails to retrieve that preference and instead recommends economy.

Real-world example: In July 2025, a Replit coding agent deleted a live production database during an active code freeze, even after the user repeatedly typed “DON’T DO IT.”

The instruction existed in the agent’s context, but the system failed to apply it correctly. The agent later generated thousands of fake user records to hide the issue. Replit’s CEO publicly apologized and later introduced an automatic separation between development and production environments.

Why it matters: Stored memory is useless if retrieval, ranking, or relevance systems fail to surface the right information at the right moment.

Storing memory is easy. But managing it correctly over time is the hard part.

Preventing these three failures is a major part of building reliable AI agents…

Remember the right things, NOT everything

Here’s a checklist for designing stateful, reliable AI agents:

1. Separate state from memory. State is the current workflow. Memory is short- and long-term knowledge.

2. Checkpoint and roll back. Roll back only the steps affected by a correction. Keep the rest.

3. Keep memory outside the prompt. Retrieve only what’s needed. Apply short-term and long-term rules differently.

4. Budget context carefully. Treat it as a working set. Summarize old checkpoints. Drop stale summaries.

5. Let the system of record win. External systems override stored memory when they disagree.

6. Balance cost, latency, and reliability. Spend precision where mistakes are expensive.

7. Monitor in production. Track context usage, retrieval hits, tool calls, and response times.

8. Add safeguards. Freshness rules, clear approval before writing to long-term memory, scope-aware updates.

When the three pieces work together, agents stop reacting and start adapting…

They survive failure, stay honest with real-world data, and scale without their context budget falling apart. The trick isn’t more memory or a bigger model. It’s knowing what to track, what to recall, and when to act.

👋 I’d like to thank Sivasankar for writing this newsletter!

He is a Technical Director and GenAI practitioner with over 20+ years of experience in architecting and building Big Data, Cloud, and GenAI solutions.

Louis and I launched the GENERATIVE AI MASTERCLASS (newsletter series exclusive to PAID subscribers).

When you upgrade, you’ll get:

• Simple breakdown of real-world architectures

• Frameworks you can plug into your work or business

• Proven systems behind ChatGPT, Perplexity, and Copilot

👉 CLICK HERE TO JOIN THE GENERATIVE AI MASTERCLASS

(Golden members will get the next Generative AI newsletter in the first week of June.)

If you find this newsletter valuable, share it with a friend, and subscribe if you haven’t already. There are group discounts, gift options, and referral rewards available.

Subscribe now

Want to reach 210K+ tech professionals at scale? 📰

If your company wants to reach 210K+ tech professionals, advertise with me.

Thank you for supporting this newsletter.

You are now 210,001+ readers strong, very close to 210k. Let’s try to get 211k readers by 29 May. Consider sharing this post with your friends and get rewards.

Y’all are the best.

Share

References

• Anthropic Engineering. Building effective agents. December 2024. https://www.anthropic.com/research/building-effective-agents

• Anthropic Engineering. Effective context engineering for AI agents. https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents

• Anthropic. Tool use with Claude. Claude API documentation. https://platform.claude.com/docs/en/agents-and-tools/tool-use/overview

• LangChain. Memory overview. LangGraph documentation. https://docs.langchain.com/oss/python/langgraph/memory

• LangChain. Persistence. LangGraph documentation. https://docs.langchain.com/oss/python/langgraph/persistence

• Lewis, P., et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Facebook AI Research, 2020. https://arxiv.org/abs/2005.11401

• Packer, C., et al. MemGPT: Towards LLMs as Operating Systems. UC Berkeley, 2023. https://arxiv.org/abs/2310.08560

• Pinecone. What is a Vector Database & How Does it Work? https://www.pinecone.io/learn/vector-database/

• Chhikara, P., et al. Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory. 2025. https://arxiv.org/abs/2504.19413

• Anthropic Engineering. Advanced tool use. https://www.anthropic.com/engineering/advanced-tool-use

• Anthropic Engineering. Building effective agents. December 2024. https://www.anthropic.com/research/building-effective-agents

• LangChain. Memory for agents. LangChain Blog, 2024. https://www.langchain.com/blog/memory-for-agents

• LangChain. Is LangGraph used in production? LangChain Blog. https://www.langchain.com/blog/is-langgraph-used-in-production

• AI Incident Database. Incident 1152: Replit AI agent deleted production database. https://incidentdatabase.ai/cite/1152/

• Share this post & I'll send you some rewards for the referrals.

You’ve probably walked into a machine learning (ML) system design interview, been asked to “design a recommendation system,” and suddenly everyone’s throwing around terms like embeddings, feature stores, and model drift, and you’re nodding along while silently drowning.

Most prep resources offer algorithm deep-dives or LeetCode-style drills.

But first, what you actually need is a solid understanding of how ML systems work: vocabulary, building blocks, and how they fit together.

This newsletter is a plain-English field guide to ML system design.

You don’t need to know TensorFlow, write Python, or understand gradient descent math. Just what these terms mean in practice, so you finally understand how these systems actually work.

Along the way, we’ll trace how real systems (like Netflix recommendations, Uber’s ETA predictions, and Spotify’s Discover Weekly) actually work under the hood. Don’t worry about memorizing every detail; this guide exists so your next ML interview feels less like an interrogation and more like a conversation.

Let’s start with the question every ML interview is really asking.

Treat coding agents like services, not terminals (Partner)

You use Claude Code, Codex, Cursor, or Gemini in your terminal every day. What if you could call them in code?

AgentField just open-sourced harness orchestration.

• Call those same coding agents programmatically from Python, TypeScript, or Go.

• Compose them into real automations: ship a PR, audit a cloud, run a code review - end-to-end pipelines, not toy demos.

For the architecture rationale, their latest post - harness-as-membrane - is worth a read.

Fork and build

(Thanks to AgentField for partnering on this post.)

I want to introduce as a guest author.

He’s an ML engineer with 8+ years of production AI experience, read by 1M+ AI/ML engineers on LinkedIn and Substack. He runs a tech content agency serving NVIDIA, Google, and Y Combinator, and writes The AI Engineer, the newsletter that makes engineers dangerously good at AI engineering.

What is an ML System?

Traditional software runs on rules you write by hand.

If the temperature is above 90°F, turn on the AC. If the user clicks “buy,” charge the card. Every decision is hardcoded in code.

ML systems work differently.

Instead of writing rules, you feed the system examples (thousands or millions of them), and it figures out the rules on its own. Show it a million past rides, and it learns to predict how long your Uber will take. Show it billions of watch sessions, and it learns which movie to recommend next.

This sounds magical, but it’s just data engineering. Data flows in, patterns get extracted, predictions flow out, and the system keeps learning from its own results.

Every concept in this newsletter maps to a stage in this pipeline.

Master the pipeline, master the interview…

Raw Material: How Data Becomes Intelligence

You can have the fanciest model in the world. If your data is garbage, your predictions will be garbage.

1. Features

A feature is a single, measurable piece of information model uses to make predictions.

Think of features as the columns in a spreadsheet that describe each example.

If you’re predicting whether someone will click on an ad, your features might include the user’s age, the time of day, the device they’re using, and how many ads they’ve already seen this session. Each of these gives the model a different angle on the same question.

In Spotify’s Discover Weekly, features might include your most-played genres, the time of day you usually listen, and how often you skip songs. The model doesn’t “hear” the music. Instead, it reads the features.

The quality of your features often matters more than the sophistication of your model. Which raises the question: where do good features come from?

2. Feature Engineering

Feature engineering is the process of transforming raw data into features that help a model learn better.

Raw data is messy and unstructured. Feature engineering is about cleaning it up, reshaping it, and extracting the signal.

Think of it like preparing ingredients before cooking. You don’t throw a whole chicken into the pan. You cut it, season it, and lay it out first.

For example, Uber’s raw data might say “ride requested at 2024-12-25 08:47:32 UTC.”

Feature engineering transforms that into: day of week (Wednesday), time of day (morning rush), holiday (yes), weather (rainy). Each transformation gives the model a more useful signal than the raw timestamp alone.

Sometimes the most powerful features aren’t in the raw data at all. A feature like “average number of rides this user takes per week” requires aggregation across many records. That kind of creativity is what separates good feature engineering from great ones.

But features alone aren’t enough. The model also needs to know what the “right answer” looks like.

3. Labels & Ground Truth

A label is the answer you’re training the model to predict. Ground truth is what actually happened in the real world.

If you’re building a spam filter, the label for each email is “spam” or “not spam.” If you’re predicting home prices, the label is the actual sale price. The model learns by comparing its guesses against these known answers.

Ground truth sounds simple, but it’s often surprisingly hard to get right.

When Netflix asks “did the user enjoy this movie?”, the ground truth is ambiguous. Did they watch to the end? Did they rate it? Did they watch something similar next? Different definitions of ground truth lead to different model behaviors.

💡 How you define your label shapes what the model actually learns. Choose carefully. Pick the wrong definition of success, and the model will optimize for the wrong thing.

With features and labels defined, the next step is splitting the data so the model can actually learn and prove it learned the right things.

4. Training, Validation & Test Sets

You don’t test a student with the same questions they practiced on.

The same principle applies to ML. The data gets split into three groups, each with a specific job.

1. Training set is what the model actually learns from. It sees these examples during training.

2. Validation set is your mid-training checkpoint. It tells you which model version performs best.

3. Test set is the final exam. If the model performs well here, it’s ready for the real world.

Most teams split their data 70/15/15 or 80/10/10.

The exact ratio depends on how much data you have, but one rule is non-negotiable: test set must never influence model development. Otherwise, you’ve let the student peek at the final exam.

At Netflix, this might mean training on viewing data from January through October, validating in November, and testing in December. This time-based split is especially important for ML systems because real-world data changes over time, a pattern we’ll revisit in Section 5.

Now that we have our data organized, there’s one more data problem that trips up many real-world systems.

5. Class Imbalance & Data Leakage

Class imbalance happens when one outcome is far more common than the others.

In fraud detection, maybe 1 in 10,000 transactions is fraudulent. In medical diagnosis, most patients don’t have the rare disease. The model can achieve 99.99% accuracy by simply predicting “not fraud” every time and still be completely useless.

Fixing class imbalance usually involves techniques like oversampling the rare class, undersampling the common class, or adjusting the model’s loss function to penalize mistakes on rare examples more heavily.

Data leakage occurs when the answer is hidden within the training data.

You’re building a model to predict late deliveries, but your dataset includes “customer complaint filed.” That complaint only exists because the delivery was already late. Of course, the model looks accurate. Remove that feature, and it’s useless.

💡 If your model scores above 95% on the first try, check for leakage. If it scores above 99%, check for class imbalance too.

Clean data is only useful if it actually reaches your model. That’s where pipelines come in.

6. Data Pipelines & ETL

Data pipelines are automated workflows that move data from its raw sources (databases, event logs, third-party APIs) through a series of transformations and into a format the model can use.

ETL stands for Extract, Transform, Load. Extract the raw data from its source.

Transform: clean, join, aggregate, and engineer features from it. Load the results into a feature store or a training dataset.

This is the plumbing part of ML.

And like real plumbing, nobody thinks about it until something breaks. Not the model, not the algorithm. A dropped row, a null value, a late pipeline run. No errors thrown, just silently worse predictions.

Uber runs thousands of data pipelines daily. Everyone has monitoring, alerting, and quality checks. Bring this up in an interview, and you’ll immediately stand out as the guy who actually thinks about how ML works in practice.

Pipelines produce data, but how do you keep track of which data a given model was trained on?

7. Data Versioning

Data versioning means tracking exactly which version of a dataset was used to train each model.

Just like software engineers use Git to track code changes, ML teams need to track data changes.

Why does this matter? Imagine your model’s performance suddenly drops. Without data versioning, you have no way to answer: “What changed? Was it the model or the data?” With versioning, you can compare the current training data with previous versions and pinpoint exactly what changed.

Data versioning also enables reproducibility, a requirement in regulated industries like finance and healthcare.

If an auditor asks, “Why did the model make this decision six months ago?”, you need to be able to recreate the exact model and training data from that point in time.

Tools like DVC (Data Version Control) and MLflow help teams manage this.

In an interview, mentioning data versioning shows you understand that ML systems aren’t just about models. They’re about the entire lifecycle of data and code working together.

With clean, well-structured, and tracked data in hand, it’s time to teach the model how to find patterns in it.

Pattern Machines: How Models Actually Learn

A model doesn’t “know” anything. It finds patterns in data and uses them to make predictions.

8. Model Training & Loss Functions

Model training is the process of showing the model examples and letting it adjust its internal settings until its predictions get closer to the ground truth.

Think of it as a student taking practice tests over and over, getting a little better each time.

The mechanism that drives improvement is the loss function. A loss function measures how wrong the model’s predictions are. After each batch of examples, the model calculates the loss (how far off it was) and adjusts its internal settings to reduce that number.

Over millions of examples, those small adjustments add up.

Different problems need different loss functions.

Predicting a number (like Uber’s ETA) might use mean squared error, penalizing big misses more than small ones. Classifying something (like spam vs. not spam) typically uses cross-entropy loss, which measures how confident the model was in the wrong answer.

The choice of loss function shapes what the model optimizes for. A model trained with mean squared error will try to be accurate on average. A model trained with mean absolute error will be more robust to outliers. Small decisions here have big downstream effects.

But what exactly is the model adjusting? That’s where parameters come in.

9. Parameters & Hyperparameters

Parameters are what the model actually learns.

Think of them as the model’s memory: every pattern gets encoded as a number. GPT-4 has hundreds of billions. A simple recommendation model, a few million. You don’t write them. The model discovers them on its own.

Hyperparameters are the settings that control how the model trains.

You set them before training starts: how fast the model adjusts (learning rate), how many examples it processes at once (batch size), how many times it loops through the data (epochs), and the model’s architecture.

If the model is a student, parameters are the knowledge in their head, and hyperparameters are the study plan: how many hours a day, which subjects first, and how many practice tests.

Bad hyperparameters can make a good model fail.

A learning rate that’s too high, and the model never converges. Too low, and it takes forever to train. Too few epochs and it underfits. Too many and it overfits. In practice, teams run many training experiments with different hyperparameters and compare results on the validation set.

There’s one type of parameter that powers almost every ML system you’ll encounter in interviews: embeddings.

10. Embeddings

An embedding is a way to represent something (a word, a user, a product) as a list of numbers that captures its meaning.

Instead of treating “cat” and “kitten” as completely different labels, embeddings place them close together in a vectorial space because they’re related concepts.

Think of it as a map. On a regular map, cities that are near each other geographically are plotted close together. In an embedding space, concepts that are similar in meaning are plotted close together. “Comedy” and “rom-com” would be neighbors; “horror” would be distant, but still in the same galaxy as other genres. “Accounting” would be in another universe.

Embeddings are everywhere in real ML systems.

Spotify uses song embeddings to find tracks that are musically similar to what you’ve been listening to. Netflix uses movie embeddings and user embeddings to match viewers with films they’re likely to enjoy.

Embeddings are how ML systems make sense of the real world. A user, a product, a sentence. Turn them into vectors, and suddenly you can calculate which ones are similar, which ones are related, and which ones to recommend.

With all this learning machinery, there’s a critical danger: the model might learn too well.

11. Overfitting & Regularization

Read more

• Share this post & I'll send you some rewards for the referrals.

• Block diagrams created using Eraser.

Onwards ‘n downwards:

Following is the second of a premium 3-part newsletter series…

If you design backend systems and want to avoid building APIs that break under flaky networks, retries, and real-world usage patterns… read this newsletter and start designing for production apps…

On with part 2 of the newsletter:

Less prototype. More production. 🚀 (Partner)

The biggest hurdle for AI Agents in production is trust.

Hard-coded credentials work for a demo, but production agents need a way to handle critical actions—like making a purchase or sending a document—without over-privileged access.

Auth0 for AI Agents features Asynchronous Authorization backchannel authentication (CIBA), so your agents can work autonomously in the background and only trigger a notification when human approval is required.

• Rich Consent: Agents render specific authorization data so users know exactly what they are approving.

• Secure User Identity: Link agent actions directly to specific user preferences and history.

Move past the “framework” stage and build agentic workflows that are secure by design.

Start building for $0

And stop worrying about DIY-ing auth.

(Thanks to Auth0 for partnering on this post.)

In this newsletter, you’ll learn 20 more concepts:

20. Conflict Resolution Strategies,

21. Delta Sync,

22. Eventual Consistency in Mobile Systems,

23. Background Sync & Retry Queues,

24. Optimistic UI Updates,

25. Authentication (JWT/OAuth),

26. Secure Token Storage,

27. Certificate Pinning,

28. Biometric Authentication,

29. Code Obfuscation & Tamper Detection,

30. Secure Deep Linking,

31. User Data Privacy,

32. App Lifecycle & State Management,

33. Background Processing Limits,

34. Battery Optimisation,

35. Network Optimisation & Rate Limiting,

36. Startup Time Optimisation,

37. Rendering Performance,

38. Memory Management & Leak Prevention,

39. App Size Optimisation.

(…and much more in part 3!)

For each concept, I’ll cover:

• What it is and how it works

• Real-world example

• The tradeoffs

• Why it matters for mobile

Let’s get into it…

I want to reintroduce Shefali Jangid as a guest author.

She’s a web developer, technical writer, and content creator with a love for frontend architecture and building things that scale.

Check out her work and socials:

• Shefali.dev

• GitHub

• Twitter

You’ll often find her writing about web development, sharing UI tips, and building tools that make developers’ lives easier.

Data Sync

When apps store data on the device, they also need to keep that data in sync with the server.

This section explains how apps synchronize data, handle conflicts when changes occur across multiple places, and maintain app consistency even when the network connection is slow or unreliable.

20. Conflict Resolution Strategies

When mobile apps allow users to edit data offline, a new challenge appears: conflicts.

Why Conflicts Happen

Imagine two users editing the same document:

1. User A edits the document while offline.

2. User B edits the same document at the same time.

3. Later, both devices reconnect and try to sync.

Now, the server has two different versions of the same data. The system must decide how to merge or choose between them.

To solve this problem, apps use various conflict-resolution strategies.

Common Conflict Resolution Strategies

There are several ways apps handle these conflicts:

a. Last-Write-Wins (LWW)

This is the simplest approach.

Each update includes a “timestamp”. When two versions conflict, the system keeps the latest update and discards the older one.

For example:

• User A updates at 10:01 p.m.

• User B updates at 10:02 p.m.

Since User B’s change happened later, the system keeps that version.

This approach is simple to implement and works well for simple data, but earlier changes may be lost.

b. Versioning

Versioning tracks different versions of the same data.

Each update increases the version number or records which device made the change. When a conflict occurs, the system can compare the versions and either automatically merge them or ask the user to choose.

This approach is more flexible, but requires extra logic to manage versions.

c. CRDTs (Conflict-Free Replicated Data Types)

CRDTs are special data structures designed to merge changes automatically without conflicts.

They use mathematical rules that ensure that different versions of the same data can always be merged into a single, consistent result, even when updates occur in different places.

This approach is often used in real-time collaboration tools.

Why It Matters

Without a proper conflict strategy, one user’s changes might silently overwrite another user’s work.

For example, someone may edit a document, sync it later, and suddenly see that their changes have disappeared.

This can break user trust and cause data loss.

Real-World Example

Many well-known applications use these strategies:

• Figma and Notion use “CRDT” based systems to support real-time collaborative editing.

• Dropbox often uses a “last-write-wins” approach and may generate a “conflicted copy” when it cannot merge two versions.

Trade-offs

Each approach has its advantages and limitations:

• Last-Write-Wins: Very simple to implement, but can cause data loss because earlier changes may be overwritten.

• CRDTs: Automatically merge changes without conflicts, but are complex to implement and not suitable for every type of data.

• Versioning: Keeps track of multiple changes clearly, but may require users to manually resolve conflicts.

Once conflicts can be resolved safely, syncing becomes more efficient when only changes are transferred.

Reminder: this is a teaser of the subscriber-only newsletter, exclusive to my golden members.

When you upgrade, you’ll get:

• High-level architecture of real-world systems.

• Deep dive into how popular real-world systems work.

• How real-world systems handle scale, reliability, and performance.

And much more!

Unlock Full Access

21. Delta Sync

When mobile apps sync data with a server, downloading the entire dataset every time can be slow and waste a lot of data. This is especially true for apps that store large amounts of information.

To make syncing more efficient, many apps use a technique called delta sync.

Delta sync means the app downloads only the data that has changed since the last sync, rather than downloading everything again.

How It Works

When the app syncs for the first time, it downloads the full dataset from the server.

After that, the server gives the app a sync token or timestamp that represents the latest state of the data.

On the next sync:

1. App sends the last sync token to the server.

2. The server checks what has changed since that token.

3. Server sends only new, updated, or deleted records.

This makes syncing much faster and more efficient.

Why It Matters

Imagine a contacts app with 5,000 contacts.

If the app downloaded all contacts every time it synced, it would waste a large amount of bandwidth and battery. With delta sync, the app downloads only the contacts that were added or changed, saving both data and processing time.

This makes background syncing practical even on mobile networks.

Real-World Example

Many well-known platforms use delta syncing:

• Google Contacts uses a “syncToken” to track changes.

• Apple CloudKit uses a “serverChangeToken” for the same purpose.

The app stores this token and sends it during the next sync request, so the server knows exactly what updates to return.

Trade-offs

Delta sync improves efficiency but requires extra work on the server.

The server must keep track of which data has changed. This usually means storing timestamps or maintaining a change log.

Deletions must also be tracked. Instead of completely removing a record, the system may temporarily store a “tombstone” record that indicates the item was deleted, so the client knows to remove it as well.

Practical Tip

The sync system should rely on timestamps or sequence numbers generated by the server, not the client.

Client devices may have incorrect clocks, which can cause sync problems. Using a server-controlled timestamp or sequence number ensures that the order of updates is always correct.

Even with efficient syncing methods like delta sync, distributed systems can still struggle to maintain perfect consistency across devices.

22. Eventual Consistency

Mobile apps often run in environments with slow, unstable, or completely unavailable network connections. Because of this, different devices may not always have the exact same data at the same time.

This situation is called eventual consistency.

Eventual consistency means that data may temporarily differ across devices, but all devices will eventually reach the same state once they sync with the server.

How It Works

In mobile apps, users may perform actions while offline or on unstable networks.

For example:

1. A user updates some data on their phone.

2. The change gets saved locally on the device.

3. When the network connection becomes available, the app sends the update to the server.

4. Server then sends the updated data to other devices.

For a short period, different devices may show slightly different versions of the data, but eventually they all become consistent after syncing.

Why It Matters

Trying to keep data perfectly synchronized at all times would require the app to wait for the server to confirm every action before updating the interface.

This would make the app feel slow and unresponsive.

Instead, mobile apps usually update the UI immediately, while ‌server synchronization happens in the background.

This approach makes the app feel fast and responsive while still keeping the data consistent over time.

Real-World Example

Messaging apps are a common example of eventual consistency.

For example, WhatsApp shows different message states:

• A single check mark means the message has been sent from the device.

• A double check mark means the message has been delivered to the recipient.

The message appears in the chat immediately, even before the server fully confirms it. Later, the status updates when the system finishes syncing.

Trade-offs

Eventual consistency improves user experience, but it also introduces some challenges…

Sometimes the server may reject a change or update the data differently. When this happens, the app must update the UI carefully to avoid confusing the user.

To handle this, many apps display clear states such as pending, confirmed, or failed. This helps users understand what is happening with their actions.

To handle temporary failures during syncing, apps often rely on background processes that automatically retry operations, as discussed in the next section.

23. Background Sync & Retry Queues

Mobile apps often perform actions that need to be sent to a server, such as sending a message, liking a post, or submitting a form. But sometimes the network connection may be weak or unavailable, causing these actions to fail.

Instead of losing these actions, apps use a system called a retry queue.

A retry queue stores failed operations locally on the device so they can be retried later when the network connection is available.

Mobile operating systems also provide tools that allow apps to perform tasks in the background efficiently.

For example:

• WorkManager on Android

• BGTaskScheduler on iOS

These tools allow apps to run background tasks while managing battery usage and system resources.

How It Works

When a user performs an action that requires a network request, the app tries to send it to the server:

1. User performs an action (for example, sending a message).

2. If the request succeeds, the server processes it normally.

3. If the request fails because of network issues, the app stores the action in a retry queue.

4. When the device reconnects to the network, a background sync process retries the queued operations.

5. Once the server confirms the request, the action is removed from the queue.

This ensures that user actions are not lost even when the network is unreliable.

Why It Matters

Users expect their actions to work, even if the network connection is unstable.

For example, if a user taps “send message” or “like”, the app should not simply ignore the action because the network was unavailable.

By storing the action and retrying it later, the app respects the user’s intent and completes the action when conditions improve.

Real-World Example

Many apps use retry queues and background syncing:

• Email apps like Gmail and Outlook allow users to send emails while offline. The email is queued and automatically sent when the network returns.

• E-commerce apps may queue actions such as adding items to a cart or saving user activity and sync them later.

Trade-offs

Background processing on mobile devices has limitations.

Operating systems control how long apps can run in the background to protect battery life. These background windows are usually short and unpredictable. Because of this, apps should not rely only on background syncing. Synchronization should also happen when the user opens the app again.

These mechanisms also allow the app’s interface to respond instantly, while the actual synchronization happens quietly in the background.

24. Optimistic UI Updates

In many apps, users expect actions to happen instantly. For example, when you tap “like”, send a message, or add a comment, you expect to see the result immediately.

If the app waited for the server to respond before updating the screen, it could take a few hundred milliseconds. Even that slight delay can make the app feel slow.

To solve this, apps use a technique called optimistic UI updates.

Optimistic UI means the app assumes the request will succeed and updates the interface immediately, even before the server confirms it.

The server request still happens in the background. If everything succeeds, nothing changes. If it fails, the app corrects the UI and shows an error.

How It Works

The process usually follows these steps:

1. User performs an action (e.g., tapping a like).

2. App updates the UI immediately and stores the change locally.

3. App sends the request to the server.

4. If the server confirms the request, the change remains unchanged.

5. If the server rejects the request, the app reverts the change and shows a small error message.

This makes the app feel fast while still keeping the data accurate.

Why It Matters

Users expect apps to respond instantly to their actions.

If the app shows a loading spinner every time the user taps a button, the experience feels slow and frustrating.

Optimistic UI removes this delay, making the app feel smooth and responsive, similar to native device interactions.

Real-World Example

Many popular apps rely heavily on optimistic UI:

• Twitter shows the like animation immediately when you tap the heart.

• Instagram displays your comment instantly in the thread.

• Slack shows a message in the chat immediately after you send it.

In the background, the app syncs with the server and updates the data if needed.

Trade-offs

An optimistic UI improves the user experience, but it also adds complexity.

If the server rejects the request, the app must roll back the UI change smoothly without confusing the user. Because of this, optimistic UI is usually avoided for actions that cannot be easily reversed, such as permanent deletion, payments, and financial transactions.

As app stores and sync sensitive user data, security becomes a critical part of system design…

Let’s keep going!

Read more

• Share this post & I'll send you some rewards for the referrals.

ChatGPT, Claude, and Gemini handle general-purpose tasks and one-off conversations well…

The moment your assistant is customer-facing, the requirements change:

Your users need an assistant that knows your product, your policies, and their history with you. That knowledge lives in your systems, NOT in a model’s training data. You could try to inject it at runtime, but proprietary data comes with real constraints: it is often too large to fit in a context window, too sensitive to send to a third-party API, and too dynamic to stay current in a static prompt.

Beyond data, you cannot fully configure its behavior; you cannot integrate it past a browser tab, and costs scale with every conversation…

Here is what changes when you build your own:

• Privacy and data control. When you build your own assistant, conversations stay on your infrastructure. You decide what gets logged, how long it gets stored, and when it gets deleted. For healthcare, legal, finance, and enterprise use cases, this is a compliance requirement, not a nice-to-have.

• Full control over behavior. You own the system prompt, persona, and the guardrails. You can enforce a specific tone, restrict responses to your product domain, swap the underlying model, or A/B test different configurations. With an off-the-shelf product, you get what they ship.

• Cost at scale. You can implement prompt caching, model routing (a cheap model for simple questions, an expensive one for hard ones), and context management strategies that dramatically cut costs. With thousands of conversations per day, the savings are significant.

• Product integration. Your assistant lives inside your product, not in a separate tab. It shares your auth system, your UI, and your user context. That level of integration is not possible when you are wrapping someone else’s chat interface.

• Security. You control the full request path: what goes into the model, what comes out, and what filters run in between. You implement your own prompt injection defenses, output guardrails, and content policies.

Off-the-shelf tools are great for personal use and internal prototyping. Building your own is for when the assistant is the product or a core feature.

Enterprise auth from day one. No upgrade required. 🚀 (Partner)

Enterprise auth is intimidating without a doubt - SSO, SCIM, etc.

You may feel these features are next year’s problem because the price jump to unlock basic SAML or OIDC connections is just too brutal for an early-stage roadmap.

With Auth0, ship your first Enterprise Connection and SCIM setup for $0 on their free tier. If you’re scaling AI Agents, security stack is built for production:

• Token Vault: Securely connect 3rd-party tokens (Slack, GitHub) to stop hard-coding API keys.

• FGA for RAG: Ensure LLMs respect fine-grained permissions to prevent data leaks and also control access to specific AI tools/models.

• Async Authorization: Use backchannel authentication for human-in-the-loop approvals on critical agent actions.

• User Authentication: Securely identify users to unlock chat history, order tracking, and personalized settings within your AI agents.

Start building for $0

And stop worrying about DIY-ing auth.

(Thanks to Auth0 for partnering on this post.)

I want to reintroduce Louis-François Bouchard as the author of this newsletter.

He’s a best-selling author (Building LLMs for Production), the co-founder of Towards AI, and the creator of the YouTube Channel, What’s AI, where he helps people understand AI and learn how to apply it in the real world. Through his development work with clients and his content, teaching, and AI training programs on the Towards AI Academy, Louis focuses on making AI practical for builders, engineers, and curious learners alike.

At Towards AI, he and his team train AI engineers through courses built for every stage, from beginner to advanced. That educational mission and the real-world experience building for his clients are exactly why I wanted him in this newsletter series.

Inside this newsletter, you’ll get:

• What happens when you send a message. Tokenization, prefill, and decode phases, KV-cache, and why longer conversations get slower and more expensive.

• How the model got here. Pretraining, post-training (SFT and RLHF), and scaling laws, and what each one means for the assistant you are building.

• Output controls. Temperature, top-p, top-k, why hallucination happens, and which settings to use for which task.

• Context engineering. System prompts, few-shot, chain-of-thought, structured outputs, context window management, and persistent memory across sessions.

• Cost and how to know it works. Worked token math across model tiers, prompt caching, model routing, golden test sets, and LLM-as-Judge evaluation.

• A practical eight-step build. From a single API call to a production-minded assistant with streaming, system prompts, JSON output, multi-turn compaction, prompt injection defense, and persistent memory.

What We Are Building

We’re building a ChatGPT-style conversational assistant that you fully control: your own system prompt, your own personality, your own rules.

Here are the requirements:

• Multi-turn conversation with context preserved across turns.

• Streaming responses, where words appear as they’re generated.

• Configurable behavior, creative for brainstorming, precise for data extraction.

• Resistant to prompt injection attacks.

• Cost-effective at scale, with thousands of conversations per day.

One extra feature covered later: persistent memory across sessions, so the assistant remembers user preferences between conversations.

What this system does not do: it does NOT retrieve external documents, it does NOT have domain expertise beyond its training data, and it does NOT take actions or call tools. It’s purely conversational.

We use a model API (OpenAI, Anthropic, or similar) for this build.

The provider handles GPUs, scaling, and inference optimization, the right starting point for most teams. When you outgrow APIs, you move to self-hosted open-source models served with vLLM, llama.cpp, or SGLang.

This newsletter is organized around 4 core design decisions:

1. What happens when you send a message?

2. How do you control what the model says?

3. How do you engineer the context for every request?

4. What does it cost, and how do you know it works?

(Then we build the whole thing from scratch in Part 5.)

Let’s start with what happens under the hood when you send a message…

Part 1: What Happens When You Send a Message

When you call a model API, a lot happens between your request and the first token appearing on screen.

Even if you never need to use a GPU, this helps you understand the latency, cost, and trade-offs…

From Text to Tokens

Before the model can process your message, it needs to convert it from text into numbers.

Models do not read words. They work with tokens, small chunks of text mapped to numerical IDs. A token might be a whole word (“the”), part of a word (“un”, “believ”, “able”), or a single character.

The process of splitting text into tokens is called tokenization.

As we covered in Part 1, there are different ways to split text into tokens:

You could split by individual characters, but then the model has to process very long sequences just to read a short sentence. You could split by whole words, but then the vocabulary becomes enormous, and the model cannot handle any word it has not seen before.

Subword tokenization is the sweet spot, and virtually all modern models use it.

The most common algorithm is Byte Pair Encoding (BPE). BPE starts with individual characters and iteratively merges the most frequent pairs into a single token. Common words like “the” or “and” become single tokens. Rare or long words get split into meaningful pieces. The word “unbelievable” might be broken into three tokens: “un” + “believ” + “able.” This keeps the vocabulary manageable, typically 30,000 to 100,000 tokens. And if the model encounters a word it has never seen before, it can still process it by breaking it into known subword pieces.

This is why language models struggle with certain tasks that seem trivial to humans…

Ask “How many r’s are in strawberry?” and the model might get it wrong because it never sees the individual letters. The word “strawberry” might be tokenized as “str” + “aw” + “berry,” so the model literally cannot count the r’s because they are split across token boundaries. Reversing a word, counting specific characters, and other character-level operations are all hard for the same reason: the model operates on tokens, not letters.

Tokenization has direct practical consequences for building a chat assistant.

Token count does not equal word count: a typical English sentence of 10 words might be 13 to 15 tokens. Code is usually more token-dense than prose. Non-English languages often require more tokens per word because BPE vocabularies are built primarily from English text. Since API pricing, context window limits, and latency all scale with token count, understanding this conversion is essential for cost estimation and context management, both of which we cover later in this newsletter.

Reminder: this is a teaser of the subscriber-only newsletter series, exclusive to my golden members.

When you upgrade, you’ll get:

• Simple breakdown of real-world architectures

• Frameworks you can plug into your work or business

• Proven systems behind ChatGPT, Perplexity, and Copilot

Unlock Full Access

How the Model Got Here: From Pretraining to Chat

Before we look at what happens on each request, it helps to understand what happened before you ever sent a message…

The model you are calling went through a multi-stage pipeline to become a chat assistant, and each stage has direct consequences for the product you are building.

Pretraining is where the model learns language.

As covered in Part 1, the model is trained on a massive corpus, books, websites, code, research papers, and learns to predict the next token given all the tokens before it. This is self-supervised: no one labels the data. The model just reads trillions of tokens and learns patterns. Pretraining is what gives the model its general knowledge, its grammar, its ability to write code, and its tendency to produce plausible-sounding text, whether or not it is true.

When your chat assistant confidently states a wrong fact, that behavior traces back to pretraining: the model learned to produce likely continuations, not verified truths.

Post-training is what turns a text predictor into a chat assistant.

The pretrained model is good at completing text, but it does not know how to follow instructions, hold a conversation, or refuse harmful requests. Post-training fixes this in stages.

First, supervised fine-tuning (SFT) trains the model on curated examples of good conversations: a user asks a question, and a human-written response shows what the ideal answer looks like. This teaches the model the format and style of a helpful assistant.

Then, reinforcement learning from human feedback (RLHF) refines the model further. Human raters compare pairs of model responses and pick the better one. The model learns from those preferences, becoming more helpful, more accurate, and less likely to produce harmful or unhelpful output.

This is why the same base model can feel completely different depending on how it was post-trained. It is also why system prompts work: the model was specifically trained to follow developer-level instructions during post-training.

Scaling laws govern the relationship between model size, training data, and performance.

Research (notably Kaplan et al. at OpenAI and the Chinchilla paper from DeepMind) showed model performance improves predictably as you increase the number of parameters and training tokens, following power-law curves.

This is why the industry keeps building bigger models: doubling model size produces measurable, predictable gains in capability. For you as a builder, scaling laws explain the cost-capability tradeoff you face.

A frontier model (hundreds of billions of parameters) costs more per token but handles harder tasks. A lightweight model (fewer parameters, less training) is cheaper and faster but less capable. The tiered pricing you see from API providers, frontier versus mini versus nano, maps directly to where each model sits on the scaling curve.

Understanding this helps you make the model routing decisions we discuss later: use the big model when you need its capabilities, and the small model when you do not.

Let’s keep going!

Read more

• Share this post & I'll send you some rewards for the referrals.

A single agent1 has one context window2, one set of tools3, and one running loop4.

When a task outgrows any of those three, you need more than one agent. That’s what a multi-agent system is. Instead of one agent doing everything, you split the work across many agents. Each one has its own role, tools, and context.

Most multi-agent systems fail not because the model is weak but because you chose many agents before you actually need them, or chose the wrong architecture once you have many agents.

So you shouldn’t split anything yet.

Onward.

[Webinar] Stop babysitting your coding agents (Partner)

Agents can generate code. Getting it right for your system, team conventions, and past decisions is the hard part – you end up wasting time and tokens in correction loops.

More MCPs give agents access to information, but not understanding. The teams pulling ahead use a context engine to give agents exactly what they need.

Join Unblocked live on May 6 (FREE) to see:

• Where teams get stuck on the AI maturity curve

• How a context engine solves for quality, efficiency, and cost

• Live demo: the same coding task with and without a context engine

Register Now

(Thanks to Unblocked for partnering on this post.)

Here’s what’s inside:

• Why single agents break. Context overflow, slow serial work, and why one agent can’t always hold every tool, model, or permission.

• The six architectures. Orchestrator-worker, pipeline, hierarchical, swarm, mesh, and handoffs. Plus, where each one works and breaks.

• How agents coordinate. Run loops, MCP vs A2A, shared state, isolated state, memory, and stopping conditions.

• The real cost. Why more agents mean more tokens, more latency, more coordination overhead, and more ways to fail.

• Failure and security risks. Bad instructions, misalignment, weak verification, prompt injection, context contamination, and privilege creep.

• Case study. How Spotify used an orchestrator-worker system to turn ad planning from a 15-minute workflow into 5 seconds.

Limits of single-agent systems

One agent with good prompts and the right tools handles more than most people expect.

For example, Cognition’s Devin processed 5 million lines of COBOL (Common Business-Oriented Language) across 500GB of repositories with a single agent, raising its pull request merge rate from 34% to 67%.

But a single agent has three HARD limits. When your task runs into any of them, better prompts won’t help:

1. Context overflow

A context window can only hold so much.

Past that limit, the earliest information drops out, and the agent starts losing track of its own plan. When compression5 alone can’t fix the overflow, a second agent with its own context can.

2. Parallelism

Independent tasks shouldn’t wait in line.

If you have four research queries that don’t depend on each other, running them one at a time wastes time. Running them across four separate agents takes roughly as long as the slowest one.

Anthropic’s research system uses this exact pattern and reduced total query time by up to 90%.

3. Specialization

Different parts of a task often need different models6, tools, or access levels:

• A code-writing agent needs a sandbox

• A research agent needs web search

• A customer-facing agent needs user data, but shouldn’t have access to production databases

When one agent can’t hold all the tools and permissions the task needs, you give each role its own agent.

If none of those three conditions apply, stay with one agent.

Better prompts and better tools solve most problems without adding the extra work of coordinating agents. But once you know you need many agents, the next question is what shape the system takes...

Multi-agent architectures

Every multi-agent system makes a different choice about who coordinates work.

Here are 6 architectures that range from tight central control to NO coordinator at all:

1. Orchestrator-worker

One central agent breaks a task into pieces, assigns each piece to a worker agent, and then puts the results together:

• Workers don’t talk to each other; all communication goes through a central agent

• Orchestrator calls each worker as a tool call7, waits for a result, and decides what to do next

This is like an air traffic control tower that talks to every plane, while no plane talks directly to another.

Anthropic’s Claude Research system works this way:

A central agent running Opus 4 breaks a research query into parts and creates 2 to 10 worker agents on Sonnet 4 (sometimes more), all at the same time. The workers search the web, read documents, and gather evidence in parallel. When they finish, the central agent reads their results and writes a single research report.

This setup beat single-agent Opus 4 by 90.2% on Anthropic’s internal research eval8.

Tradeoffs

Central agent is both coordinator and bottleneck…

It talks to workers one at a time. If each call takes 3 seconds and 20 workers are waiting, the ceiling is about 7 tasks per second. So the central agent becomes the slowest part of the system.

Anthropic’s Claude Research system had this problem as well: workers duplicated each other’s work.

Without specific instructions, many workers run overlapping searches on the same topic, wasting both tokens9 and time. The takeaway is orchestrator-worker depends entirely on the quality of the lead agent’s instructions.

Vague task splitting turns parallelism into duplicated work.

2. Pipeline

Agents run in a fixed order, one after another.

Each agent’s output becomes the next agent’s input, and entire sequence is set in advance. While orchestrator-worker lets the central agent decide what to do as it goes, pipeline removes that choice.

An assembly line works the same way: each station does one job, passes the result forward, and never sees the finished product.

Stripe uses this pattern to check whether new businesses on its platform are legitimate.

Before agents, a human reviewer had to jump between customer databases, legal sources, and support tickets to decide whether a business was safe to approve. Now their engineering team broke that work into a fixed flow of agent stages using a directed acyclic graph (DAG). So work moves forward through stages without looping back.

Order is set at design time, and each stage has a contract: defined output format next stage expects.

Stripe calls these contracts “rails” because they keep any single agent from spending too much time on irrelevant data. This setup cut their average handling time by 26%, and reviewers rated the agent outputs 96% helpful, with a full record of every decision at every step.

Tradeoffs

Latency adds up.

A 5-stage pipeline where each stage takes 2 seconds means a 10-second wait before any output, and adding a stage to improve quality increases the response time.

Yet the upside is predictability.

When every stage has a narrow contract, you can trace any failure back to exactly one step. That’s why regulated workflows like Stripe’s continue to use pipelines.

When the cost of being wrong is a regulator flagging your process, the extra seconds are a small price to pay.

3. Hierarchical

A top-level manager agent gives work to one or more layers of manager agents below it, which then give work to individual workers.

Two levels are the minimum; big systems stack more. The result is a tree…

No single agent needs the full context.

The top-level agent holds the high-level goal and a summary from each branch, while each lower level sees only what its narrower role needs. Picture a military chain of command where orders flow down, reports flow up, and no one skips a level.

IBM watsonx Orchestrate runs on this pattern:

A top-level supervisor agent acts as a router and planner across 80+ pre-built domain agents for HR, sales, and procurement. Let’s say a user tries to “order new laptops for the design team”. The request reaches a Procure Equipment supervisor, who then hands the work to three specialized child agents:

• One requests quotes from approved vendors

• Another checks responses

• A third submits a purchase request

The supervisor decides only who gets called and in which order.

Tradeoffs

Details might get lost at each level.

A worker produces a detailed result. The mid-level manager shortens it to one sentence. By the time it reaches the top, the detail that matters might be gone. Hierarchical structures trade detail for coverage: the higher you go, the wider the scope of each agent and the less it knows about any specific piece.

These three architectures share one thing: a clear chain of command tells you exactly who’s in charge…

The next three architectures drop the chain entirely. So they're harder to debug, but they survive partial failures better…

Reminder: this is a teaser of the subscriber-only newsletter, exclusive to my golden members.

When you upgrade, you’ll get:

• High-level architecture of real-world systems.

• Deep dive into how popular real-world systems work.

• How real-world systems handle scale, reliability, and performance.

Unlock Full Access

4. Swarm

In a swarm, many agents work as equals.

They coordinate through a shared blackboard: a data store (typically Redis cache, database table, or vector store) that every agent can read from and write to. Yet there are NO direct messages between agents…

Read more

• Share this post & I'll send you some rewards for the referrals.

You ship an LLM feature.

It passes your manual tests. A day later, a user posts a screenshot of it hallucinating wildly! You tweak the prompt, run it again, and it works fine. Did you fix it, or did you just get lucky?

Welcome to the central frustration of LLM engineering: you can NO longer just run a test and call it done.

This isn’t a debugging problem…

It’s a measurement problem. And measurement has a name: evaluation.

Most articles on LLM evaluation are written for ML researchers. This one is for engineers building real applications. You know how to ship software. You’re just new to the ways LLMs fail.

We’ll cover the vocabulary, methods, and mental models. By the end, you’ll have a framework for building an eval system from scratch. Not just an understanding of why it matters.

Let’s start with why your existing testing instincts don’t work here…

Your team’s second brain. Now in Slack. (Partner)

Your engineers talk on Slack. They code in the terminal. Somewhere between those two things, context goes to die.

• A bug was debated in #incidents at 2 AM.

• An architectural call was made in a DM.

Every handoff leaks context, and every leak costs you. That’s the context tax - and your team pays it every day.

CodeRabbit Agent for Slack is built for agentic SDLC workflows. One agent for your entire Software Development Lifecycle, living in the channel where the work already happens. It’s built on four things:

• Context - your org’s operating picture, pulled from across code, tickets, docs, monitoring and cloud.

• Knowledge Base - a living memory of your team. Every run leaves a trace, so yesterday’s decisions don’t become tomorrow’s debates.

• Multi-Player - works in shared threads alongside your team. Steerable, resumable and aligned as work evolves.

• Governance - scoped access, cost attribution. Every run explainable and attributed.

Your team keeps shipping. Agent keeps the context.

From the team that pioneered AI code reviews. 2M code reviews every week. 6M repos. 15K customers. And now, one agent for your entire SDLC, right in Slack.

Try CodeRabbit's Agent Today

(Thanks to CodeRabbit for partnering on this post.)

I want to introduce Anshuman as a guest author.

He leads evals efforts at Zomato, where he built Gavel -- an internal LLM eval platform that started as a handful of scripts, got pitched to a VP, and now serves AI and ops teams across the company. Now he’s on the ground floor of making a large organization AI-native, one eval system at a time.

I highly recommend you checkout his newsletter, AI Proof -- it’ll help you stay relevant in the AI era.

Before we get into solutions, you need to understand something…

LLM evaluation feels nothing like regular software testing. It’s not that it’s harder. It’s that the rules changed.

Let’s dive in!

1. Non-determinism Problem

Write a function.

Call it with the same input twice, and you’ll get the same output twice.

But LLMs don’t work that way.

The same prompt can produce a different response on every run.

Sometimes slightly different. Sometimes completely different. This isn’t a bug. It’s called temperature.

It controls how random the outputs are. (More on this later.)

This breaks something deeper. Your entire mental model for testing assumes determinism. You run a test. It passes or fails. You know what you know.

With LLMs, a passing test is a data point, not a verdict. You’re not testing a function. Instead, you’re sampling from a probability distribution. One sample tells you almost nothing.

This is the first reason your existing instincts mislead you.

2. Fuzzy Correctness Problem

When a regex matches, it matches. Binary. Objective. Done.

But what’s the correct response to “summarize this support ticket empathetically?”

• Is the correct answer the one that’s most concise?

• The one that uses the warmest language?

• The one that captures all the key facts?

• The one a human reviewer would score highest?

LLM quality is multi-dimensional and subjective.

There usually isn’t one correct answer. There’s a range of acceptable answers and a range of bad ones. The line between them is a judgment call.

You can’t measure what you haven’t defined.

What does ‘good’ mean for your use case? You need a clear answer to that before you can evaluate anything. Most people skip this step. It comes back to haunt them.

3. Silent Regression Problem

You updated your prompt.

Ran it a few times. The outputs seemed better. You shipped it.

But did quality actually improve, or did you get lucky with the samples you checked?

This is the silent regression problem. Without a systematic evaluation process, every prompt change is a blind bet. You might be making things better. You might be fixing one failure mode while introducing another. And you have no way of knowing.

In traditional software, CI (Continuous Integration) catches regressions before they reach users. In LLM engineering, there’s NO such equivalent. So people rely on their gut feelings and manual spot checks. You find out about regressions when a user complains!

These three problems make LLM evaluation its own discipline…

Now let’s build the vocabulary to actually talk about it…

Primitives of Eval

These are six terms you’ll hear constantly.

Learn these, and the rest falls into place:

4. Criteria

Criteria are the dimensions of quality that matter for your specific use case.

A customer support bot might care about:

• Does it address the user’s actual issue?

• Is the tone empathetic?

• Does it avoid triggering escalation unnecessarily?

A code generation tool cares about something completely different:

• Is the output syntactically valid?

• Does it follow the project’s conventions?

• Does it actually solve the stated problem?

Same technology, but entirely different criteria.

This is a product decision, NOT a technical one. You can’t outsource it to your model or your eval framework. Someone on your team needs to sit down and answer: What does a good output actually look like here?

Get this wrong, and everything downstream is measuring the wrong thing…

5. Quality Dimensions

Quality dimensions are the standard industry vocabulary for LLM output quality.

Here are five of them that come up constantly:

• Relevance: Did the output address what was actually asked? A response can be accurate, well-written, and completely beside the point.

• Coherence: Does it hold together logically? No contradictions. No mid-sentence topic shifts. Flows like something a thoughtful person wrote.

• Factual Accuracy: Is what it says actually true? Distinct from relevance. A response can be relevant and wrong.

• Helpfulness: Does it give the user what they need to move forward? The difference between a technically correct answer and a useful one.

• Safety: Does it avoid harmful, biased, or inappropriate content? This dimension matters more in some domains than others -- but it always matters.

Knowing these helps you write rubrics that don’t miss important failure modes.

6. Rubric

Once you have the criteria, you need to operationalize them.

That’s what a rubric does.

Take a vague criterion like “helpfulness.” A rubric breaks it into specific, scorable questions. Does it directly answer the question? Does it avoid unnecessary hedging? Is it under 200 words? Can a non-technical user understand it?

Think of it like a code review checklist.

Instead of “Is this good code?”, a checklist asks: Are there tests? Is the function under 30 lines? Are variable names descriptive? Rubrics do the same thing for LLM outputs.

The rubric is what makes evaluation reproducible.

Two different reviewers, human or AI, should arrive at similar scores. Same output, same rubric, same conclusion.

Without a rubric, every evaluation is just someone’s opinion.

7. Test Cases

A test case is an input/output pair that forms one unit of your evaluation.

Input is a prompt: ideally, one representative of real user traffic.

Output can be one of two things. A reference answer showing what good output looks like. Or the live model output you’re about to score.

Think of test cases like unit tests…

Except that a failing test case doesn’t mean the output was wrong. It means it scored below your defined threshold on your rubric.

That distinction matters.

You need a lot of them. A handful of test cases gives you anecdotes. A few hundred gives you a signal.

8. Golden Set

Everything in your eval gets measured against one thing: your golden set.

A curated collection of high-quality test cases.

Building a good golden set is harder than it sounds:

The instinct is to write examples yourself, covering the use cases you anticipate. That’s a start. But users phrase things differently than you expect. They hit edge cases you didn’t imagine.

And they might misuse features in creative ways.

A golden set built from your imagination reflects your imagination and not your users’. So seed it with real production queries instead. Anonymized, cleaned, representative.

Your golden set is your ground truth.

Treat it like a critical system artifact. Version and update it when you discover new failure modes.

9. Pass/Fail Threshold

Eval scores are rarely binary.

A rubric usually produces a score of 1 to 5, 0 to 10, or a percentage. The pass/fail threshold is what converts that score into a decision.

If your rubric scores range from 1 to 5 and your threshold is 3, any score below 3 is a failure. Simple in theory, but HARD in practice.

Setting the right threshold is a product call, not a technical one.

It depends on a few things: how much imperfection your users can tolerate. How severe are failures in your domain? The cost of false positives.

Set your threshold too low, and you’re shipping garbage. And if you set it too high, you’re shipping nothing.

10. Eval Coverage

Eval coverage is how well your golden set reflects real user inputs.

Most teams have low coverage and don’t know it. They built their golden set from examples they had written. Happy path, a few obvious edge cases. Meanwhile, production traffic is different: weird inputs, unusual phrasings, use cases nobody anticipated.

Low coverage means your eval suite is optimistic…

i.e., you’ll pass your tests but fail your users.

The fix isn’t writing more examples yourself. Instead, sample from production regularly and review failures. Then add the inputs that exposed new weaknesses to your golden set.

Eval coverage is something you build over time.

11. Temperature, Top-p, and Reproducibility

Temperature is the dial that controls how random your model’s outputs are.

Low temperature (close to 0) makes the model nearly deterministic: same input, same output, every time. High temperature makes it more creative. It samples from a wider range of probable tokens, producing more varied responses.

Top-p (nucleus sampling) is a related setting.

Instead of a randomness dial, it sets a probability cutoff. Only the most likely tokens make the cut. Top-p = 0.9 means the model considers only the top 90% most likely next tokens.

Both settings directly affect your eval results…

Run evals at temperature = 1.0 and the same prompt might pass today and fail tomorrow. NOT because your model changed, but because randomness swung against you.

Here’s the standard practice: set temperature to 0 during eval runs. And lock in determinism. If you need creative variance in production, test at your production temperature.

Just know you’re accepting noisier results…

12. Statistical Rigor

Even at temperature = 0, a single eval run isn’t enough.

Your golden set is a sample of your input space. That sample has its own variance. One unlucky set of examples can make a good prompt look bad. And one lucky set can make a bad prompt look good.

So run many evaluations across different samples.

Then report the mean and the variance, not just the score. When comparing two prompt versions, check whether the difference is real. Or if it’s just noise.

In practice: if you change a prompt and your score goes from 4.1 to 4.3, that might be a real improvement.

It might be a random fluctuation. Without variance1 data across many runs, you can’t tell the difference. Most teams run once, report the number, and ship. That’s how confident regressions get deployed…

How Do You Score Outputs?

You have criteria, a rubric, and test cases2.

Now the question is: who does the scoring? There are three options.

Each has different tradeoffs in cost, speed, and accuracy.

13. Human Evaluation

This is the gold standard.

A human reviews the output using your rubric and gives a score. It’s slow, expensive, and can be inconsistent. But it’s the closest thing to ground truth.

You can’t use humans for every eval run; it doesn’t scale.

But you shouldn’t remove human evaluation completely either. It’s what keeps your system grounded. Everything else: metrics, LLM judges, tests, is just an approximation of human judgment.

So use human evaluation strategically:

• To build and validate your golden dataset.

• To periodically check the accuracy of your automated evals.

• To debug when something breaks and you don’t know why.

This way, you balance cost with reliability.

14. Heuristic/Code Based Evaluation

This is the fastest and cheapest type of evaluation.

It uses simple code checks to validate the output's structural properties. For example:

• Is the response valid JSON?

• Is it within the character limit?

• Does it include all required fields?

• Does it avoid banned phrases?

• Does it match a specific format (like a regex)?

Heuristic eval is good at catching structural problems.

But they don’t measure quality. They can’t tell if a response is helpful, accurate, or well-written.

Think of heuristics as your first line of defense.

They catch basic issues before you run more expensive evaluations. They’re NOT enough on their own, but they’re an essential part of a complete eval system…

15. Semantic Similarity Evaluation

Sometimes you have a reference answer, a known-good response representing ideal output.

Semantic similarity evaluation measures how close your model’s output is to the reference. i.e., in meaning, and not exact wording.

This is where embeddings come in…

Each piece of text gets converted into a vector: a list of numbers that represents its meaning. Texts with similar meanings have vectors that are close to each other. Cosine similarity measures the angle between two vectors. A score of 1.0 means identical meaning. A score near 0 means unrelated.

This matters because string matching is too strict.

Take two sentences: “API returns a 404 error” and “Endpoint responds with a not found status.” They mean the same thing, but use different words. An exact match would call the second one wrong. Semantic similarity would call them equivalent.

But here’s the limitation: it only measures closeness to your reference.

It can’t catch a response that’s fluent and factually wrong. Not if the wrong answer happens to be semantically similar to the right one. So use it as a fast, scalable layer. And don’t rely on it alone!

16. Task Specific Metrics (BLEU, ROUGE, Execution-based)

For certain tasks, there are established metrics purpose-built for automated evaluation:

• BLEU (Bilingual Evaluation Understudy): Originally for machine translation. Measures n-gram overlap between generated text and a reference. Good for tasks where exact phrasing matters.

• ROUGE (Recall-Oriented Understudy for Gisting Evaluation): Designed for summarization. Measures recall: how much of the reference answer appears in the output? Useful when you care about coverage: did the model hit all the key points?

• Execution-based evaluation: For code generation, one metric matters. Does it run? Does it produce the correct output? Execution-based eval runs the generated code against test cases and checks the results. A function that returns the wrong answer fails. Another function that passes all tests succeeds.

But all three share the same trade-off:

They measure surface-level similarity to a reference, not actual quality. So use these where they fit.

17. LLM as Judge

This is what makes evaluation scalable…

LLM-as-judge uses a more capable model to evaluate your application’s outputs. You give it three things:

• Original input,

• Output you’re evaluating,

• Your rubric.

The judge then returns a score and an explanation.

Think of it like automated load testing.

Nobody clicks through 10,000 user flows to check for regressions. You can instead automate it. LLM-as-judge does the same thing for quality. It lets you run your rubric across thousands of outputs without thousands of human hours.

GPT-5 and Claude Opus are commonly used as judges. They’re capable enough to apply nuanced rubrics reliably. You’re calling them via API, NO custom training required.

But here’s the catch: judges can be wrong.

They have their own biases and blind spots. An LLM judge is an approximation of human judgment, not a replacement for it.

More on how to handle this in a moment…

18. Pointwise vs Pairwise Evaluation

These are 2 flavors of LLM-as-judge evaluation:

• Pointwise

  • You ask, “Score this output on a scale of 1–5.”

  • It’s simple and fast.

  • And each output needs one evaluation call.

• Pairwise

  • You ask: “Here are two outputs. Which one is better?”

  • This is usually more reliable because comparing is easier than scoring.

  • It’s especially useful when you want to know if a new prompt is actually better.

But the downside of pairwise is the cost…

You’re evaluating two outputs instead of one, so it can be twice as expensive. At a small scale, this doesn’t matter, but at a production scale, it adds up quickly.

Because of this, the industry standard is to use a tiered approach:

• Online evaluation (production monitoring)

  • Use fast, cheap methods like heuristics or smaller, fine-tuned judge models.

  • These can run continuously without high cost or latency.

• Offline evaluation (pre-ship testing)

  • Use the best LLM judge you have.

  • Run pointwise and pairwise comparisons.

  • This is acceptable because you’re only running it before deployments.

Think of the powerful judge model as a gate before deployment, and NOT something you run on every request.

19. Judge Calibration

Before you rely on an LLM judge, you need to know how well it matches human judgment…

Judge calibration measures this:

You take a sample of outputs, have both humans and the judge score them, and then check how often they agree.

• If agreement is high, your judge is a good proxy for human evaluation.

• If agreement is low, the judge may be measuring the wrong thing.

A poorly calibrated judge is risky.

It can give you confidence in results that aren’t actually correct. So calibrate your judge before using it. Then recalibrate periodically, especially after changing models or prompts.

Get 20% savings by getting a group subscription right now:

Get 20% off a group subscription

RAG System Evaluation

Most LLM applications today aren’t just “prompt in, response out.”

They use Retrieval-Augmented Generation (RAG): first fetch relevant documents, inject them into the context, and then generate a grounded response.

This helps reduce hallucinations because the model can rely on real data rather than just its training data. But it doesn’t solve the problem completely…

It also introduces new failure modes that standard LLM evaluation doesn’t catch…

20. RAG Triad

The standard framework for evaluating RAG systems has three dimensions:

1 Faithfulness

Did the answer actually come from the retrieved context?

This often surprises people. You can retrieve the perfect documents, and the model can still ignore them. Faithfulness checks whether the response is grounded in the provided sources.

A failure here looks like a confident answer that isn’t supported by the retrieved documents. So it’s just a hallucination with extra steps.

2 Answer Relevance

Did the response actually answer the user’s question?

A response can be faithful to the context and still miss the point. The model may use the right documents but answer the wrong question.

Answer relevance measures how well the response matches the user’s intent. While Faithfulness checks if the model stayed within the context.

3 Context Precision

Did the retrieval step fetch the right documents?

Even a perfect generator will fail if the input context is poor. If retrieval brings in weak or loosely related documents, the model either guesses or tells the user it doesn’t know.

Context precision evaluates the retrieval stage. It checks how many of the retrieved documents were actually relevant.

Think of a RAG system as three stages: retrieval, augmentation, generation.

Each stage can fail on its own… The RAG triad is your observability layer to isolate where the problem is, so you know what to fix.

21. RAG-Specific Failure Patterns

Understanding the RAG triad is one thing. Spotting failures in practice is another…

Here are the three most common failure patterns:

1 Retrieval returns irrelevant chunks (Context precision failure)

The system retrieves the wrong or loosely related documents. And the model then either:

• Hallucinates to fill the gaps,

• Or says it doesn’t have enough information.

The fix is in your retrieval layer: embedding quality, chunking strategy, re-ranking.

2 Retrieval is correct, but the model ignores it (Faithfulness failure)

The right documents get retrieved, but the model doesn’t use them.

This is usually a prompting issue. The model isn’t being forced to rely on the provided context. The fix is to strengthen the prompt, so the model stays grounded to the sources.

3 The answer is grounded but doesn’t help the user (Answer relevance failure)

The answer is technically correct and grounded, but doesn’t solve the user’s problem. This usually means your knowledge base is missing the right information. The fix is to improve or expand your data, NOT your prompt.

Get 20% savings by getting a group subscription right now:

Get 20% off a group subscription

Offline vs Online

Scoring a single output is one problem.

Building a system that reliably catches failures (before and after deployment) is another. There are two environments where evaluation happens:

22. Offline Evaluation

Offline evaluation happens before you ship.

You make a change: a new prompt, model version, or retrieval strategy. Before it reaches users, you test it against your golden dataset. Then you score the outputs and compare them to your current system.

Think of this like a continuous integration (CI) pipeline for LLMs.

A change shouldn’t go live unless it passes your evals. If quality drops, you catch it before deployment.

Plus, this is where you use your best (most accurate) judge model. For example, GPT-5, Claude Opus. Cost matters less here because you’re only evaluating a finite set of examples, and NOT each user request.

23. Online Evaluation

Online evaluation occurs in production.

You’re sampling live outputs continuously and scoring them. The goal is to catch issues that offline evals miss: edge cases, unusual inputs, and failures that only appear at scale.

Think of online eval as your monitoring system…It helps you catch problems before users report them.

Yet the main constraint is cost. You can’t run your most expensive models on each request. So online eval relies on heuristics and smaller judge models; it’s cheap enough to run continuously.

This means you trade some accuracy for scale…

Offline and online eval should work together:

• Offline eval catches known issues before deployment

• Online eval catches unexpected issues in production

So you need both for a reliable system.

24. Prompt Versioning and Regression Testing

Your prompt is code; treat it that way.

Track every prompt change: use version control, keep history, and compare differences. When something breaks, you should know exactly what changed and when.

Without versioning, you’d NOT know what changed.

Regression testing means running your eval suite against each prompt version. If your old prompt scored 4.2 and the new one scores 3.8, you’ve introduced a regression and caught it before it reached users.

This sounds simple, but most people don’t do it. They lack a solid eval setup: no golden dataset, no clear rubric, no infrastructure.

This is what separates people who iterate confidently from people who ship & pray.

25. Benchmark-Based Evaluation

Benchmarks are standard tests used to compare different LLM models:

1 MMLU (Massive Multitask Language Understanding)

Tests knowledge across many subjects like math, science, law, and medicine. Think of it as a general knowledge exam for LLMs.

2 HellaSwag

Tests common sense reasoning. The model is given the start of a scenario and must predict what happens next. (Many earlier models struggled with this.)

3 HumanEval

Test code generation. The model gets a function signature and must write the correct implementation. It’s measured using pass@k: how often the model gets the right answer within k attempts.

Here are two useful leaderboards:

• Hugging Face Open LLM Leaderboard for open-weight models

• Chatbot Arena (LMSYS), where models get ranked based on human preferences

Remember, these benchmarks are useful for comparing models, but they don’t always reflect real-world performance for your specific use case…

26. Benchmark vs Real-World Tradeoff

Benchmarks don’t tell you if a model will work for your specific use case.

A model that scores 90% on MMLU might still struggle with your domain-specific language. A model that performs well on HumanEval might generate code that doesn’t fit your standards… Benchmarks measure general capability, but your application might need a specific capability.

So use benchmarks to narrow down your options. Then use your own eval on your own golden set to make the final decision.

27. Dataset Contamination / Data Leakage

There’s a problem with benchmarks that most engineers overlook:

Dataset contamination occurs when evaluation data overlaps with the model’s training data. The model has already seen the answers. So its high benchmark score reflects memorization, NOT capability.

This happens because training data and benchmark datasets come from the internet. And the data overlap is often unknown. Over time, widely used benchmarks become less reliable, which is why new ones keep getting created…

To avoid this, don’t rely on public examples for your evals…Plus, use real user queries and create your own datasets instead.

Failure Modes (What Not to Do)

You can have the right tools and still build a BAD eval system.

Here are the common mistakes:

28. Eval Anti-Patterns

Vibe-based evaluation.

“I tried it a few times, and it looked good.”

This is the most common mistake. Informal spot-checking creates false confidence. It doesn’t catch edge cases, track performance over time, or scale beyond one person looking at a few outputs.

Vibes are a starting point, but not an eval system.

The single-sample trap.

You run your eval suite once and report the results.

But LLMs are nondeterministic…

• A bad prompt might look good

• A good prompt might look bad

So run many samples and aggregate results. Remember, report variance, not just average scores.

Goodhart’s Law in disguise.

“When a measure becomes a target, it stops being a good measure.”

If you optimize for a metric too aggressively, the metric stops measuring what you care about…

• Reward confidence: you’ll get confident hallucinations

• Reward length: you’ll get long, low-quality answers

Metrics are only a proxy for quality. So don’t mistake them for the actual goal.

Eval-production mismatch.

Your golden dataset reflects what you expect.

Yet real users behave differently. They’re vague, unpredictable, and use your system in ways you didn’t anticipate. If your evals don’t reflect real usage, your scores are misleading. A high pass rate on unrealistic data doesn’t mean your system works.

Ignoring tail failures.

A 92% pass rate sounds good. But what about the 8%?

LLM failures are often catastrophic, not graceful. One harmful or incorrect response can matter more than hundreds of correct ones. So always review failure cases, not just averages.

The worst outputs tell you the most…

Get 20% savings by getting a group subscription right now:

Get 20% off a group subscription

Decision Framework

There’s no single tool that solves LLM evaluation.

What works is a layered system… Each layer catches problems the previous one misses…

29. Eval Stack

Layer 1: Heuristics.

Fast, deterministic, and cheap. These run on every output and catch structural issues such as incorrect format, missing fields, or banned content.

Layer 2: Semantic similarity and task-specific metrics.

These run automatically against reference answers. They help catch meaning-level failures without the cost of an LLM judge.

Layer 3: LLM-as-judge (offline).

This runs before deployment on your golden dataset using your best judge model. Its job is to catch quality regressions before they reach users.

Layer 4: LLM-as-judge (online, smaller model).

This runs in production on a sample of live outputs using a smaller, cheaper judge model. It helps catch failures at scale that offline evals missed.

Layer 5: Human spot checks.

These run periodically. And keep you calibrated, verify automated evals scoring still match human judgment, and add new failure cases back into your golden dataset.

Each layer has a different cost profile and catches a different failure type…

Eventually, you want all five. But you do NOT need all of them on day one.

So if you are starting from scratch, here is a simple three-step MVP:

Step 1: Build a golden set of 50 examples.

Don’t write all of them yourself.

Use real user queries if you have them. If not, create 30 representative examples and 20 edge cases. Also, include normal cases, adversarial inputs, and strange phrasings you expect users to try.

This dataset becomes the foundation of your eval system.

Step 2: Add one deterministic heuristic.

Pick the single most important structural requirement for your output.

That might be length, format, a required field, or a banned phrase. Then write a simple code check for it. This is quick to build and catches more failures than most people expect.

Step 3: Add one LLM judge prompt.

Take your rubric and turn it into a judge prompt.

Then use a strong model to score outputs on one important quality dimension, and run it on your golden dataset. Read the scores and the explanations carefully.

You will find something surprising…that’s the point.

That is your MVP eval system.

Everything else builds on top of it: pairwise comparisons, online monitoring, RAG scoring, regression testing, and more.

Start here…

Closing Thoughts

Evaluation isn’t something you do at the end.

Instead, it’s what makes iteration possible. Without it, you’re not doing LLM engineering. But guessing and shipping changes, only to find out from users that they didn’t.

Traditional software engineering learned this lesson the hard way: CI, automated tests, production monitoring…these aren’t optional. They’re what make fast, reliable development possible.

LLM systems need the same discipline. The difference is that the outputs are nondeterministic and hard to measure.

Now you have the vocabulary to deal with that…

Start simple and build your eval stack step by step. Over time, you’ll know whether a change improved quality before users ever see it.

👋 I’d like to thank Anshuman for writing this newsletter!

If you’re building LLM applications and want to go deeper, follow him on LinkedIn and X.

Don’t forget to check out his newsletter, AI Proof -- it’ll help you stay relevant in the AI era.

Louis and I launched the GENERATIVE AI MASTERCLASS (newsletter series exclusive to PAID subscribers) this month.

When you upgrade, you’ll get:

• Simple breakdown of real-world architectures

• Frameworks you can plug into your work or business

• Proven systems behind ChatGPT, Perplexity, and Copilot

👉 CLICK HERE TO JOIN THE GENERATIVE AI MASTERCLASS

(Golden members will get the next Generative AI newsletter in the first week of May.)

If you find this newsletter valuable, share it with a friend, and subscribe if you haven’t already. There are group discounts, gift options, and referral rewards available.

Subscribe now

Want to reach 200K+ tech professionals at scale? 📰

If your company wants to reach 200K+ tech professionals, advertise with me.

Thank you for supporting this newsletter.

You are now 210,001+ readers strong, very close to 210k. Let’s try to get 211k readers by 29 April. Consider sharing this post with your friends and get rewards.

Y’all are the best.

Share

• Share this post & I'll send you some rewards for the referrals.

You’ve built a search feature.

Then a user types a query, your application hits the database, and results come back ranked by relevance.

Simple enough, it works perfectly.

But you thought about how search needs to actually understand what users meant, not just match the exact words they typed.

A user searching for “how do I log in” should find results about “account access” and “sign in” even if those exact words never appear in their query.

This kind of search, where the database understands meaning rather than just matching words, is called semantic search. It requires a completely different approach to how you store and search your data.

Yet if you try adding this in as a feature, everything that made your traditional relational database fast will stop working.

The tools you relied on: indexes, WHERE clauses, keyword matching were all built for finding exact values.

Semantic search does NOT work with exact values.

Instead, it works with meaning, and meaning is harder to search efficiently. A search that took milliseconds before now takes seconds, and the bigger your dataset, the worse it gets.

Your database is NOT broken.

It’s doing exactly what you told it to do: comparing your search against every single item in your database, one by one, because there is no other way to find “similar” when there is NO exact match.

This is where vector databases come in.

Not because they are trendy, but because traditional databases were never designed to handle this kind of search at scale.

If you’re a software engineer encountering the term “vector database” for the first time, whether your team is adding AI features or you keep seeing it mentioned everywhere and want to understand what it actually means, this newsletter explains it from the ground up…

[Webinar] Stop babysitting your coding agents (Partner)

Agents can generate code. Getting it right for your system, team conventions, and past decisions is the hard part – you end up wasting time and tokens in correction loops.

More MCPs give agents access to information, but not understanding. The teams pulling ahead use a context engine to give agents exactly what they need.

Join Unblocked live on May 6 (FREE) to see:

• Where teams get stuck on the AI maturity curve

• How a context engine solves for quality, efficiency, and cost

• Live demo: the same coding task with and without a context engine

Register Now

(Thanks to Unblocked for partnering on this post.)

I want to introduce Maxine as the guest author.

She’s a cloud infrastructure engineer who spends her days scaling databases, debugging production incidents at 2 am, and writing about what actually works in production.

You can get a copy of her LLMs for Humans: From Prompts to Production (at 30% off right now). It’s 20 chapters of practical applied AI with real production context, not theory. And it’ll help you get smarter about using AI tools in infrastructure workflows.

Checkout her work:

• Substack - I Love DevOps

• LinkedIn

• Threads

Plus, if you’re thinking about making a career move into cloud or DevOps and want a structured path to get there, get a copy of her The DevOps Career Switch Blueprint.

Inside this newsletter, you’ll get:

• Why traditional search breaks. How exact-match databases fail with semantic meaning and why O(n) similarity search doesn’t scale.

• How vector search works. Embeddings, high-dimensional space, and why similarity is measured by distance instead of exact matches.

• The core architecture. Storage, indexing (HNSW), and query layers, with key tradeoffs in cost, latency, and accuracy.

• Query execution in practice. ANN search, metadata filtering (pre vs post), and ranking, including how real systems return results.

• Production realities. Scaling, replication, cost structure, monitoring, and failure modes teams hit in production.

• Decision framework. When pgvector is enough, when to use a dedicated vector DB, and when distributed systems become necessary.

Why Traditional Databases Struggle with Embeddings

Traditional relational databases are good at exact matches and range queries.

They use indexes and data structures so the database can jump directly to relevant rows without scanning the entire table.

When you query WHERE user_id = 12345, database uses an index to find that row instantly, the same way you’d look up a word in a dictionary: you don’t read every page, you jump straight to the right section. Also when you query, WHERE created_at > ‘2026-01-01’, it works the same way for dates.

But embeddings break this model because there is no exact value to look up!

An embedding is a list of numbers that represents the meaning of a text.

Words, sentences, or entire paragraphs get converted into these numbers by an AI model, and similar meanings produce similar numbers.

Instead of finding a specific row, you find rows closer in meaning, and “close” requires measuring distance in high-dimensional space rather than matching values. Think of it like a regular map with latitude and longitude, except you’re working with 1,536 dimensions instead of just 2.

Let’s walk through a concrete example to see why this creates a problem…

Imagine you have a document database where each document gets split into chunks.

And think of chunks like cutting a book into individual paragraphs so you can search each one separately. Each chunk gets converted into a 1,536-dimensional vector using an embedding model like OpenAI’s text-embedding-3-small:

1. A user submits a search query

2. App converts the query into a vector using the same embedding model

3. Now you have to find the “most similar” document chunks

Without specialized indexing, the only way to find “most similar” ones is to calculate the distance between your query vector and every chunk vector in your database, sort by distance, and return the top results.

Let’s break down what it means…

A vector is simply a list of numbers. In AI, those numbers represent meaning.

When you convert text into an embedding, you’re assigning it a location in a very high-dimensional space (imagine a map with 1,536 dimensions instead of the normal 2, latitude and longitude). Similar pieces of text end up close to each other on this map, while unrelated text ends up far apart.

When a user searches for something, their query also gets converted into coordinates on this map. To find the most relevant results, you need to measure the distance from the query’s location to every document chunk’s location in your database, then return the closest ones.

But here’s the problem…

Without an index, there’s no shortcut. You must measure the distance to each item.

Now imagine you’re standing in Times Square in New York City, and you need to find the 10 closest coffee shops. Without a map or any organized system, your only option is to:

1. Walk to every building

2. Check if it’s a coffee shop

3. Measure how far it is from Times Square

4. Keep a list of the 10 closest ones you’ve found

If there are 10k buildings, you check all 10k. And if there are 10 million buildings, you check all 10 million.

The more buildings that exist, the longer it takes…

Put simply, it takes O(n) time, and it grows linearly with the size of your dataset.

In practical terms, if you have 10,000 document chunks and each distance calculation takes 0.0001 seconds, one query takes 1 second. That’s not terrible.

But what if you have 10 million chunks?

That same query now takes 1,000 seconds, i.e., 16 minutes. And if your application serves 100 queries per second from different users? Your system immediately grinds to a halt!

This is why traditional databases struggle with embeddings…

Their indexes are built for exact matches.

They can jump straight to user_id = 12345 without checking every row. But with vectors, there are no exact matches, only “similar” matches, and similarity requires measuring distances.

Let’s now imagine finding a similar song to the one you already like…

You cannot just look it up by name; you have to compare it against every other song across hundreds of characteristics: tempo, key, energy, mood, and so on. The more songs in your library, the longer it takes.

Vector search works the same way, except instead of songs, you compare text, and instead of a few characteristics, you are comparing across 1,536 dimensions at the same time.

Without specialized techniques to avoid checking everything, the math simply doesn’t scale.

This is where people often bring up pgvector, a PostgreSQL extension that adds vector search capabilities to Postgres. It’s a good option for smaller datasets and lets you avoid adding another database to your stack. This keeps operational efforts very low.

For datasets with fewer than 100,000 vectors and moderate query volumes, pgvector works well and keeps your infrastructure simpler.

But as your dataset grows into the millions and your query latency requirements tighten, you hit the limits of what a general-purpose database can do for this specific workload.

You don’t need a specialized vector database just because you’re working with embeddings, but you need to understand when the operational trade-offs tip in favor of dedicated infrastructure.

This brings us to the core challenge that vector databases solve…

How do you find approximate nearest neighbors in high-dimensional space without comparing against every vector, while still returning results that are “good enough” in terms of accuracy?

The answer involves building specialized index structures that trade accuracy for massive speed improvements, and understanding those trade-offs is essential for operating these systems in production.

Architecture: How It Works Under the Hood

Vector databases have three key layers: storage, index, and query.

Each layer involves trade-offs that directly affect your operational concerns: cost, latency, accuracy, and scalability.

Let’s break down what matters from an infrastructure perspective…

Storage Layer: Where Vectors Live

Vectors are massive.

A single 1,536-dimensional vector using 32-bit floats takes up 6KB of storage. If you multiply that across millions of vectors, storage costs add up fast. This is a problem every vector database must solve at its foundation: the storage layer determines how vectors are physically stored, compressed, and retrieved from disk. This layer focuses on compression techniques that reduce both storage costs and the memory footprint while preserving as much accuracy as possible.

The most common approach is quantization.

Quantization reduces the precision of each dimension to use fewer bits.

Instead of storing each dimension as a 32-bit float, you quantize to 8-bit integers (scalar quantization) or binary representations (binary quantization).

This can reduce storage and memory requirements by 4x or more, but it introduces approximation error. Approximation error is the gap between the compressed version of data and the original.

When you round numbers to save space, your calculations become slightly less precise. In vector search, this means you might occasionally miss a result that was truly the closest match, but in practice, the difference is small enough that most applications never notice it.

In production, you’re constantly trading off storage costs for search accuracy, and the right balance depends on your use case.

Another critical decision is the trade-off between memory and disk…

For maximum query speed, you want your vector index in memory, but that’s expensive at scale. Some vector databases support tiered storage, where frequently accessed vectors stay in memory while cold data lives on disk, or hybrid approaches in which the index structure stays in memory while raw vectors get pulled from disk when needed.

These architectural choices directly affect your cost structure and query latency percentiles.

From an operational perspective, you need to understand how your chosen vector database handles durability and recovery:

• What happens if a node crashes mid-write?

• How long does it take to rebuild indexes after a restart?

• Are vectors written to disk synchronously or asynchronously?

These aren’t theoretical questions. They’re the difference between acceptable downtime and a major incident when something goes wrong.

Indexing Layer: Graph Structure

This is where the “magic” happens, and by magic I mean carefully engineered data structures that let you find approximate neighbors without exhaustive search.

The most popular algorithm in production vector databases is HNSW (Hierarchical Navigable Small World), and it’s worth understanding at a conceptual level even if you’re not implementing it yourself.

The core idea behind HNSW is based on “small world” networks, the same principle that says you’re only a few connections away from any other person on the planet.

Instead of comparing your query vector against every vector in your database, HNSW builds a multi-layer graph in which similar vectors are connected to each other. When you query, the algorithm starts at the top layer and navigates toward your target by following edges to similar neighbors, dropping through layers until it reaches the most similar vectors at the bottom layer.

Remember our map analogy?

Your vectors are locations on a high-dimensional map, and you’re trying to find the closest ones to your query without checking every single location. HNSW solves this by building a navigation system with different zoom levels.

Think of it like Google Maps navigating from New York to a specific coffee shop in San Francisco:

• Top layer (zoomed out): From New York, HNSW identifies the general direction of your target, the West Coast, and follows connections to get you to California. You’re not considering every city in America, just the major waypoints that get you closer to where you need to be.

• Middle layers (zooming in): Now you’re looking at California. The algorithm narrows down to the San Francisco Bay Area, following connections between nearby locations. You’re still not checking every street, just the neighborhoods that are directionally correct toward the coffee shop you want to visit.

• Bottom layer (street level): Finally, you’re at the specific neighborhood and can see individual addresses. The algorithm follows the last few connections to find the exact coffee shop location (or, in vector terms, the most similar document chunks).

At each zoom level, you’re following connections between “nearby” locations without checking everywhere.

You only need to follow the connections that get you closer to your target. This is how HNSW turns an O(n) problem into something much faster: it builds a navigation structure that lets you skip large portions of the map in the wrong direction.

The key trade-off in HNSW is speed versus accuracy, controlled by two parameters: the number of connections each vector has (called “M” or “max connections”) and the number of neighbors to check during search (called “ef” or “exploration factor”).

Higher values mean better accuracy, but slower queries and larger indexes…

Lower values mean faster queries, but you might miss some relevant results. In production, you tune these parameters based on your latency requirements and tolerance for accuracy.

Building the HNSW index is expensive.

Inserting vectors requires calculating distances and updating graph connections, which is why bulk index building often happens offline or during low-traffic periods. Index build time matters because it affects how quickly you can recover from failures, deploy updates, or onboard new data.

You do not need to understand the math behind HNSW to work with vector databases.

Just like driving a car, you might not know how the engine works, and that doesn’t affect how you drive. But knowing that the engine exists and what it does helps you figure out what is wrong when the car breaks down.

What you need to know is that HNSW produces approximate results, not perfect ones, and it’s possible to adjust its approximation. When search results seem wrong, queries suddenly slow down, or the index fails to build, this understanding helps you identify the problem.

Get a copy of LLMs for Humans: From Prompts to Production right now and save 30%.

Reminder: this is a teaser of the subscriber-only newsletter, exclusive to my golden members.

When you upgrade, you’ll get:

• High-level architecture of real-world systems.

• Deep dive into how popular real-world systems work.

• How real-world systems handle scale, reliability, and performance.

And much more!

Unlock Full Access

Query Layer: How Requests Move Through System

When a query hits your vector database, it follows a specific path:

Read more

• Share this post & I'll send you some rewards for the referrals.

When you build software with LLMs1, you need to decide how much control to give the model.

Does your code run every step, or does the model figure out the steps on its own?

Agentic patterns are design patterns for making that decision. They’re the same architecture choices you already make in any system: who controls what happens next, what happens on failure, and how data moves between components.

The difference is that some of those components are now language models.

This newsletter covers nine of those patterns…

On one end, workflow patterns where your code controls every step. On the other hand, agent patterns where the Large Language Model (LLM) decides what to do next.

That boundary is a decision about how much control to hand to the model: at some point, you stop telling the model what to do and start letting it figure that out.

The first question isn’t which pattern to use. It’s whether you need one at all.

Let’s start there...

Find out why enterprise leaders are doubling down on data foundations for AI (Partner)

AI is moving fast, but your data foundation isn’t keeping up?

That’s exactly why leaders from JPMorgan, Mercedes-Benz, Siemens, and Roche contributed to this AWS book on agentic analytics.

Here’s what you get:

• Real strategies from enterprise leaders: Learn how top companies are building data foundations for AI systems that actually scale.

• Practical frameworks you can apply: From data strategy to data products, ML, and agentic AI.

• Perspectives from 15+ enterprise leaders: Each chapter brings a unique view from senior leaders across industries.

• Complex topics broken down clearly: Understand what matters without getting lost in theory.

An AWS book designed to help you build data foundations for intelligent agents that scale.

Download it now and learn how to build systems ready for intelligent agents.

Get the AWS Data & AI Book

(Thanks to AWS for partnering on this post.)

Inside this newsletter, you’ll get:

• The shift from prompts to systems. Why simple LLM calls break down, and how workflows and agents introduce structured decision-making.

• A practical escalation ladder. When to stay simple, when to use workflows, and when agents are actually justified.

• Workflow patterns. chaining, routing, parallelization, and orchestrator-workers, with clear tradeoffs in cost, latency, and failure modes.

• Agent patterns. Reflection, tool use, ReAct, and planning, focusing on how the model makes decisions and where things go wrong.

• Evaluator-optimizer loop. How to add a reliability layer with evaluation, iteration limits, and cost control.

• Real-world case study. Combining multiple patterns in an AI code review pipeline, showing how to choose the simplest architecture that works.

Not everything needs an agent

Most tasks don’t need an agent2…

Most tasks don’t even need a workflow…The default should be the simplest setup that works, and you escalate only when that setup breaks.

Start with a direct API call to an LLM. You can do more with this than most people think:

• Summarization

• Classification

• Extraction

• Rewriting

• Translation

• Code generation with clear specs

Most people skip this level too quickly!

Workflow patterns are the next step up.

Escalate when a task has many steps, and when focused attention at each step improves the output. Between those steps, you define validation gates: checks that verify the output before passing it forward. The defining feature: you can write every step down before the system runs. Your code owns the control flow.

And the LLM handles the details.

Agent patterns are for when the number and type of steps are unknown until the system runs.

The model needs to look at results, decide what to do next, and choose its own path. For example, say you built a support agent that handles order cancellations. A customer asks to cancel, but the agent doesn’t know upfront whether it needs to check the refund policy, look up the shipping status, or pass the issue to a human.

The steps depend on what it finds…

Here’s how you know it’s time to switch: you write more code handling errors and exceptions than doing the actual work. Or you keep adding special cases for situations the LLM encounters you didn’t plan for.

If you can still write down all the steps before the system runs, stick with a workflow.

Agent patterns sit within multi-agent architectures, i.e., many agents cooperate on the same problem, each with its own role.

One common mistake: you get your system to 70-80% of a prototype and assume the architecture needs upgrading. It usually doesn’t…The real issue is usually prompt quality or missing validation gates. Move to a more complex pattern only after the simpler one has actually failed, not when it feels limiting.

With that baseline set, here are the four workflow patterns:

Workflow patterns: you control the flow

With workflow patterns, your code controls what happens.

You decide the steps, the order, and the checks between them. The LLM handles each step, but your code is in charge…

1. Prompt chaining

Prompt chaining breaks a task into a series of LLM calls.

Each call takes the output from the previous step, and a check runs between them to ensure the result is correct before moving on.

If you’ve ever set up a CI/CD (Continuous Integration/Continuous Deployment) pipeline, this is the same idea. A build pipeline compiles, lints, tests, and packages. Each step must pass before the next one runs.

If tests fail, the pipeline stops.

For example, in legal contract review, this looks like: extract all clauses from a contract, classify each clause by risk level, then generate a summary of high-risk items for human review. Products like Thomson Reuters CoCounsel and Robin AI run variations of this pipeline.

Each step gets a focused prompt with a narrow task, and a validation gate checks the output before passing it forward.

Tradeoffs

Latency grows linearly…

Five steps mean five round-trip. Errors carry forward: if step two misclassifies a clause, steps three through five process bad input. The gates between steps catch these failures early, but they only catch what the conditions you write can catch.

2. Routing

A routing pattern classifies the input and sends it to the right handler: a different prompt, a different model, or a different sub-workflow.

Each handler is built for one type of input.

This is like a hospital front desk. A patient walks in, the front desk checks their symptoms, and then sends them to the right doctor. The front desk doesn’t treat anyone. And the doctor doesn’t check in patients.

Sierra AI routes across 15+ models this way.

Classification tasks go to one model, tool calling to another, and response generation to a third. Customer support platforms like Intercom Fin route each customer request based on what the customer wants and how they feel about it, sending requests to either an AI handler or a human agent.

The cost argument is built into the pattern…

Simple queries hit cheap, fast models. While complex ones hit expensive, capable ones. You’re not paying the highest prices for work that a smaller model handles well.

Tradeoffs

In 1 word: misclassification.

A complex query routed to the cheaper model produces a poor answer. A simple query routed to the expensive model is just a waste of money.

Either way, the router is a single point of failure, and its accuracy sets the upper limit for the entire system.

3. Parallelization

In software, parallelization means running many tasks simultaneously rather than one after another. With LLMs, it has two variants that solve different problems…

Sectioning splits a task into independent parts, runs them in parallel, and merges the results. This is like a construction crew. Electricians, plumbers, and carpenters work in the same building at the same time. Their work gets combined at the end.

GitHub Advanced Security works this way: CodeQL and third-party tools like Snyk and Semgrep scan the same pull request in parallel, each producing independent security alerts.

Voting runs the same task many times with different prompts and combines the answers. This is like a jury. Each juror looks at the same evidence and gives their own opinion, and the group decides together.

You’d build this for something like a security review: three separate prompts analyze the same code change for vulnerabilities, and you flag an issue only if two or more agree. Running the task many times is more expensive, but it catches fewer false alarms.

In short, sectioning gives each worker a different job; voting gives every worker the same job.

Tradeoffs

Cost multiplies with every parallel branch.

Partial failures are a design decision you need to plan for up front: if one branch fails, do you retry it, proceed with the others, or fail the whole operation? There’s no universal answer, and a wrong choice is expensive.

4. Orchestrator-workers

The orchestrator-workers pattern uses one central LLM to break a task into subtasks, assign each subtask to a worker LLM, and combine the results.

The difference from the earlier patterns is that the subtasks aren’t known in advance. Orchestrator figures them out as it goes. This is like a general contractor building a house:

• They hire the right people, but don’t know every subcontractor they’ll need upfront

• Plumber finds a structural problem, so the contractor brings in a structural engineer

• The work plan changes as the project reveals new problems

Cursor’s agent mode works this way. It starts with different agents for different parts of the codebase:

• One adding tests

• Another updating documentation

• A third refactoring shared utilities

You’re trusting the LLM to figure out what work needs to be done, not just to do the work you’ve assigned.

This uses several LLMs, but it’s NOT a multi-agent architecture: a single central LLM remains in control of the entire process. In multi-agent systems, agents can call each other directly without a central coordinator, and control can transfer between them.

Tradeoffs

The orchestrator can lose track of the original goal.

It breaks simple tasks into too many subtasks. And when workers finish faster than it can combine results, the orchestrator itself becomes the bottleneck.

These four patterns get gradually more powerful and more expensive…

Chaining and routing are cheap and predictable: you know what will happen before it runs. Parallelization saves time but costs more. Orchestrator-workers handle problems you can’t predict upfront, but it introduces failure modes you can’t predict either. The first three patterns are simple to understand.

Most production systems shouldn’t need to go further than parallelization. When they do, the next four patterns hand control to the model itself...

Reminder: this is a teaser of the subscriber-only newsletter, exclusive to my golden members.

When you upgrade, you’ll get:

• High-level architecture of real-world systems.

• Deep dive into how popular real-world systems work.

• How real-world systems handle scale, reliability, and performance.

Unlock Full Access

Agent patterns: LLM controls the flow

With agent patterns, you STOP defining what happens and in what order.

Instead of defining steps, you define constraints: what tools are available, how much the model can spend, and when to stop. The model observes, reasons, and chooses what happens next.

That trust decision appears differently in each pattern: how much autonomy, over what actions, with what limits.

5. Reflection

Reflection is when the LLM generates output, reviews its own work, and then fixes the problems it found.

Read more

• Share this post & I'll send you some rewards for the referrals.

Most engineering resume advice you come across is junk.

At best, it comes from a tech recruiter who’s looked at 100s of resumes. At worst, it’s from yet another senior engineer cobbling together a blog post based on their experience applying to MAANGO.

This post is not like that.

I’ve made thousands of hiring and leveling decisions as mobile Hiring Committee Chair at Meta. I also led Meta’s engineering team responsible for finding and ingesting over 2M engineers’ resumes into their internal ATS (Applicant Tracking System).

I’ve seen a lot of resumes :)

And because of that, I know that you should treat your resume like a product. Think of it as a landing page for your career, designed to drive each target user who touches it to take action. And yes, your resume has multiple target users—four, in fact—and you should learn what each one is looking for.

(Oh, and btw, a lot of this applies to your LinkedIn profile too, since that is literally a landing page.)

Onward.

The tax you pay to run multiple agents (Partner)

If you’ve spent any time with coding agents, you know the feeling.

You start the morning with a clean plan. Spin up a few agents. One is refactoring the auth module. Another is writing tests. A third is scaffolding a new API endpoint. You’re flying.

Then, around 10:30 AM, you look up and realize you have 20 terminal windows open. One agent is blocked waiting for a decision you forgot to make. Another finished 40 minutes ago, and you never noticed. A third went sideways three commits back. You’re no longer flying. You’re drowning.

You’ve shifted from human as driver to human as director.

When running coding agents in parallel, the bottleneck isn’t just context. It’s your own attention trying to manage 10 agents across 10 terminals. You’re losing your mind to terminal chaos.

Meet Cline Kanban, a CLI-agnostic visual orchestration layer that makes multi-agent workflows usable across providers. Multiple agents, one UI. It’s the air traffic controller for the agents you’re already running, regardless of where they live.

• Interoperable: Claude Code and Codex compatible, with more coming soon.

• Full Visibility: Confidently run multiple agents and work through your backlog faster.

• Smart Triage: See which agents are blocked or in review and jump in to unblock them.

• Chain Tasks: Set dependencies so Agent B won’t start until Agent A is complete.

• Familiar UI: Everything in a single Kanban view.

Stop tracking agents and start directing them. Get a meaningful edge with the beta release.

Install Cline Kanban Today: npm i -g cline

Get Started Today

(Thanks to Cline for partnering on this post.)

I want to introduce you to Austen McDonald as a guest author.

Austen is a former Senior Engineering Manager and Hiring Committee Chair at Meta, where he spent almost 10 years and conducted over 1,000+ interviews. He’s the industry’s resident expert on behavioral interviews and writes about them on his Substack and in his book, Mastering Behavioral Interviews. As AI raises the bar for ownership and independent judgment, behavioral interviews matter more than ever.

For a few days only, use code SDN26 for 20% off Premium Access, which includes a print copy of his book and an AI copilot that helps you find your best career stories, shape them into high-signal answers, and deliver them well.

Part 1: Your Resume is a Product

Your resume is a landing page for your career search.

It’s the first thing these four personas see when they consider you for an engineering job:

• Machines (Applicant Tracking System),

• Recruiters (we’ll include sourcers for all you sticklers),

• Interviewers,

• And Hiring Managers.