• Share this letter & I’ll send you some rewards for the referrals.

Traditional software gets cheaper per user as it scales.

Yet AI software on an API does not!…

Every request runs a model, every model call has a per-token price, and the price stays “flat” whether ten people use it or a million do.

Self-hosting changes this, as GPU costs amortize across users.

But it introduces a new problem: capacity planning, hardware management, and a minimum spend that has to be justified before the first user arrives.

So scaling an AI product forces cost decisions most software never encounters, and most engineering teams face them only after users have already arrived.

Building an AI system that produces correct, grounded, and safe responses is one challenge…Serving those responses reliably and affordably to a million users is another.

This newsletter shows you how to redesign a chat assistant for large-scale production, reduce inference costs & build the infrastructure needed to handle millions of users.

§

Make your agents realtime (Partner)

Agents are killing your apps with compounding latency across workflows.

Your customers want their agent workflows to be efficient, so you’re losing deals if you don’t make them event-driven. Webhooks solve this issue, but implementing them is harder than it looks.

If you homebrewed your webhooks and are getting pinged at 4 in the morning for failed deliveries, this is for you.

Svix handles everything: retries, idempotency, security, and compliance. Qualified startups get $12,000 in free credits.

Get started for free

(Thanks to Svix for partnering on this newsletter.)

§

I want to introduce Louis-François Bouchard as a guest author.

He’s a best-selling author (Building LLMs for Production), the co-founder of Towards AI, and the creator of the YouTube Channel, What’s AI, where he helps people understand AI and learn how to apply it in the real world. Through his development work with clients, content, teaching, AI training programs, and the new Mentorship program at Towards AI Academy, Louis focuses on making AI practical for builders, engineers, and curious learners alike.

At Towards AI, he and his team train AI engineers through courses and mentorship built for every stage, from beginner to advanced. That educational mission and the real-world experience building for his clients are exactly why I wanted him in this newsletter series.

§

Here’s what’s inside this newsletter:

• Your $193,000 AI bill could drop to $48,000. Three optimizations cut the projected monthly cost by 75% at 100,000 daily conversations. Most teams overlook where the biggest savings actually come from.

• Your smartest AI model could be your biggest money leak. Up to 70% of requests in our example go to a cheaper model, while the expensive one handles the rest. The surprising part is what happens to the total bill.

• At 100K users, renting AI could beat owning GPUs. A $48,000 API bill competes with a $49,560 self-hosted setup. At 1 million daily conversations, the economics flip dramatically.

• One GPU benchmark could wipe out $182,000 in monthly savings. Self-hosting looks 38% cheaper at scale, until lower throughput forces you to rent dozens of additional GPUs.

• Your AI could pass every test and still break in production. The hidden gaps between evaluations, guardrails, and real user behavior, and why production failures need to become tests before the next release.

§

Should You Plan for a Million Users?

For most AI products, planning for a million users too early wastes time & money.

An API key and a credit card are often enough to launch a prototype. You should focus on building a useful product before investing in complex infrastructure.

But as your user base grows, these challenges become HARD to ignore:

• Cost increases with usage. A chat assistant costs approximately $0.064 per conversation using GPT-5.4.

  • At 100 daily users, the monthly model bill reaches roughly $193.

  • At 100,000 daily users, it grows to $193,000.

  • At one million, it reaches $1.93 million.

  Model routing & caching reduce the cost per conversation, but total spending still grows with usage.

• Latency increases under heavy traffic. An AI system that responds in 400 milliseconds with ten concurrent users could take two seconds with a thousand. As incoming requests approach the system’s serving capacity, queues grow & response times increase.

• Small prompt changes become expensive. Adding 200 tokens to your system prompt across one million daily requests produces six billion additional input tokens per month. At GPT-5.4’s input price of $2.50 per million tokens, that adds $15,000 to your monthly bill before prompt caching.

• API rate limits restrict scalability. AI providers enforce limits on requests & tokens per minute. As traffic grows, these limits could prevent your application from serving more users, even when your infrastructure has spare capacity.

• Unit economics determine profitability. If each conversation costs more than the revenue it generates, acquiring more users increases your losses. You need to measure cost per conversation against the revenue it produces.

These challenges determine when your AI application needs more advanced production infrastructure.

So before moving beyond a managed API, evaluate these four constraints:

• Unit economics: Does the cost per successful conversation support your target profit margin?

• Service level objectives (SLOs): Does the system meet your latency, availability, and reliability targets?

• Rate limits: Does your provider support your expected request and token volume?

• Data residency: Does the provider meet your requirements for where user data gets processed & stored?

As long as your API provider meets these requirements, staying with managed infrastructure is often the “simplest” choice.

The rest of this newsletter shows you how to estimate costs, identify scalability bottlenecks, and decide when more advanced infrastructure becomes necessary.

Share

§

Target System Design

Our goal is to scale an AI chat assistant to one million daily users while keeping inference costs under control.

For simplicity, we assume each daily active user starts one conversation. Each conversation contains eight requests to the language model.

Let’s define the token usage for each request:

• Static input: 1,024 tokens for the system prompt and a fixed few-shot examples. This meets OpenAI’s minimum prefix length for prompt caching.

• Dynamic input: 400 tokens for conversation history & current user message.

• Output: 300 billed tokens generated by the model.

Each request therefore processes 1,424 input tokens & generates 300 output tokens.

With eight requests per conversation, the total becomes:

• Input: 11,392 tokens per conversation.

• Output: 2,400 tokens per conversation.

(These are illustrative assumptions. In production, you should replace them with actual token usage collected from your application.)

System Requirements

Our production system must meet seven requirements:

• Cost: Keep model costs below a budget of $0.02 per conversation. Determine your actual budget from revenue per successful conversation and required gross margin.

• Latency: Define service level objectives (SLOs) for each workload, including time to first token (TTFT) for interactive chat & completion deadlines for asynchronous jobs.

• Availability: Maintain 99.9% uptime, including during model & prompt updates.

• Safe deployments: Support model changes with a reliable rollback mechanism.

• Observability: Record token usage, cost, latency & model information for each request.

• Feedback: Collect user feedback & incorporate reviewed examples into evaluation datasets.

• Cost forecasting: Estimate the financial impact of every change before deployment.

Our Approach

We’ll make five major design decisions:

1. Cost engineering: Reduce model costs by 50% to 80% through model routing, semantic caching, and prompt optimization.

2. Inference optimization: Explore quantization, GPU utilization, and serving frameworks to determine when self-hosting makes financial and technical sense.

3. Production infrastructure: Handle concurrent traffic, control overload, and deploy model updates without disrupting users.

4. Data flywheels: Use production feedback to improve evaluation datasets & model quality over time.

5. Capacity and cost estimation: Calculate infrastructure requirements & monthly costs at 10,000, 100,000, and one million daily users.

We’ll start with cost engineering because the budget per request influences every infrastructure decision that follows…

§

Part 1: Cost Engineering

Let’s dive in!

Three Types of Token Costs

Every request to a language model creates costs based on token usage.

For this example, we’ll focus on three types:

• Uncached input tokens: Tokens you send to the model, including the system prompt, conversation history, and user message, without a cached-input discount.

• Cached input tokens: Previously processed input prefixes that the provider reuses at a discounted rate.

• Output tokens: Tokens the model generates, including any billable reasoning tokens.

Output tokens typically cost more than input tokens because generating tokens one at a time (decode) has lower throughput than processing input tokens in parallel (prefill).

Some providers charge separately for extra features:

• Cache writes: Anthropic charges approximately 1.25 to 2 times the standard input rate to create a cache entry, depending on its duration. Cache reads cost approximately 10% of the standard input rate.

• Tool usage: Certain tools or external services introduce extra charges.

• Priority processing: Higher-priority inference tiers could have different pricing.

• Data residency: Regional processing requirements could affect the final bill.

So check your provider’s billing structure before estimating production costs.

For the two OpenAI models below, output tokens cost six times as much as uncached input tokens.

These prices are a historical snapshot… Model availability & pricing change over time, so verify the latest rates before making deployment decisions.

Next, let’s use these token prices to calculate the monthly cost of serving thousands/millions of conversations.

Estimation Funnel

Estimate costs before building the system.

Start with requests per day, estimate input, output, and cached-input tokens per request, then apply their respective rates. Multiply the daily cost by the number of days to get the monthly bill.

The arithmetic is straightforward…The challenge is accounting for every cost.

Retries & failed calls increase usage.

Agent loops multiply model calls per task. Embeddings and reranking add separate costs. Image and video processing introduce media-specific charges.

Semantic caching adds infrastructure costs, while provider-side prompt caching may introduce cache-write charges. Storage, network egress, observability, and idle GPU capacity when self-hosting also contribute to the total.

Even a complete cost estimate does not tell you whether the system produces useful answers. Track cost per successful task alongside total spending to measure efficiency.

Cost per Successful Task

Cost per successful task measures total spending divided by the number of tasks completed successfully.

Track it alongside quality and latency to understand the actual cost of delivering useful results. For example, suppose each conversation costs $0.064 on average. If 10% of conversations fail, the cost per successful conversation increases to approximately $0.071, assuming the average cost already includes failed attempts and retries.

The same principle applies to self-hosted models. Estimate the required token throughput, translate it into GPU capacity & calculate infrastructure costs.

H100 GPU rentals may cost around $2 to $6 per GPU-hour, depending on the provider, region, and rental terms. Newer Blackwell-class accelerators may have higher hourly rental prices.

But GPU rental prices alone do not determine inference cost:

Utilization, batching efficiency, memory requirements & model quality also affect the cost per successful task.

Faster or Cheaper, and When You Get Both

Latency & cost typically involve tradeoffs.

More powerful GPUs and lower batching levels reduce response times but increase cost per request.

Some optimizations improve both.

Smaller models require fewer resources, while shorter prompts reduce input processing. Prompt caching, quantization, and efficient attention kernels also reduce latency and cost, depending on the workload.

Streaming lets users see the response as it’s generated.

This improves perceived responsiveness without necessarily reducing total computation. Measure time to first token (TTFT), total response time & stream reliability separately.

Batching improves GPU utilization by processing multiple requests together.

Yet excessive batching can increase tail latency (p99). So interactive applications need different batching configurations from background workloads.

For tasks that do not require immediate responses, consider a provider’s Batch API. These asynchronous endpoints process requests within a specified window at a lower price.

OpenAI’s Batch API offers a 50% discount on supported workloads with a 24-hour completion window. Other providers have different pricing and processing limits.

Reasoning-Token Line Item

Some reasoning models generate internal reasoning tokens that users never see.

These tokens count toward output usage & get billed at output token rates.

This increases costs when switching models. Different models use different default reasoning-effort settings. If you leave the setting unspecified, a model upgrade could generate more reasoning tokens and increase the bill, even when the visible response stays the same.

Set reasoning effort explicitly where supported.

Track reasoning-token usage and total output costs for each request. Increase reasoning effort only when evaluations show enough improvement in answer quality to justify the extra cost.

The Anchor: 100K Conversations a Day

Picture an AI assistant handling 100,000 conversations per day, with eight model requests per conversation.

Each request uses 1,424 input tokens and 300 billed output tokens.

Over 30 days, the system processes 34.176 billion input tokens and 7.2 billion output tokens.

At GPT-5.4 rates of $2.50 per million input tokens and $15 per million output tokens, the monthly cost is:

• Input: $85,440

• Output: $108,000

• Total: $193,440 or $0.064 per conversation.

Suppose 70% of requests use GPT-5.4-mini, which costs $0.75 per million input tokens and $4.50 per million output tokens.

Assuming identical token usage across both models, routing reduces the monthly bill to approximately $98,635, a 49% reduction.

Plus, adding semantic caching, provider-side prompt caching, and shorter prompts could further reduce costs. Under a scenario with a 30% semantic-cache hit rate and additional prompt optimizations, the monthly bill could approach $48,000.

That represents approximately 75% savings compared with the original system.

Actual savings depend on traffic patterns, cache hit rates, token usage & model quality. Validate these assumptions against production measurements before relying on the estimate.

Model Routing

Model routing sends simpler requests to a cheaper model & complex requests to a more capable model.

A lightweight classifier or a smaller language model could decide which model should handle each request.

At scale, this significantly reduces inference costs. For example, routing 70% of requests to a cheaper model could lower the monthly bill. But the actual split depends on your workload and should come from production traffic & evaluation results.

The biggest risk is routing a complex request to a model that cannot handle it reliably. The smaller model may produce a confident but incorrect answer, reducing quality without an obvious failure.

To prevent this, evaluate the route system against the original system using a representative test dataset. Measure task success, factual accuracy, relevance, and other application-specific metrics.

The router should reduce costs while keeping quality within predefined acceptance thresholds. Track routing accuracy and cost per successful task to verify that the savings are worthwhile.

Semantic Caching

Model routing reduces costs by sending simpler requests to cheaper models.

Semantic caching goes further by reusing previously generated answers for similar questions, thus avoiding repeated model inference.

Semantic caching uses embeddings to find previously answered questions with similar meanings. If a cached answer passes the similarity and validity checks, the system returns it without calling the language model.

GPTCache is one open-source library for implementing this approach.

Cache effectiveness depends on traffic patterns. Frequently repeated questions, such as FAQs, produce higher hit rates, while unique or context-dependent queries benefit less.

Yet two failure modes require attention:

1. False cache hit

Two questions may have similar embeddings but require different answers.

Reusing an incorrect or outdated response silently reduces answer quality. Similarity thresholds, time-to-live (TTL) policies, and extra validation help reduce this risk.

2. Cross-user data leakage

Cache entries must be isolated by tenant & authorization scope.

Cache validity should account for relevant prompt, model, and retrieval-index versions, along with data freshness requirements. Authorization checks must remain enforced even when a cached answer exists.

Personalized responses should only be cached within properly isolated scopes.

Requests that perform actions or modify data must execute the required operations rather than replay a cached response.

Finally, semantic caching has its own costs, including embedding generation, vector search, and cache storage. So measure cache hit rate, false-hit rate, latency, and net cost savings to determine whether it improves the system.

Prompt Optimization

The system prompt gets included in every model request.

So its length directly affects input-token usage and cost. In our example, the static prefix accounts for nearly three-quarters of each request’s input tokens.

Shortening instructions and removing unnecessary few-shot examples reduce token usage. But the actual savings depend on how much of the prompt already benefits from caching. Plus, removing important instructions reduces answer quality, so evaluate every change before deployment.

Prompt caching introduces another consideration:

Providers reuse previously processed input when requests share an identical token prefix. To maximize cache hits, place reusable instructions and examples first, followed by dynamic content such as user questions and retrieved documents.

For OpenAI’s automatic prompt caching, eligible prompts must contain at least 1,024 tokens. Shortening a reusable prefix below this threshold eliminates its caching benefit and potentially increases costs, even when the prompt becomes shorter.

In our cost example, we retain the 1,024-token static prefix and optimize the dynamic portion instead. This preserves the opportunity for prompt-cache savings while reducing unnecessary input tokens.

The goal is to minimize total input cost without sacrificing response quality.

Compare cached-token usage, total spending, and evaluation results before deciding which prompt version performs best.

When Smaller Models Are Good Enough

Smaller models significantly reduce inference costs, but only if they maintain acceptable answer quality.

Evaluate each candidate model against a representative test dataset that includes common requests, complex queries, and edge cases. Measure task success, factual accuracy, relevance, and other application-specific metrics.

If the smaller model meets predefined quality thresholds, it passes the quality gate. Before deployment, it must also satisfy latency, reliability, security & rollback requirements.

Use evaluation results to decide when a smaller model is sufficient rather than relying on model size or benchmark rankings alone.

Model routing reduces average inference costs by selecting cheaper models, while semantic caching avoids repeated model calls.

The next step is optimizing the inference stack itself to reduce the computational cost of generating each token…

§

§

Part 2: Inference Optimization and Self-Hosting

Read more

• Share this letter & I’ll send you some rewards for the referrals.

Every task you give Claude runs while you sit there & watch it.

But some work does not need your presence at all… You can check your website for availability issues daily, watch a repo for changes, or let Claude send you a weekly report via Slack.

Anthropic created two different features for this:

• Claude Code calls them Routines1,

• Cowork2 calls them Scheduled Tasks.

You could create them from scratch or simply by prompting Claude.

Some of these automations run on your laptop. To make sure they work, your laptop needs to stay awake. But you can also run your code in the cloud on Anthropic’s servers. This works even when your laptop is closed.

Let’s build two of them in this newsletter & explore their limitations.

At the end, we’ll also look at guardrails, because we all know that conversations between us and AI can leak, and we should take steps to make them safer.

Let’s get started!

§

[Webinar] How to stop babysitting your agents (Partner)

Agents can generate code. Getting it right for your system, team conventions, and past decisions is the hard part. You end up wasting time and tokens in correction loops.

More MCPs give agents access to information but not understanding. The teams pulling ahead use a context layer to give agents exactly what they need.

Join live on Oct 7 (FREE) to see:

• Where teams get stuck on the AI maturity curve

• How a context layer solves for quality, efficiency, and cost

• Live demo: the same coding task with and without a context layer

Register now

(Thanks to Unblocked for partnering on this newsletter.)

§

I want to introduce Gencay as the guest author.

He has been building with AI since ChatGPT first shipped in 2022.

For years, he built for clients on Upwork: 7,300+ hours of it.

Now he builds in the open and shares every piece of it on LearnAIWithMe, every Monday, Wednesday & Friday.

Not just theory for its own sake. One real build at a time, from idea to running system.

§

Here’s what you’ll find inside this newsletter:

• What a schedule changes about an agent. Routines in Claude Code, Scheduled Tasks in Cowork, and why a job that runs without you is a different kind of worker than one you watch.

• Where your automation actually runs. Local routines that need your computer awake, cloud routines that keep running with the laptop closed, and the triggers (schedule, Git event, API) and delivery options (push, email, Slack) the cloud version adds.

• Headless mode, the engine underneath. claude -p runs Claude with no chat window at all, and every scheduled job is built on it.

• Two routines you can build, and how to reach them from anywhere. A morning site check and a GitHub watcher, each created from a single prompt, then dispatched from your phone with reports landing in Slack or a custom Telegram bot.

• How to trust a job you were asleep for. Guardrails on permissions, keys, and connectors, a validation harness where tests and hooks check the agent’s work, and reports worth reading.

§

What routines actually are

Everything Claude does normally starts with you!

You type, it works, and when you leave the desk, the work stops…

A schedule breaks this link... The agent stops being an extension of your hands and becomes a worker in your absence, since the job now runs whether you are there/not. And that changes what you use it for, because the tasks worth scheduling are the ones you kept forgetting/postponing.

Routines let Claude work on a schedule you define.

They have two different modes:

• Local

• Cloud

(We’ll cover both in the next sections.)

Let’s first create a routine manually…

Manual

To create a routine, open your Claude app, click on “Routines” and “New routine”.

You will see two options:

Local

Click on Local.

You can choose the model, set permissions & schedule when you want the routine to run.

Cloud

Now let’s check the cloud version.

This version lets you customize a lot more.

First, you can select a trigger: Schedule, Git Event, or API.

Next, click Notifications to choose how the report is delivered to you.

The options are push notifications in the app, email, or Slack.

You can also enable “Auto-fix” for pull requests.

This way Claude watches CI and review comments and pushes fixes automatically.

Create Routine by prompting

Let’s create one routine by prompting.

Here is a site I run… I change it often, so pages break sometimes, and I check them by hand.

Follow along with any site you run, since the prompt only needs a URL.

Let’s hand this task off to Claude Code to save time.

Here is the prompt:

Create a routine that runs every morning at 8.
It opens my AI Labs and AI Academy pages in Chrome,
checks for broken links, layout problems, and console errors.

If everything is fine, report one line saying clean.
Otherwise, list only the real issues to Slack.

After pasting this prompt, Claude asks for my approval.

The first approval is to list the scheduled tasks.

Next, it asks a couple of questions…

After my answers, it started working on it…

Then it asked the final question before creating this routine.

I allowed & it created the routine.

When I went back to the routine tab, I saw mine.

Testing The Routine

I clicked on the routine and “Run now”.

And I saw this notification:

To see more details, you need to click on History. (I clicked it twice by mistake.)

And Claude took control of my laptop & started testing…

When I check Claude app, I see Claude took screenshots & analyzed the pages.

A couple of minutes later, I got this Slack message:

“Clean”, simple yet effective.

Here is a more detailed report:

§

§

Routine vs scheduled

Read more

• Share this letter & I’ll send you some rewards for the referrals.

You ask an AI assistant a question…

…it answers in seconds!

But what actually happened between your question & the answer?

Behind that simple interaction sits a stack of models, tokens, context, retrieval, tools, evaluations, and infrastructure.

These 64 concepts1 will help you understand how modern AI systems actually work, from the neural networks underneath them to agents & applications built on top.

Curious to know how many were new to you:

1. Large language models,

2. Generative AI,

3. Machine learning,

4. Deep learning,

5. Neural networks,

6. Retrieval augmented generation,

7. AI agents & agentic workflows,

8. Prompt engineering,

9. Transformers and self-attention,

10. Embeddings,

11. Tool & function calling,

12. Model evaluation and evaluation-driven development,

13. Context engineering,

14. Multimodal AI,

15. Fine tuning and instruction tuning,

16. Tokenization,

17. Reasoning & chain-of-thought.

( …and much more in Parts 2, 3 & 4!)

For each, I’ll share:

• What it means and how it works,

• A real-world analogy where useful,

• A common mistake to avoid,

• Why it matters.

Let’s dive in!

§

Make your agents realtime (Partner)

Agents are killing your apps with compounding latency across workflows.

Your customers want their agent workflows to be efficient, so you’re losing deals if you don’t make them event-driven. Webhooks solve this issue, but implementing them is harder than it looks.

If you homebrewed your webhooks and are getting pinged at 4 in the morning for failed deliveries, this is for you.

Svix handles everything: retries, idempotency, security, and compliance. Qualified startups get $12,000 in free credits.

Get started for free

(Thanks to Svix for partnering on this newsletter.)

§

1. Large Language Models

A large language model (LLM) is an AI model trained on vast amounts of text to learn language patterns.

You type “Why is the sky blue?” into a chatbot & hit Enter.

The model processes your question as small pieces of text called tokens. A token could be a word, part of a word, or punctuation. A typical text-generating LLM selects the next token based on your question and the text it has produced so far.

Then it repeats. Token by token, the answer appears…

During training, the model adjusted internal numerical values called weights. Those weights capture patterns useful for language tasks, from sentence structure to relationships between ideas.

Your prompt gives the model context for its response.

Analogy

Think of autocomplete on your phone:

You type “See you,” and the keyboard suggests “soon.” An LLM takes this idea much further, using the available conversation to produce paragraphs, explanations, or code.

Predicting a plausible continuation still leaves room for a wrong answer.

Common mistake

Trusting an answer simply because it sounds confident.

Suppose you ask for code to resize an image. The model could suggest a function with a sensible name that the library doesn’t provide. Check the documentation & run the code before you rely on it.

Why it matters

You don’t need a separate model for every language task.

Different instructions let the same LLM summarize a document, explain an error, or draft a reply.

That flexibility makes LLMs useful across many software products.

2. Generative AI

Generative AI produces content such as text, images, audio & video from patterns learned during training.

Ask an image generator for “a red bicycle outside a bakery.”

Your description guides the model toward a result. Some image models begin with random noise and refine it step by step into an image. A text model uses a different process, typically selecting tokens in sequence.

Both produce content from learned patterns.

Generative AI is the broader category… Text-generating LLMs are one example; image and audio generators are others.

Analogy

Think of a cook who has learned from hundreds of recipes:

You ask for a spicy vegetable dish. The cook draws on familiar ingredients and techniques to prepare something that fits your request.

You still taste the dish before you serve it.

Common mistake

Assuming the generated content is ready to use.

An image could contain unreadable labels. A summary could leave out a key decision. Review the result against your original request and check factual details against the source.

Why it matters

Generative AI helps you get from a blank page to a first draft.

A few notes become an email, a description becomes an illustration, or a specification becomes starter code.

The next step is to check whether the draft does the job.

3. Machine Learning

Machine Learning (ML) is a branch of AI in which systems learn from data to perform tasks without explicit rules for every situation.

Think about the spam folder in your inbox:

You could write a rule to block every email containing “free money.” But a spammer could change the wording; also, a legitimate email could contain the same phrase. A model learns from examples & considers several clues together.

Those clues help it assess a new message.

For a spam filter, training examples typically include the correct label: spam or legitimate. The training algorithm adjusts the model to reduce classification errors.

This approach is called supervised learning.

Other approaches discover patterns in unlabeled data or learn through rewards after actions.

Analogy

Think about how you learn to choose ripe fruit:

Someone shows you a few good examples & a few bad ones. Over time, you connect color, firmness, and smell with ripeness.

You use those clues on your next trip to the market.

Common mistake

Testing the model with the same examples it learned from.

That’s like giving a student the practice questions again and calling it a final exam. Reserve separate examples to check whether the model handles unfamiliar data.

Why it matters

ML helps when a task involves more patterns than you could reasonably capture in hand-written rules. It powers uses such as spam filters, fraud detection, and product recommendations.

The goal is useful predictions on new data.

§

§

4. Deep Learning

Deep learning (DL2) is a type of ML that uses multiple layers of a neural network to learn complex patterns.

Show a model a photograph of a bicycle.

The model receives numbers representing the image’s pixels. Early layers often detect simple features, such as edges & textures. Later layers combine these features into more complex shapes & object parts.

Together, the layers help the model recognize the bicycle.

The word “deep” refers to these multiple layers. During training, the model adjusts them to improve its predictions.

Analogy

Think of a sketch as someone draws it:

First, you see a few lines and circles. Add a frame, handlebars, and pedals, and the shapes start to make sense together.

A deep network also combines simpler features into more complex patterns.

Common mistake

Assuming every problem needs deep learning.

For a small spreadsheet of house sizes and prices, a simpler model could do the job well. Test a basic approach first, then compare whether a deep network earns its extra cost and complexity.

Why it matters

Images, speech & language contain patterns which are difficult to describe with fixed rules. Deep learning helps models learn useful features from these inputs.

It underpins tools such as speech recognition & image classification.

5. Neural Networks

A neural network3 is a model made of connected mathematical units that transform inputs into an output.

Let’s look inside the layers we just discussed:

Each unit receives numbers, combines them, and passes a result onward. Connections have adjustable values called weights, which control how much each input contributes. Layers also apply transformations, so the network learns more complex relationships.

The final layer produces the model’s output.

During training, an error measure tells the system how far its predictions are from the target. The training algorithm uses this feedback to adjust weights and other values across the network.

Tiny adjustments, repeated across many examples, change its predictions.

Analogy

Think of the controls on a sound mixing desk:

One control increases the voice; another reduces the drums. Changing the controls changes the final sound.

Weights play a similar role in how a network combines its inputs.

Common mistake

Taking the word “neural” too literally.

The name draws inspiration from the brain, but these units perform mathematical operations. A network’s ability to recognize a face or produce a sentence doesn’t establish human-like thought.

Why it matters

Neural networks provide the structure behind deep learning.

Understanding weights also explains what training changes: numerical values a model uses to turn an input into a prediction.

6. Retrieval-Augmented Generation

RAG is an approach that retrieves relevant information & supplies it to a model to help generate an answer.

Ask a company assistant, “How long do I have to return this order?”

The application searches the company’s documents for the relevant return policy. It places matching passages alongside your question. The model uses those passages to write its response.

The answer has information from the company’s documents.

Updating those documents gives the application new material to retrieve. It doesn’t require a new training run for every policy change.

Analogy

Think of an open-book exam:

You find the relevant pages before you answer. Your result depends on both finding the right material & interpreting it correctly.

Common mistake

Assuming retrieval guarantees a correct answer.

The search could return an outdated policy and/or miss an exception. So check the retrieved passages & final response. When the evidence is missing, the assistant should say so.

Why it matters

RAG helps assistants answer questions about private or frequently updated information. Also it gives readers source material to check against the answer.

7. AI Agents and Agentic Workflows

An AI agent uses a model to choose actions toward a goal and respond to the results. Agentic workflows organize model and tool steps into a process.

Ask an assistant to investigate a failed software test.

An agent would inspect the error, open a file, make an edit, and rerun the tests. The result helps it decide what to do next. A fixed workflow follows a prewritten sequence, such as “summarize the ticket, classify it, then route it.”

The difference is how much control the model has over the next step.

The term “agentic workflow” covers a range of designs. Some follow mostly fixed steps; others give the model more freedom to choose.

Analogy

Think of a technician with a checklist:

For a routine inspection, the checklist determines the order. For an unfamiliar fault, the technician chooses the next check based on what the previous one revealed.

Common mistake

Giving an agent tools without clear limits.

A failed action could trigger repeated retries/unwanted changes. So set permissions, spending limits, and stopping conditions. Plus, require approval before consequential actions, such as a payment/deletion.

Why it matters

Agents help with tasks whose steps depend on discoveries along the way.

Fixed workflows work when the sequence is known and predictable execution matters.

§

§

8. Prompt Engineering

Read more

Download the “Agentic AI” e-book for free

AWS isn’t winning the AI race with the smartest model. It’s winning by owning everything that surrounds the model. Six layers. Sixty-plus services.

But how do you actually put all those pieces together?

AWS asked 15+ data and AI leaders from companies including PayPal, Siemens, Morgan Stanley, and Sanofi how they are approaching agentic AI in the enterprise.

The result is The Agentic Enterprise: a 16-chapter e-book covering practical topics such as:

• Building, monitoring, and governing reliable AI agents

• Improving agent performance and scalability

• Implementing guardrails throughout the AI lifecycle

• Moving agentic systems toward production

If you’re working on AI agents beyond the prototype stage, it’s worth a read.

Download the e-book for free here »

(Thanks to AWS for partnering on this newsletter.)

Imagine you’re building Shazam1.

A user hums a 10-second song clip…Your system has 100 million songs.

Your job is simple: find the most similar songs in less than 50 milliseconds.

So how’d you do it?

Here’s the obvious solution: compare the user’s song with each song in the database.

Query song
   ↓
Song 1
Song 2
Song 3
...
Song 100,000,000

Yet this does NOT work at scale.

Even if each comparison is fast, 100 million comparisons are “expensive”.

When you search documents in a Retrieval-Augmented Generation (RAG2) system, find similar images, or generate recommendations, the system is doing the same thing:

Find the most similar items in a massive collection.

This is where Hierarchical Navigable Small World (HNSW3) comes in…

§

• Share this letter & I’ll send you some rewards for the referrals.

§

I want to introduce Avani Chaskar as the guest author.

She writes technical guides on building, evaluating & scaling deterministic AI systems.

Subscribe to her newsletter to join developers receiving her latest engineering breakdowns.

§

The Idea: Search Only What Matters

The idea behind HNSW is simple:

Instead of comparing against every item, build shortcuts that lead you close to the answer.

Think about finding a coffee shop in a city you do not know.

You would not walk down every street…Instead, you’d:

1. Use highways to reach the right area.

2. Take the main road to get closer.

3. Then use local streets to find the exact place.

§

First Attempt: Build a Graph

Let’s go back to our 100 million songs.

Suppose each song is connected to similar songs:

If Song B is similar to Song E, we connect them…

So instead of checking every song, we can jump from one song to another.

Start somewhere:

• Move to a neighbor that is closer to the query,

• Then repeat.

This is much faster!..

But there is still a problem: a graph with 100 million nodes still requires “many” hops.

i.e., we need a faster way to move…

§

But What Does “Similar” Actually Mean?

Before HNSW navigates the graph, it needs a way to measure how close two vectors4 are.

Here are the most common distance or similarity metrics:

• Cosine similarity: Measures the angle between vectors. A smaller angle means higher similarity. It ignores vector length, making it ideal for text and NLP5.

• Dot product/inner product: Multiplies & sums elements. A larger positive value means higher similarity (it grows with both alignment and length). It’s commonly used in recommendation systems.

• Euclidean distance (L2): Measures straight-line distance. A smaller distance means higher similarity. It captures absolute size differences, making it great for images.

So the right similarity metric depends on the embedding model6 & use case.

For HNSW, the important idea is simple:

The closer a vector is to the query according to the chosen metric, the better the match.

§

Exact Search vs Approximate Search

There are two ways to find the nearest vectors:

• Exact search: Compares the query against every vector & returns the mathematically closest results. That gives us the true nearest neighbors, but at 100 million or a billion vectors, checking everything is “expensive”.

• Approximate Nearest Neighbor (ANN): Instead of examining the entire dataset, it explores the most promising clusters of the index and returns results that are very likely to be the true nearest neighbors.

HNSW is an ANN algorithm.

Yet this creates an important tradeoff:

• Explore more → higher recall7, higher latency.

• Explore less → lower latency, potentially lower recall.

To measure search effectiveness, we use Recall@k.

It measures how many of the true top-k nearest neighbors an approximate search actually retrieves.

HNSW gives us a way to search a tiny fraction of the dataset while still achieving high recall.

§

The Real Trick: Add Layers

This is the key idea behind HNSW:

HNSW builds many layers of graphs8.

The bottom layer contains every song.

Higher layers contain fewer songs. The higher you go, the fewer nodes exist. These upper layers create long-distance shortcuts.

This makes HNSW fast!

§

How HNSW Search Works

Here’s how:

• A user hums a song: the system converts it into a vector.

  • A vector is simply a list of numbers: [0.12, 0.87, 0.34, 0.91, ...].

  • Songs with similar vectors sound similar.

  

2. Now the search begins…

   • Step 1 (Top Layer): Performs a high-level traversal on a sparse graph (A → C → D) to locate the general neighborhood rapidly.

   • Step 2 (Middle Layers): Drops down through denser layers (D → F → G), narrowing the distance to refine the search space.

   • Step 3 (Bottom Layer): Executes a targeted, local search among millions of nodes- exploring only a tiny fraction of the dataset to return Approximate Nearest Neighbors.

§

Why the “Small World” Matters

Think about social networks:

You know someone. That person knows someone else. After a few connections, you could reach almost anybody.

Graphs with this property are called small-world graphs9.

HNSW uses this idea:

Most connections are local…some connections jump far.

Together, they create short paths through huge datasets.

§

Why “Hierarchical”?

Because there are layers…

Each higher layer contains exponentially fewer nodes.

Think of it like this:

• Layer 3: Highways

• Layer 2: Major roads

• Layer 1: Streets

• Layer 0: Every location

Search starts “fast”…then becomes “precise”.

§

How Layers Get Built?

When inserting a new vector into an HNSW index:

• The algorithm assigns the vector a maximum height layer based on a decaying probability distribution10.

• Most vectors reside exclusively on Layer 0.

• A small fraction are promoted to middle layers, and only a tiny minority reach the sparse top layer.

This is similar to a data structure called a skip list11.

Skip lists let you jump over many elements.

HNSW applies the same idea to vectors.

Randomness creates shortcuts → Those shortcuts make navigation efficient.

§

How HNSW Builds the Graph

When a new vector gets inserted, HNSW searches the existing graph to find promising neighbors.

It does NOT simply connect the new vector to the closest M vectors it happens to find. Instead, a neighbor-selection heuristic tries to keep connections useful and diverse, avoiding too many redundant links.

The process looks roughly like this:

This is where efConstruction12 matters:

• A higher efConstruction means the algorithm considers a larger candidate set while building the graph. That usually produces a better-connected graph, but indexing takes longer.

• A lower value makes construction faster, but the resulting graph may provide fewer useful paths during search.

Good graph construction matters because graph quality directly affects both search speed & recall.

§

§

Three Knobs Engineers Tune

In production systems, these three parameters matter most:

1. M

M controls how many connections each node has.

More connections mean:

• Better search quality

• More memory usage

Think of it as building more roads:

• More roads improve navigation.

• But roads are expensive.

2. efConstruction

This controls how carefully the graph gets built.

Higher values mean:

• Better graph quality

• Slower indexing

Lower values mean:

• Faster indexing

• Slightly lower search quality

This matters if you insert data frequently.

3. efSearch

This determines how many candidate nodes the system explores during the search.

Higher values mean:

• Better recall

• Higher latency

Lower values mean:

• Faster queries

• Lower recall

For example:

efSearch = 50
>Recall = 92%
>Latency = 8 ms


efSearch = 200
>Recall = 98%
>Latency = 20 ms

The exact values depend on the dataset...this is an engineering tradeoff.

§

Why HNSW Uses So Much Memory

HNSW stores:

• Vectors,

• Graph connections,

• Metadata.

The graph itself could become large…for millions/billions of vectors, memory usage grows quickly…

So many systems combine HNSW with compression techniques such as:

• Scalar Quantization (SQ8): Compresses float32 vectors down to int8, reducing RAM footprint by ~75% with minimal recall impact.

• Product Quantization13 (PQ): Segments vectors into sub-vectors and quantizes them into cluster centroids, unlocking extreme memory savings for billion-scale indexes.

§

When to Use HNSW

HNSW is a good choice when:

• Search speed matters: get results in milliseconds at large scale.

• High recall matters: for the search to capture most of the truly similar results.

• The dataset fits in memory: larger datasets demand more RAM, since HNSW keeps its graph in memory

• Data updates are moderate: HNSW works best when vectors get added occasionally, and not constantly.

Best use cases:

• RAG systems,

• Image search,

• Audio matching,

• Semantic search,

• Recommendations.

But HNSW is NOT ideal when:

• Memory is very limited: Graph connections add significant RAM overhead.

• Data changes constantly: Frequent updates make the graph expensive to maintain.

• Exact nearest neighbors are required: HNSW is approximate, so it occasionally misses the mathematically closest match.

No algorithm is free → HNSW trades memory for speed & search quality.

§

The HNSW Mental Model

HNSW is built around one simple idea:

Don’t make the search faster…Make the search smaller.

Instead of looking at every vector, build a structure that tells you where to look.

The idea is simple…The scale it unlocks is not.

Millions of vectors. Billions of vectors. Milliseconds.

That’s the power of HNSW.

§

I’d like to thank Avani Chaskar for writing this newsletter.

She writes technical guides on building, evaluating & scaling deterministic AI systems.

Subscribe to her newsletter for more such quick reads on AI Engineering topics.

§

If you’re serious about AI engineering, you really can’t miss out on the footnotes:

Get Instant Access Now

Want to reach 250K+ tech professionals at scale? 📰

If your company wants to reach 250K+ tech professionals, advertise with me.

Thank you for supporting this newsletter.

You are now 250,001+ readers strong, very close to 251k. Let’s try to get 251k readers by 17 October. Consider sharing this letter with your friends & get rewards.

Y’all are the best.

Share

Read more

• Share this letter & I’ll send you some rewards for the referrals.

No time for small talk today:

The following is the second of a premium 2-part newsletter... if you’re serious about API design in 2026, then this newsletter is for YOU.

On with part 2 of the newsletter:

===

Some of these API design concepts are foundational, and some are advanced. ALL of them are super useful to software engineers building scalable APIs.

Curious to know how many were new to you:

18. JSON Web Token,

19. Throttling,

20. Endpoint,

21. Request-Response,

22. Headers,

23. OpenAPI,

24. Timeouts,

25. Microservices,

26. WebSockets,

27. CORS,

28. HTTPS and TLS,

29. Filtering & Sorting,

30. Retries with Backoff,

31. Load Balancing,

32. Circuit Breaker,

33. Error Handling.

For each, I’ll share:

• What it is & how it works (in plain English),

• A real-world analogy (if I found one),

• Tradeoffs,

• Why it matters.

Let’s go!

§

[Webinar] How to stop babysitting your agents (Partner)

Agents can generate code. Getting it right for your system, team conventions, and past decisions is the hard part. You end up wasting time and tokens in correction loops.

More MCPs give agents access to information but not understanding. The teams pulling ahead use a context layer to give agents exactly what they need.

Join live on Oct 7 (FREE) to see:

• Where teams get stuck on the AI maturity curve

• How a context layer solves for quality, efficiency, and cost

• Live demo: the same coding task with and without a context layer

Register now

(Thanks to Unblocked for partnering on this newsletter.)

§

18. JSON Web Token

JSON Web Token (JWT) is a signed token that contains claims about the caller.

Three parts, separated by dots.

• Header names the algorithm.

• Payload holds the claims: this is user 42, they expire at noon, and have the admin role.

• Signature proves nobody altered the first two parts. The server verifies signature with a key it already holds, reads the claims, and answers the request without touching a session store.

The token carries its own proof, so verification costs no database call.

Analogy

A festival wristband.

It says which zones you may enter and which day it dies. Security reads it in a glance without radioing the ticket office. If you tamper with it, the seal breaks, so the guard knows.

Tradeoff

You cannot easily revoke a token.

Fire an employee at nine, and their token still opens doors until it expires. The payload is signed, not encrypted, so anyone holding the token can read every claim inside it.

So keep expiry short, keep secrets out of payload, and use a refresh token to renew.

Why it matters

Stateless authentication across microservices, and the access token most OAuth providers hand back.

Its structure is fixed by a public specification, which is why every language has a library for it. Pair it with Authentication, which is the job it exists to do.

19. Throttling

Throttling slows down a caller instead of turning them away.

Rate limiting & throttling get used interchangeably, and they do different things.

A rate limiter counts requests and rejects the ones over the cap with a 429. A throttle accepts the request and delays it. It queues the call, releases it when capacity frees up, or answers with a Retry-After that asks the client to come back shortly. The work still happens, just later and more slowly.

Rejection protects the server…Throttling protects the server & keeps the caller alive.

Analogy

A metered on-ramp to a motorway.

The light drips cars onto the carriageway one at a time. Nobody is sent home. The queue on the ramp is annoying, and it's why the motorway itself keeps moving at seventy.

Tradeoff

A queued request holds a connection, and enough of them exhaust the very resource you were protecting.

Plus, delay hides the problem, so a service that has been struggling for an hour looks merely sluggish. So cap the queue & alert on the delay rather than the errors.

Why it matters

Expensive operations, shared infrastructure, and any burst you would rather absorb than reject.

Envoy, Kong, and most message queues implement it. Pair it with rate limiting, and be explicit in your documentation about which one you are doing.

20. Endpoint

An endpoint is a uniform resource locator (URL) that points to one resource on your server.

It’s the address you aim a request at.

• /users is the collection.

• /users/42 is one user.

• /users/42/orders is that user’s orders.

The path names the resource & the method names the action; together, they define one operation. Get the path wrong, and you receive a 404 rather than the data you meant to ask for.

Read a good endpoint aloud, and it explains itself.

Analogy

A street address.

The number and the road identify one door out of a million. Post a letter to the wrong number, and it arrives somewhere real, which is worse than it not arriving at all.

Tradeoff

Endpoint names drift as the API grows.

One person writes /users, the next writes /getUser, and six months later your API reads like it was designed by a committee that never met.

Use plural nouns for collections and let the verb live in the method.

Why it matters

The first thing anybody learns about your API, and the thing they will curse if you got it wrong.

Every REST framework routes on them. Pair it with OpenAPI, which turns your endpoint list into documentation nobody has to maintain by hand.

Share

21. Request-Response

The request-response cycle is the basic way clients & servers communicate. The client sends a request, and server returns a response.

Every call follows the same simple script.

A request names what you want and where: a method, a path, a few headers, a body when there is one to send. The server reads it, does the work, and answers with a status code and usually a body of its own. The exchange is synchronous, so the client holds the line open and waits for that single reply before it does anything else.

Send, wait, read the reply. That is the entire cycle.

Analogy

Posting a letter and waiting by the mailbox.

You write the address on the envelope, the details inside, and a return address on the back. Then you stand there. Nothing else happens until the reply lands.

Tradeoff

The client blocks and a slow server holds that connection, and work that takes two minutes leaves a user staring at a spinner. Nothing comes back until everything comes back.

Accept the request, return a 202, and finish the work in the background.

Why it matters

The foundation under REST, GraphQL, and almost every API you will build.

Its rules are set by the HTTP specification. Pair it with Webhooks and WebSockets, the two answers for when waiting is the wrong shape.

Want life to feel a little easier?

Ben Meer’s new book, How to Be Good at Life, is packed with practical systems for work, health, money, relationships, home, and more.

Get the book →

22. Headers

Headers are key-value pairs that carry information about a request/response, separate from the body.

They describe the message rather than contain it.

• Content-Type says what format the body is in.

• Authorization carries the credential.

• Cache-Control tells every cache along the path how long to keep a copy.

• Accept states what the client can understand.

• User-Agent names the software that called.

Servers, proxies, and browsers all read them before touching the body.

Metadata belongs on the outside.

Analogy

The outside of an envelope.

Stamps, postcodes, fragile stickers, return address. Every courier along the route reads the outside and acts on it. None of them open the letter.

Tradeoff

Headers are easy to forget, which is why so many bugs end in a Content-Type nobody set.

Sizes are capped, casing rules surprise people, and a header stripped by a proxy fails silently rather than loudly. So log the headers you rely on, and set Content-Type explicitly every time.

Why it matters

Authentication, caching, versioning, compression, and cross-origin rules all ride on them.

Any browser’s network tab shows you every one. Pair it with Caching, which does nothing at all unless the headers say so.

23. OpenAPI

OpenAPI is a standard file that describes every endpoint, parameter, and response of your API in one place.

You write it in YAML/JSON, and machines read it.

From that one file, you generate interactive documentation, client libraries in eight languages, a mock server for the front end to build against, and a test suite that catches the day your response shape quietly changed. The specification becomes the contract, and the contract becomes the source of truth.

Write it once & stop maintaining documentation by hand.

Analogy

An architect’s blueprint.

The plumber, electrician, and joiner all work from the same drawing without ever meeting. Change the drawing, and everyone sees the change. Change the wall without changing the drawing, and somebody drills through a pipe.

Tradeoff

A hand-written spec drifts from the code within weeks, and a spec that lies is worse than no spec at all. The file grows long, and reviewing a two-thousand-line YAML diff is nobody’s favorite afternoon.

Generate the spec from your code, or test your code against the spec.

Why it matters

Public APIs, partner integrations, and any backend whose consumers sit in a different room.

Swagger UI, Redoc, and Stoplight render it. Pair it with API Versioning, since each version deserves its own contract.

24. Timeouts

A timeout is the longest a caller will wait for a response before giving up.

Without one, a caller waits forever.

Set a connect timeout for reaching the server and a read timeout for hearing back. When a downstream service hangs, your request fails in two seconds instead of hanging alongside it. This matters because every waiting request holds a thread, a connection, and a slot in your pool. Enough of them & your healthy service falls over next to the sick one.

A fast failure beats a slow hang.

Analogy

Hanging up after thirty seconds of ringing.

Nobody is home. Listening to the ringtone for an hour won't make them answer, and it stops you from calling anybody else.

Tradeoff

Set one too tight, and you kill healthy requests that were simply slow; then you retry them, and you have doubled the load on a service that was only having a bad minute.

Measure your ninety-ninth percentile, then set the timeout above it.

Why it matters

Every network call you make is the first line of defense against a cascading failure.

Every HTTP client has a default, and the default is usually wrong. Pair it with Circuit Breaker, which you reach for when timeouts stop being occasional.

25. Microservices

Microservices split one application into small, independent services that talk to each other over APIs.

Each service owns one job and its own data.

The orders service owns the orders table. The payments service owns payments. Neither reaches into the other’s database. They call each other’s APIs, deploy on their own schedule, and scale independently, so a search feature under load gets more servers without anybody touching checkout.

Independence is the whole point, and the network is the whole cost.

Analogy

A food court instead of one enormous kitchen.

Each unit runs its own stoves, staff, and menu. The pizza place closes for a refit, and everyone else keeps serving. Ordering a meal from three units means three queues.

Tradeoff

Every function call you used to make in memory is now a network call that can time out, arrive twice, or never arrive.

A bug spans four services and six log files. Keeping data consistent across services is a hard problem with no free answer. So start with a monolith, and split it when the pain is real, not theoretical.

Why it matters

Large systems, large organizations, and any codebase where two changes cannot ship without waiting for each other.

The patterns that make it survivable deserve an article of their own. Pair it with API Gateway and gRPC, which keep the pieces reachable and fast.

§

§

26. WebSockets

Read more

• Share this letter & I’ll send you some rewards for the referrals.

Some of these API design concepts are foundational, and some are advanced. ALL of them are super useful to software engineers building scalable APIs.

Curious to know how many were new to you:

1. REST,

2. GraphQL,

3. Authentication,

4. Authorization,

5. Rate Limiting,

6. Caching,

7. Pagination,

8. Status Codes,

9. Webhooks,

10. Idempotency,

11. API Gateway,

12. API Versioning,

13. JSON,

14. OAuth 2.0,

15. HTTP Methods,

16. gRPC,

17. API Keys.

(…and much more in Part 2!)

For each, I’ll share:

• What it is & how it works (in plain English),

• A real-world analogy (if I found one),

• Tradeoffs,

• Why it matters.

Let’s go!

§

Beyond the Coding Agent: Building at Enterprise Scale (Partner)

Coding agents made individual tasks faster. They didn’t remove the coordination, validation, and handoffs that actually determine how fast enterprise software ships.

Join Sid, CTO and Co-founder of Blitzy, for a live session on what changes when a system can hold context across an entire codebase, not just a single file or function. Sid will walk through a real project in Blitzy, covering everything from reverse engineering and tech spec generation to the Agent Action Plan, code generation, and PR Review. You’ll see what happens at each stage and where human review gates the process.

What you’ll see:

• Reverse engineering and tech spec generation on a real codebase

• Where human review sits in an autonomous build: sign-off at the plan stage, approval at the merge stage

• Enterprise modernization results

We’ll close with an open Q&A, so bring your questions on agent orchestration and governance at scale.

Save Your Seat

(Thanks to Blitzy for partnering on this newsletter.)

§

1. REST

Representational State Transfer (REST) is a way to build APIs that organizes data as resources and uses standard HTTP methods to work with them.

Every resource gets its own address.

Each resource has its own URL. For example, a user is at /users/42 and an order is at /orders/17.

Use:

• GET to read it,

• POST to create it,

• PUT to replace it,

• DELETE to remove it.

The HTTP method defines the action, and the URL identifies the resource. Both the client and server follow the same contract.

REST also stays stateless, so any server can answer any request.

Analogy

A library with a fixed shelf number for every book:

You hand the librarian a shelf number and say borrow, return, or replace. They never ask what you meant.

The number identifies the book; the verb identifies the action; they leave no room for guessing.

Tradeoff

REST wastes requests & bandwidth.

One screen may need a user, their orders, and their payment method, which costs three round trips. Ask for a user, and the server sends every field, including the twenty you ignore. Add a field for one client, and you risk breaking another.

So keep your resources small and reach for GraphQL when clients need to shape their own responses.

Why it matters

Public APIs, mobile and web clients, anything cached at the edge, and any service where a new developer should guess the right endpoint on the first try.

Spring, Express, Django REST Framework, and FastAPI all lean on it. Pair it with OpenAPI so the contract stays readable, and with API Versioning so today’s shape survives tomorrow’s change.

2. GraphQL

GraphQL is a query language for APIs that lets clients request exactly the data they need in a single response.

There is one endpoint instead of dozens.

Client sends a query that names the fields it needs. The server walks the query, fetches each field, and returns a response shaped like the request. Queries read data, mutations change it, and subscriptions stream updates as they happen. Ask for three things at once, and you get one round trip.

The client decides the shape, so the server stops guessing.

Analogy

A sandwich counter where you name every ingredient:

A fixed combo hands you the pickles you scrape off and skips the extra cheese you wanted. Naming each ingredient gets you the sandwich you meant to order, in one pass down the counter.

Tradeoff

GraphQL costs you more.

Caching gets harder because every query is a POST to the same URL. Plus one careless nested query could walk your database for a minute. You carry a schema, a resolver for every field, and a plan for the N+1 problem.

So set a depth limit and a query cost budget on day one.

Why it matters

Mobile clients on slow networks, screens that stitch together several resources, and front ends that change faster than the backend.

Apollo, Relay, Hasura, and graphql-js are common choices. Pair it with Rate Limiting, because a single expensive query can cost more than a thousand REST calls.

3. Authentication

Authentication proves “who” is calling your API.

The caller presents a credential & server checks it.

That credential could be a password, an API key, or a signed token. The server verifies it against something it trusts, then attaches an identity to the request. Everything downstream reads that identity.

A caller who fails the check gets a 401 and never reaches the business logic.

Analogy

Showing your passport at the border desk.

The officer compares the photo to your face & the name to a list. They are not deciding where you may travel inside the country. They only confirm you are the person on the document.

Tradeoff

Each check costs a lookup/signature verification.

Secrets leak through logs, screenshots, and public repositories. Sessions expire at the worst moment for the user.

So rotate credentials on a schedule & keep them out of your URLs.

Why it matters

Every API that touches private data, money, or a user account.

Auth0, Okta, Keycloak, and Firebase Auth handle the hard parts. Pair it with Authorization, because proving who you are says nothing about what you may do.

Share

4. Authorization

Authorization decides what a proven caller is allowed to do.

Authentication runs first, then authorization runs.

The server knows the identity & now checks the rules:

• Does this user own this order?

• Does this token carry the write scope?

• Does this role permit a delete?

A caller who fails the check gets a 403 i.e., the server knows exactly who you are and still says no.

Authorization answers a second question: what may you touch?

Analogy

A hotel keycard.

The card proves nothing about your name. It opens your room and the gym on the third floor. It stays dead against the manager’s office and the room next door, no matter how many times you tap it.

Tradeoff

Rules sprawl across the codebase, and one missing check becomes a breach.

Test the negative cases, because a passing test that proves a user can read their own order says nothing about whether they can read somebody else’s.

So check ownership at the data layer & at the route.

Why it matters

Multi-tenant systems, admin panels, and any endpoint where one user’s record sits next to another’s.

Role-based access control and attribute-based access control are the two common models. Pair it with OAuth 2.0, whose scopes carry authorization decisions across service boundaries.

5. Rate Limiting

Rate limiting caps how many requests one caller can send in a time window.

The server counts & past the cap, refuses them.

A common limit is a hundred requests per minute per API key. Cross it, and the server returns 429 Too Many Requests along with a Retry-After header. Response headers usually report the ceiling and how much of it you have left, so a well-behaved client slows itself down before it gets refused.

One noisy caller stops starving everyone else.

Analogy

A nightclub with a counter at the door.

The room holds four hundred people. The doorman counts everyone in & everyone out. When the room is full, the next person waits outside, and the people already dancing keep enjoying the music.

Tradeoff

Limits frustrate honest heavy users, and a limit that seems generous in testing looks tight during a launch.

Counting across many servers means shared state, which is one more thing that can go down. Token bucket handles bursts better than a fixed window.

Why it matters

Public APIs, free tiers, login endpoints, and any expensive operations like search or export.

Redis backs most implementations, and NGINX, Envoy, and Kong ship it built in. Pair it with throttling, which slows a caller instead of turning them away.

6. Caching

Caching stores a copy of the response so next identical request skips extra work.

The fastest request is the one your server never handles.

A cache sits between the client and the origin and holds recent responses. When a matching request arrives, the cache answers from memory in a millisecond. Cache-Control headers say how long a copy stays fresh. An ETag lets the client ask whether its copy is still good, and the server answers 304 Not Modified without sending the body again.

Read-heavy APIs live/die on this.

Analogy

A barista who writes your usual on a card.

You walk in; they glance at the card, and the drink is ready before you finish greeting them. They only ask again when you tell them your order has changed.

Tradeoff

Cached data goes stale, and a user who updates their profile & still sees the old name loses trust immediately. Knowing when to throw a copy away is one of the hard problems in computing.

So cache aggressively on data that rarely changes. Plus, never cache a private response in a shared layer.

Why it matters

Product catalogs, public profiles, search results, and anything a thousand people read for every one person who writes.

Redis, Memcached, Varnish, and any content delivery network do this well. Pair it with Headers, which carry every instruction a cache obeys.

7. Pagination

Pagination returns large datasets in smaller pages instead of sending ALL results in one response.

The client asks for a slice and gets a pointer to the next one.

• Offset pagination takes a page number & size, then skips ahead.

• Cursor pagination hands back an opaque marker that says where you stopped, and you pass it back to continue.

Both return the rows plus enough information to fetch the following page…Your database, your network, and the client’s memory will thank you.

Analogy

A book with a bookmark.

Nobody reads a thousand pages in one sitting. You read twenty, slide the ribbon in, and close the cover. Tomorrow you open straight to the ribbon rather than counting pages from the front.

Tradeoff

Offset pagination drifts…Insert a row while somebody reads page two, and they see a duplicate on page three.

Cursor pagination gives up the ability to jump anywhere except forward & back.

So use cursors for feeds & offsets for admin tables.

Why it matters

Every list endpoint, every feed, and every export.

Stripe, GitHub, and Slack all publish cursor-based APIs. Pair it with Filtering and Sorting, which decide what lands on the page before you slice it.

§

§

8. Status Codes

Read more

We are in the middle of the agentic coding era, please take advantage of it (Partner)

Coding agents are changing what it means to be a software engineer.

Typing code on a screen used to be a big part of the job. But now AI agents can handle more of that work.

The pattern of technological change is familiar:

The mechanical part of work gets automated & human role moves up a layer.

For software engineers, I think it means 3 things:

1. You need to know what good looks like.

   An agent can create a feature, fix a bug, and/or refactor code. But if you don’t know what good implementation looks like, you cannot tell whether the work should ship. The faster agents write code, the more your judgment matters.

2. You need to learn how to manage multiple agents.

   One developer can already give different tasks to different coding agents. That changes the job. You become responsible for:

   • giving context to agents,

   • directing their work,

   • checking results.

3. You need a way to control this across the company.

   Imagine all engineers in your company working this way…You need to know:

   • which “agents” people use,

   • what they’re doing & what they cost,

   • what happens when something goes wrong.

This is why I’m genuinely excited about JetBrains Air.

It gives you an environment for directing multiple coding agents & reviewing their work. So you can delegate more coding work without losing visibility/control.

Plus...

• JetBrains Air Teams lets you coordinate software delivery workflows across engineers & agents. So engineering teams can manage agentic work across the development process.

• JetBrains Air Governance gives organizations policies, visibility, auditability & cost management. So they can control how agents get used & maintain accountability for what ships.

And this works in a world where engineers use different agents & models.

i.e., you don’t have to bet everything on one provider.

Read more about JetBrains Air »

§

Many AI outputs require subjective/domain-specific judgment.

A useful support response or a faithful summary is hard to grade with fixed rules in code1. People have to read & interpret these outputs. So repeating that review across many outputs takes time.

To reduce this review work, you can ask a large language model (LLM) to evaluate another model’s output. This use of a model is called LLM-as-a-judge. You give the judge the output & your evaluation criteria. It returns a decision, often with an explanation. This ability to automate subjective checks makes LLM-as-a-judge a “workhorse” in evals.

Onward.

§

I want to introduce Hamel Husain as a guest author.

Hamel writes about AI evaluations and teaches the AI Evals course with Shreya Shankar. In this newsletter, he explains how to build an LLM judge and check its decisions against human labels.

§

Here’s what’s inside this newsletter:

• A high judge score can still hide a broken evaluator. Why a judge that looks accurate could completely miss the failures you actually care about.

• Your human labels become the standard. The decisions, critiques & rubrics that determine what your LLM judge should call Pass or Fail.

• The examples your judge sees change its decisions. Why some examples belong inside the prompt, while others must stay hidden until the final test.

• One number won’t tell you whether your judge works. The measurements that reveal how often it misses real failures & incorrectly flags good outputs.

• Your judge gets stale as your AI product changes. What happens to your evals when models, user behavior & evaluation criteria change over time.

• A 6-step process takes you from real failures to a validated LLM judge. The complete framework for choosing what to evaluate, defining the human standard, building the judge & checking whether you can trust its decisions.

• Share this letter & I’ll send you some rewards for the referrals.

§

The judge can work alongside code-based checks:

• Use code for checks with clear rules, such as whether an output follows the required format.

• Use the judge for decisions that depend on meaning/context.

For example, code can check whether a support response contains the words “I am sorry.”

An LLM judge can assess whether the response addresses the customer’s frustration. Give the judge examples of acceptable responses so it can apply your support team’s standards.

Also code can check whether an answer includes a source link.

An LLM judge can compare the answer with the source to check whether the source supports the claim. Give the judge the source text so it has the evidence it needs.

For either task, the judge can make mistakes.

Before you rely on it, test whether its decisions match human judgments.

Even high agreement can hide these mistakes. If an error occurs in 5% of examples, a judge that always predicts Pass still has 95% agreement while detecting none of the errors.

To avoid this false confidence, measure whether the judge catches the failures that matter.

This newsletter starts with finding those failures and defining the human standard. It then shows how to build a judge prompt & test its decisions against that standard2.

§

1. Choose a failure worth judging

When you start building evals, it is tempting to use a metric3 someone else has already written.

Eval libraries offer ready-made scores for qualities such as helpfulness and coherence. You can run them on your outputs and get a score before deciding what a good result means for your product.

These metrics measure abstract qualities that may not matter for your use case.

Good scores on them do not mean your system works. For example, an answer might receive a high helpfulness score even though its sources do not support its claims4. You need a check for this specific failure.

Generic scores5 can still help you choose outputs to review…

For example, sort outputs by helpfulness & read some low-scoring examples. You may find a recurring problem worth checking. Read the outputs yourself to decide whether there is a failure and describe it. Keep some random examples in each review batch so you can find problems the generic score misses.

To decide which specific failures to check, review actual interactions and write notes about each problem. Then group similar problems into categories. This process is called error analysis6. Error analysis helps you decide what evals to write in the first place.

To understand an incorrect answer, you may need to see what happened before it.

Review the complete record of the user session, called a trace. It includes the user’s request & the model’s response, along with any tool calls and their results.

For example, several answers may include sources that do not support their claims. Group these cases under one failure category, such as “unsupported claims.” You can then decide whether that failure needs an automated check.

To find these patterns:

1. Gather about 100 traces that cover different users & workflows.

2. Then review at least 30 traces yourself.

3. As you read, write free-text notes about anything that seems “wrong” from the user’s perspective.

Do this initial review before reading an agent’s suggestions.

The agent may lack the product context you use to judge a good user experience. Its suggestions can direct your attention toward the problems it recognizes, causing you to miss failures it does not recognize.

After reviewing the first 30 traces, group similar failures and count the number in each category. Use your notes to ask the agent to search the remaining pool for similar examples. Review every suggestion yourself.

Continue until new traces stop revealing types of failure or changing your categories7.

The counts help you choose which failures to address first. For each failure, decide whether to fix it directly or build an automated check, often called an evaluator/eval.

Focus automated evaluators on failures that persist after fixing your prompts.

Teams sometimes discover their LLM doesn’t meet preferences they never specified, such as wanting short responses or a particular format.

Fix these obvious gaps first8.

Plus, choose the simplest check that can detect the failure.

For example, you can run generated code to check whether it produces the expected result. Use an LLM judge when the check requires human judgment.

An LLM judge needs labeled examples and ongoing maintenance9.

Before you can automate this judgment, you need a consistent human standard…

§

2. Establish the human standard

After choosing a failure to evaluate, you need to decide which outputs should “pass”.

People can disagree about whether the same output is acceptable. To establish a consistent standard, find the person who understands what a good result looks like for your product. A customer service director might be the right expert for a support assistant. An educational tool may need a teacher/curriculum developer.

For a small team, choose one domain expert to make final decisions about quality.

This reduces the work of coordinating several reviewers & resolving conflicting labels. The expert can incorporate input and feedback from others, but they drive the process. You then have a consistent standard to use when building and testing the judge.

The expert’s critiques also help uncover expectations people have not yet put into words. Use those critiques to define what the judge should check. If you are an independent developer with the relevant expertise, you can fill this role yourself.

Give the expert the information they need to evaluate each interaction.

That includes the user’s request & the model’s response. Depending on the task, they may also need results from tools or relevant information about the user. Present the relevant evidence together so they can review it without switching between several systems10.

Use Pass and Fail, with a critique

The expert needs a way to record each decision.

A 1-to-5 rating may seem useful for capturing degrees of quality. But the difference between adjacent points, like 3 versus 4, is often subjective and inconsistent across people labeling the outputs.

People can also default to middle values to avoid making a hard decision. Ask the expert to mark each example as Pass or Fail for the specific problem you are checking.

An evaluation with two possible labels is called binary.

Binary evaluations force clearer thinking & more consistent labeling. Besides the pass/fail decision, the domain expert should write a critique explaining their reasoning.

Use the same Pass or Fail outcomes for the judge. Each decision should answer one specific question about the output. A critique explains the decision; a middle score can leave it unclear whether the output is acceptable.

The critique identifies what caused the output to pass or fail and records the expectations the judge needs to follow.

You can still measure gradual improvement.

For example, you can track whether each expected fact appears in an answer. An answer that includes four of five expected facts passes four specific checks11.

Expect your criteria to change

Your evaluation criteria can change as you review outputs & discover expectations you hadn’t considered. (Shreya Shankar & her colleagues call this “criteria drift12.”)

Reviewing the judge’s critiques can help the expert notice inconsistencies in how they evaluate edge cases.

If several people label examples, check whether they apply the same standard…

Have these annotators label the same examples independently before they discuss them. Measure agreement and collect the cases where their labels differ. Discuss the rules for deciding Pass or Fail. These written rules are your rubric.

Update the rubric with a definition or example that covers the disputed case. Then relabel affected examples. If the annotators still disagree, assign a domain expert to make the final decision and record the reason13.

Start with binary labels to understand what “bad” looks like.

The expert’s labels & critiques give you the material for the judge’s prompt.

§

§

3. Write a prompt from expert examples

Use the expert’s critiques & evaluation rules to write the judge’s prompt.

Include labeled examples14 showing how to apply the rules to actual outputs.

Consider an assistant that turns a user’s request into a query for searching application data. The judge checks whether the generated query answers the request.

A user asks, “show me slowest trace.”

In this example, a trace records the steps an application takes to handle a request. The generated query found a maximum duration but didn’t group the data by each trace’s identifier.

It returned a duration without identifying the slowest trace15.

This shortened prompt includes the query & the expert’s critique.

Replace the placeholders with your application’s query-language information & evaluation guidelines16.

You are a query evaluator.
Evaluate whether the generated query answers the user’s request.

Here is information about the query language:
{{query_language_info}}

Here are the evaluation guidelines:
{{guidelines}}

Example evaluation:
<nlq>show me slowest trace</nlq>

<query>
{
  “calculations”: [{”column”: “duration_ms”, “op”: “MAX”}],
  “orders”: [{”column”: “duration_ms”, “op”: “MAX”,
              “order”: “descending”}],
  “limit”: 1
}
</query>

<critique>
{
  “critique”: “While the query attempts to find the slowest trace
  using MAX(duration_ms) and ordering correctly, it fails to group
  by trace.trace_id. Without this grouping, the query only shows
  the MAX(duration_ms) measurement over time, not the actual
  slowest trace.”,
  “outcome”: “Fail”
}
</critique>

For the following query, write a critique explaining your decision.
Then provide a Pass or Fail judgment in the same format as above.

<nlq>{{user_input}}</nlq>
<query>{{generated_query}}</query>

Include clear Pass examples alongside Fail examples to show the distinctions in your rubric.

Examples included in a prompt are often called “few-shot examples.”

Keep separate examples for testing the judge (described in the next section).

Keep each judge focused on one type of failure. Use separate judges for unrelated failures so you can see which behavior needs fixing17.

To check whether the prompt captures the expert’s judgment, compare their decisions on the same queries.

Put each user request and generated query in a spreadsheet with the judge’s critique and decision. Ask the expert to add their own critique and decision.

Use the disagreements to improve the prompt18.

Give the judge the evidence it needs

Some disagreements come from missing evidence.

To check a claim against a database, the judge needs the database result alongside the model’s answer. Your examples should contain the same information you use to evaluate.

Give each judge only the parts of the trace relevant to the failure it checks. Extra context can make its decisions less accurate.

Test your choices by comparing the judge’s decisions with human labels. Then, inspect disagreements to see whether the judge lacked necessary evidence or was distracted by irrelevant information.

Remove one piece of information at a time and check how the results change against human labels. If performance stays the same/improves, you may be able to leave it out.

Long-running agents can produce traces longer than the judge can read in one request. In those cases, consider giving the judge a tool to search for the parts it needs. Add this only when necessary, because it introduces more complexity & cost19.

You still need to check the judge on examples you did not use to develop its prompt.

§

§

4. Check whether the judge catches failures

A convincing critique can contain an incorrect decision.

Before relying on the judge, check how often its decisions match your domain expert. This check is called validation.

Run the judge on labeled examples you kept out of the prompt & inspect the mistakes20.

Separate prompt development from the final test

Repeated changes make a prompt work well on examples you use to develop it.

You also need to know how it performs on new examples… so keep a separate group for the final test.

The first 30 traces help you discover failures.

Yet testing the judge requires more labeled examples. Plan to label 100 to 200 examples for each type of failure. Reuse labeled traces from error discovery when they match that failure, then collect more until you reach that range. Include enough Pass & Fail examples to measure how well the judge recognizes each.

Split the labeled examples before you start revising the judge prompt21:

• Training set contains examples you may put in the prompt. Here, “training” means choosing prompt examples. You do not change the model itself. Choose clear Pass and Fail examples with useful critiques.

• The development set (dev set) is where you compare the judge with human labels while refining the prompt. Keep these examples out of the prompt.

• Test set stays hidden until you finish revising the prompt. Freeze the prompt and run the judge on the test set once. Use the result as your final estimate of the judge’s performance.

Use 10 to 20 percent for train examples22 you may put in the prompt.

Use 40 to 45 percent for dev while refining the judge. Reserve the remaining 40 to 45 percent for one final test. The number of Pass and Fail examples is more useful than the percentage alone. Aim for 30 to 50 Pass examples and 30 to 50 Fail examples in both dev and test when possible.

A total of 100 examples cannot meet those targets for both dev & test. Collect more when you need better estimates, especially when failures are rare23.

On the dev set, read the disagreements. Check for missing rules or evidence. Clarify ambiguous examples and correct any mistaken human labels. Run the judge on the dev set again after each revision.

If you revise the prompt after seeing test results, those examples have become part of development. Reserve new examples for the next final check.

The >90% target shown here is an example. Set targets based on the cost of missed failures and false alarms in your application.

Measure failures and passes separately

The judge can miss a failure or incorrectly mark an acceptable output as Fail.

Measure these mistakes separately. Count Fail as a positive result because a failure is what you are trying to detect. Measure:

• True positive rate (TPR) is the fraction of human-labeled failures that the judge correctly marks Fail. This is also called recall for failures.

• True negative rate (TNR) is the fraction of human-labeled passes that the judge correctly marks Pass.

Suppose your test set contains 50 human-labeled failures and 50 human-labeled passes. These numbers illustrate the calculation.

The judge catches 45 of 50 failures, so TPR is 90%.

It correctly passes 40 of 50 acceptable outputs, so TNR is 80%.

The judge from the opening, which always predicts Pass, has a TPR of 0% despite its 95% accuracy.

Report the counts alongside the rates.

A result based on a handful of failures gives you little evidence about performance on future failures. When using these rates to make decisions, include a range that accounts for uncertainty from testing only a sample.

If people review the outputs the judge flags, measure precision too. It answers, “Of the examples flagged Fail, how many were failures according to the human?” In the illustrative table, that is 45 out of 55, or about 82%. Precision also depends on how common failures are. A test set with equal numbers of passes and failures may produce a different precision from what you’ll see with real users24.

Your acceptable error rates depend on how you use the judge. Report TPR & TNR separately.

Use these same tests to compare the models you could use as your judge.

§

5. Choose the model using human labels

I like to use the most powerful model I can afford within my budget for cost & response time.

This budget might be different than my primary model, depending on the number of examples I need to score.

Using the same model is usually fine because the judge is doing a different task than your main LLM. The judges I recommend make specific Pass or Fail decisions.

Compare the judge with human labels25 to check whether it works26.

Check the judge against human labels, even when it uses a different model.

If the judge disagrees too often with human labels, inspect examples and prompt before trying another model.

You can optimize for cost later once you’ve established reliable evaluation criteria.

If prompt changes are not enough, you might consider further training the judge on labeled examples. This is called fine-tuning.

I prefer not to fine-tune LLM judges. I’d rather spend the effort fine-tuning the actual LLM instead27.

What about TypeSafe’s Jev?

TypeSafe’s Jev has been very popular recently.

You may be wondering how it fits into this process. Jev returns decisions with probabilities, which makes it a useful example of how the same evaluation approach applies to different models.

An LLM judge is ultimately a “classifier”.

A classifier assigns an input to a category. In this newsletter, the input is an output you want to evaluate, and the categories are Pass and Fail. Choosing between two categories makes this a binary classifier.

You can use any approach you want to make those decisions, including Jev.

Validate the classifier against human labels so you know how well it works. Compare candidates on the same examples and measure TPR and TNR. You can then make a deliberate tradeoff between the mistakes each classifier makes and what it costs to run. For example, you might accept more missed failures for a lower cost, if those failures are inexpensive to fix.

Once the judge meets your requirements, keep checking it as your application changes.

§

§

6. Keep the judge useful

Changes to your application can produce outputs absent from the examples used to test the judge28.

So continue to check the judge against human labels as you use it to review new interactions. Read the failures it identifies to decide what to fix in your application.

I conduct this human review at regular intervals and whenever something material changes. For example, if I update a model, I’ll rerun the process. Your product’s criteria can also change as you see more outputs. Update the rubric and relabel affected examples when that happens.

Have the expert review a sample that reflects the interactions your users have. Give extra attention to cases where the judge is least reliable.

For example, if I find an error, I’ll search for more examples I think might trigger the same error. However, I also do a bit of random sampling. Use the random sample to estimate how often users encounter errors.

Eval datasets naturally get stale as your product and users change. Review new interactions to find problems and update your examples or reference answers29.

As your eval set changes, its scores may no longer be directly comparable with older scores. Track what changed before interpreting a higher or lower pass rate.

§

Before you rely on the judge

Check these points before using the judge’s scores to make product decisions:

• Choose a specific failure you have observed in real outputs.

• Have a domain expert label examples Pass or Fail and explain each decision.

• Build the prompt from those rules and examples. Keep separate examples for the final test.

• Measure how often the judge catches failures and accepts good outputs. Report the counts with the rates.

• Repeat human review when your application or evaluation criteria change.

If the judge misses too many failures or flags too many acceptable outputs, review the disagreements before using its scores.

§

Thanks to Hamel for writing this newsletter.

Read his LLM judge guide and Evals FAQ for more examples. He teaches the AI Evals course with Shreya Shankar.

If you find this newsletter valuable, share it with a friend, and subscribe if you haven’t already. There are group discounts, gift options, and referral rewards available.

Subscribe now

Thank you for supporting this newsletter.

You are now 250,001+ readers strong, very close to 251k. Let’s try to get 251k readers by 7 October. Consider sharing this letter with your friends and get rewards.

Y’all are the best.

Share

Helpfulness check → Pass
The response appears useful.

Source-support check → Fail
The source does not support the response’s claims.

Review real interactions → find problems → group similar problems → create evals for those failures.

Evaluation rules = what to look for
Labeled examples = what those rules look like in practice

Trace A → 2 seconds
Trace B → 8 seconds
Trace C → 4 seconds

Factual accuracy → Pass
Completeness → Fail
Source support → Pass

Training set → build prompt → Dev set → improve prompt → Test set → measure final performance.

200 total labeled examples → 20 train + 90 dev + 90 test

I’m happy to partner with Svix on this newsletter.

Agents are killing your apps with compounding polling across workflows.

Your customers want their agent workflows to be efficient, so you’re losing deals if you don’t make them event-driven. Webhooks solve this issue, but implementing them is harder than it looks.

If you homebrewed your webhooks and are getting pinged at 4 in the morning for failed deliveries, this is for you.

Svix handles everything: retries, idempotency, security, and compliance. Qualified startups get $12,000 in free credits.

Get started for free

If you’ve used an AI agent1, you’d have noticed how much it can do on its own…

Give it a task; it calls a few tools, then keeps going without you guiding every step.

But agents still are NOT good at one thing: waiting.

An agent would wait for someone to approve a request, a payment to go through, a deployment to finish, or a customer to reply. The agent has done its part, but what it’s waiting for might happen seconds later or not until tomorrow…

It could keep checking for an update every few seconds, but this’d become “wasteful” pretty quickly. i.e., the agent is still doing nothing except asking whether anything has changed.

A better approach is to let the system send an event when the change actually occurs. So the agent does not need to keep asking & continues the workflow when the event arrives.

This is where “webhooks2” become useful.

It lets one system notify another when something happens, and it's been running behind many apps you use daily, long before AI agents needed it.

Now apply that to everything an agent would wait on: payments, deployments, customer replies, or human approval.

Plus, handling one of these events is easy. Yet handling thousands of them reliably across different customers & external systems gets extremely HARD…

In this newsletter, we’ll use Svix as a case study & look at how it solves this problem.

(Svix is a webhook infrastructure platform for reliable event delivery.)

§

• Share this letter & I’ll send you some rewards for the referrals.

Onward.

§

What Is a Real-Time AI Agent

What changes with a real-time agent is not how it thinks, but what it hears.

Let’s start with what an agent needs before getting into real-time…

Agent Fundamentals

Agents (real-time or not) run on these same parts:

• A Large Language Model3 (LLM) does the actual reasoning.

• A decision loop4 lets it plan a step, take it, check the result, and decide what comes next.

• Tools & integrations let it act beyond generating text, such as sending requests, running commands, and/or calling APIs.

• State tracks where it is in a task, so it doesn’t lose the thread halfway through.

Without these pieces, you’re getting closer to a simple model response than an agent that can carry out a task.

Real-time is the layer on top of this…

What Makes an Agent Real-Time

A typical agent workflow starts when you give it a task.

It works through steps, uses whatever tools it needs & eventually finishes…

With a real-time agent, the work doesn’t end there. An external event could arrive after the original run ends, and the agent needs to resume the workflow when it does.

A payment could clear somewhere in the background a few minutes after the agent finished its original task. Instead of waiting for you to come back & tell it, the agent could receive the event5 and continue the workflow on its own.

It can pick a workflow back up without another prompt from you. Nobody needs to open a new chat/type anything. An agent does the asking instead.

And it can act on something arriving well after the original task ended. The original agent run might already be complete by the time the event arrives, and the agent still has to do something with it.

Also this brings up “latency”…

A real-time agent is NOT necessarily instant.

Latency occurs at several points in the workflow:

• Network adds some delay,

• Model takes time to reason,

• Tools & external APIs take time to respond.

So when we talk about a real-time agent, we’re really talking about keeping the entire path “responsive”, not just getting a fast response from the model.

If you put these together, a real-time agent isn’t just running one job well. It’s also sitting ready for whatever comes in later, on somebody’s schedule.

Now let’s look at how these pieces actually work together inside the agent…

§

Real-Time AI Agent Architecture

A real-time agent is not one piece of code that does everything.

Instead, it’s a handful of layers, each with a different job…

The model handles the reasoning, deciding what to do next based on what it currently knows. Everything else in the agent supports this decision. It takes in input, takes action, and catches external events as they happen.

Here’s what each layer does:

• Interaction: Receives input from you/another system.

• Agent & reasoning: Runs the model & its decision loop.

• Tool & integration: Performs actions on external systems.

• Event delivery: Brings external events into agent workflow.

The agent & reasoning layer runs the model and its decision loop.

The other layers support it by bringing in input, carrying out actions & delivering external events.

Synchronous vs Asynchronous Workflows

Some operations finish immediately…

Ask to look up an order status & tool call returns the result within the same run.

Other operations don't finish right away…

Processing a file could take a minute. A payment can take a few seconds to clear, sometimes longer. An approval can sit unread for two days.

Synchronous operations return their result within the same run, so the agent could use it right away. Asynchronous operations don’t, and the workflow needs to continue once the result becomes available later.

When the result becomes available, the event gets delivered to the application, which then updates/resumes the agent’s workflow.

Let’s see how the event actually reaches the agent in the first place…

§

§

How Real-Time AI Agents Connect to External Systems

When an agent needs to react to something happening in another system, a webhook is one of the simplest ways to get the event to it.

One system notifies another when something happens. i.e., the receiving side does not need to check for changes repeatedly.

Webhooks are built for “discrete” events, not continuous streams of data.

A payment clearing counts as one. So does a build finishing or an approval landing in someone’s inbox. This is exactly the kind of event an agent needs to hear about to move a workflow forward.

The idea is simple, but it helps to see what actually happens when one of these events gets sent.

So let’s look at how webhooks work...

How Webhooks Work

Here’s how:

1. An event occurs in the source system.

2. Source sends an HTTP POST to a configured endpoint.

3. The receiving application verifies the event & triggers the appropriate workflow.

4. Sender records6 whether delivery succeeded/failed.

Four steps, each depending on the last.

If you skip verification, anyone can send a fake event to the endpoint. Or if you skip recording the result, a failed delivery disappears without anybody noticing.

Webhooks work well for individual events, but real-time systems also deal with continuous information streams.

Streaming vs Webhooks

Real-time systems typically handle information in 2 ways:

• It keeps flowing continuously,

• Or it gets sent when a specific event happens.

Streaming is useful when information needs to keep flowing.

A voice conversation, a live transcript, or an agent response generated token by token all need a connection that can keep data flowing.

Webhooks solve a different problem.

They’re useful when something happens outside the agent, and the agent needs to know about it. The event happens once, and the system sends a notification instead of making the agent keep checking.

For real-time AI agents, both matter.

• Streaming keeps the user interaction responsive.

• Webhooks handle everything happening around the agent that needs a response later.

As the number of events and external systems grows, delivering those events reliably becomes hard.

Someone has to handle failed requests, retries, duplicate deliveries, and track what was delivered & what wasn’t.

Svix handles this part of the stack…It provides the infrastructure for delivering webhook events to external systems, while the application stays focused on the agent itself.

Let’s understand the challenges of running webhook infrastructure reliably…

§

§

Challenges of Building Real-Time AI Agents

A webhook integration usually works fine in testing…

You send an event, watch it arrive & everything looks good, so you move on.

Once you put it into production, things change. Servers go down mid-deploy, networks drop packets for reasons that aren’t always obvious, and traffic suddenly comes in bursts instead of at a steady rate.

None of these things are unusual in production, but they expose problems that you simply don’t see when you’re testing with a handful of events.

Most of the problems fall into a few recurring areas.

Let’s dive in!

1. Reliable Delivery

An endpoint could go down during a production deploy…A webhook sent during that window then has nowhere to go.

You send it once and give up, and the event is gone. Whatever depended on it--an agent waiting to continue a workflow, or a customer expecting a receipt--never finds out anything happened at all.

Retries handle this; not by hammering the same failing endpoint over & over.

Each attempt waits a little longer than the last, giving a struggling server time to recover before getting hit again. The event itself needs to sit somewhere durable while all this plays out too, so a crash on the sending side doesn’t erase it along the way.

And if every retry eventually runs out, the event needs somewhere to land instead of vanishing without a trace.

2. Security and Verification

Anyone who has the webhook endpoint’s URL could send a request.

Nothing about the endpoint itself tells the difference between a real event & someone typing curl commands.

This is why legitimate events get signed before they’re sent. The receiving side checks the signature7. A mismatch rejects the event, thus preventing forged events. A timestamp on the request helps prevent an old, legitimate request from being captured & resent later. Also signing secret needs to be managed & rotated over time without breaking existing endpoints.

One risk lies on the sending side instead…

Sending a webhook means requesting a URL provided by someone else. An unchecked URL could point somewhere it should not, such as an internal system instead of a real customer endpoint.

Guarding against it matters as much as verifying what comes in.

3. Idempotency and Duplicate Events

Occasionally, an agent does everything right & still sees the same event twice.

It is NOT a bug. A delivery succeeds at the receiving end, but the confirmation gets lost on its way back to the sender. The sender has no way to know the event actually landed, so it tries again, and the same event arrives twice.

For most systems, a duplicate log entry is harmless.

Yet for an agent taking real action, it isn’t. Processing the same payment confirmation twice can mean sending a receipt twice, or worse, acting on the same instruction twice. A stable ID attached to every event, unchanged across every retry, lets the consumer tell the difference.

See the same ID again, skip it & move on.

4. Backpressure, Traffic Bursts, and Isolation

Two hundred events landing over an hour looks nothing like two hundred events landing in the same second.

Agents create this kind of burst when they run many tasks in parallel.

An agent could start 200 tasks at once. If many finish around the same time, they could generate a burst of completion events instead of a steady stream.

A slow-responding endpoint quickly becomes a bottleneck.

Deliveries pile up, and the backlog can spread into the rest of the system. Throttling8 keeps this contained by limiting how fast events go out to any one destination.

Plus, isolating each endpoint from the others matters just as much, so a single customer’s bad day never becomes everyone else’s.

5. Event Ordering

Two letters mailed on the same morning don’t always arrive in the order they were sent…One gets held up somewhere, and the other doesn’t.

Webhook events work the same way.

Update a subscription, then cancel it a moment later. The update event might fail and get retried while the cancellation goes through immediately. The cancellation shows up first. An agent would assume events always arrive in order. If an update arrives late, it’ll process it last and believe a canceled subscription is still active.

Consumers shouldn’t assume events will always arrive in order.

Handling this well means checking each event’s timestamp/version number before acting. This is safer than trusting the order in which events arrive. When strict ordering matters, you can enforce it by controlling how events get processed.

This could add latency & delay later events.

6. Observability and Recovery

When a customer says an agent never responded to their approval, what’s the actual answer?

Without logging, there isn’t one. Nobody can say whether the event got sent, whether it arrived, or what happened after. A working system tracks every delivery attempt, the response it got, and each endpoint's current status.

This turns a vague complaint into something checkable, an exact record of when an event went out, what came back, and how many times it was retried. And once whatever broke gets fixed, replay9 sends the same event again, closing the gap instead of leaving it there permanently.

Delivery, security, duplicates, bursts, ordering, visibility (and so on) need someone to solve each of these. You can build them yourself or use existing standards and infrastructure to handle the pieces you don’t want to own.

Let’s look at what a production-ready webhook needs to provide…

§

We are in the middle of the agentic AI era, please take advantage of it

Agents can crush your app with compounding latency across workflows.

Your customers expect fast, efficient agent workflows. But if your agents keep polling for updates, you risk losing deals. Webhooks make workflows event-driven, but building reliable webhook infrastructure gets complicated fast.

If your homegrown webhooks have you waking up at 4 AM to fix failed deliveries, there’s a better way.

Svix handles retries, idempotency, security & compliance for you.

Qualified startups get $12,000 in free credits.

Get started for free »

§

What Real-Time AI Agents Need to Stay Reliable

You still need to handle all these problems when you put a real-time agent into production.

You need a consistent way to identify events, verify their source, and make sure old requests can’t be reused.

This is where standard webhooks help. It gives you a common way to handle the security and verification side of webhook delivery, instead of having to define your own format and rules.

Let’s start with how a webhook gets identified & verified...

Webhook Identity and Signatures

Every webhook carries a unique message ID, a timestamp showing when it was sent, and a signature that verifies its authenticity.

The signature itself uses HMAC-SHA25610.

It runs over the ID, timestamp, and raw payload11 together, using a secret known only to the sender and receiver. If you change even one character in the payload, then the signature no longer matches.

A receiver uses the same verification logic for any provider following the standard.

For an agent workflow, this means the application could verify an event before allowing it to trigger another action.

Replay Protection and Key Rotation

A valid signature proves a request came from the sender…yet it doesn’t prove the request is new…

Without a timestamp check, someone who captures a real request could resend it days later, and the signature would still pass. The receiver could enforce an acceptable timestamp window12. This ensures a captured request eventually stops being usable.

Also secrets need to change occasionally, and a receiver still verifying against the old one shouldn’t break during the transition. The spec supports multiple signatures during secret rotation. This lets a receiver verify requests using either the old/new secret during the rotation window.

For an agent workflow, this matters when an event could trigger an action.

An old signed request shouldn’t be accepted as a new event just because its signature is still valid.

Standardized Webhook Format

A provider following the standard uses the same headers & signing format.

Consumers could use one verification implementation across providers.

This is what makes interoperability13 possible. A team building a receiver can use one verification path for events from different providers. There is no need to write custom logic for each one.

Following the spec doesn’t remove the need for infrastructure behind it, though. Something still has to run the queues, retries, and delivery logic day to day.

Let’s look at how Svix puts them into practice & handles the infrastructure behind real-time AI agent events...

§

Svix: Platform That Connects AI Agents to Outside World

Every piece covered so far, including retries, signing, replay, ordering, and observability, has to actually run somewhere. None of it is optional once real customers depend on your agent reacting to real events.

Svix is built to be this somewhere.

Your application decides when something happened & creates the event.

Svix handles delivery, including retries, security, and everything needed to get it to the right place reliably.

How Svix Models Webhooks

Svix tracks these for each webhook it sends14:

• Application: customer/tenant on your platform.

• Endpoint: actual URL to which you’re sending the event.

• Message: event itself.

• Event type: kind of event, such as a payment clearing or a task finishing, and determines which endpoints receive it.

An application could have many endpoints…One event can reach several destinations, and you can use event-type subscriptions to control which events an endpoint receives.

How an Event Moves Through Svix

There are two ways events move through a real-time agent system15:

1. Outbound events

The agent or application produces an event when something happens.

This could be a tool call finishing or a workflow reaching a new state. The application sends that event to Svix through the API, and Svix delivers it to the configured downstream endpoints.

2. Inbound events

Something happens in an external system, and it sends an event back to the application.

Svix Ingest16 receives incoming webhooks. The application then processes the event and decides what the agent should do next. This could mean starting a new workflow or continuing one that was waiting for that event.

In both cases, the delivery happens separately from the agent’s main workflow.

The application doesn’t need to keep a connection open while the event is being delivered. It can keep the workflow state & handle the event when it arrives.

Queues, Workers, and Storage

Behind the scenes, events go into a queue17 before they're delivered.

Once Svix accepts the event, separate workers handle delivery instead of making your application wait for it to finish.

The message & its delivery state are stored along the way. This means a temporary problem at the destination doesn’t make the event disappear. You can retry or recover the delivery without keeping the original request open.

For an agent workflow that suddenly produces a hundred events at once, this matters.

Your application can hand those events to Svix without waiting for every downstream request to finish, while the delivery system works through them independently.

How to implement Svix

There are a couple of ways to get events into Svix18:

The simplest approach is to send them directly from your application whenever something happens. If you already have your own event system and/or queue, you don’t have to replace it. You could have the system forward the events to Svix when they’re ready to be delivered.

From there, you can create the applications, endpoints, and event types you need through the Svix API. The application represents the customer/tenant; endpoints define where their events should go, and event types describe the kinds of events they could receive.

Also you could give customers control over their own webhook setup through the Svix Application Portal. So they manage their endpoints and inspect delivery activity themselves, rather than coming back to your team every time something needs to change.

Reliable Delivery

The normal delivery path is straightforward when the destination is “healthy”…The harder part happens when it is not…

An endpoint could be down during a deployment, take too long to respond, or start returning errors for a while. Svix retries failed deliveries with backoff, giving the endpoint some time to recover instead of repeatedly sending requests at full speed.

If an endpoint keeps failing after retries are exhausted, you can recover the delivery once the underlying problem gets fixed.

Security and Outbound Delivery

Every event Svix sends gets signed…

So the receiving application could verify the webhook actually came from the expected sender & has not been changed in transit. Also Svix includes a timestamp with the request. This could be checked along with the signature to prevent an old, valid request from being captured & replayed later.

Signing secrets need to be changed from time to time as well.

Svix supports secret rotation by signing deliveries with both the old and new secret for 24 hours, thus giving endpoints time to switch to the new secret without interrupting delivery.

Also there is a risk on the other side of the connection.

Svix is making outbound requests to URLs provided by your customers, so those URLs can’t simply be trusted to point wherever they claim. Svix includes protections against server-side request forgery (SSRF19).

This helps prevent a webhook endpoint from reaching internal services or other destinations it shouldn’t be access20.

Observability and Scaling

When a delivery fails, knowing that it failed is only the first part.

You also need to see what was sent, when it was sent, what the endpoint returned, and whether another attempt was made. Svix keeps a record of delivery attempts and their results, making it easier to see which endpoints are working & which ones need attention.

As the number of events & endpoints grow, some systems also need more control over how those events are delivered.

Svix supports ordered delivery when event order matters, throttling when a particular endpoint shouldn’t receive events too quickly, and payload transformations when different consumers need the same event in a different format.

Plus, it supports multi-region delivery for setups with regional or data residency21 requirements. These aren’t things every agent needs, but they become useful when the same event infrastructure has to serve larger or more distributed systems22.

§

§

How Realtime AI Agents Handle Events and Actions

Let’s look at what happens on the agent’s side once one of these events arrives…

Tool Calling and Asynchronous Actions

The model does not actually run the tool; your application does23.

It decides which one to call & what to pass into it. Most of the time, the result comes back before the model finishes forming its next sentence.

It isn’t always this quick, though…

Waiting on a slow third-party API or generating something heavy like a video can take minutes instead of seconds. And the application cannot keep the connection open for the entire process.

Once the result is ready, a completion event can tell the application to continue the workflow. The application can use the workflow/run ID stored with the agent’s state to find the right workflow and resume where it left off.

This event still needs reliable delivery to arrive successfully.

It uses the same durable delivery and retry logic covered earlier, but the trigger is a finished tool call instead of an external webhook.

Agent Workflows and External Events

An agent can also have workflows that depend on something happening outside its own application.

It might start a task and then wait for an external event before it can continue. Instead of constantly checking for updates, the agent can wait for that event and continue when it arrives.

For example, a coding agent might be waiting for an update to an issue before it can continue. Once that update arrives, the application can use the event to decide what the agent should do next.

Svix Ingest handles the incoming side of the workflow.

External systems send their webhooks to Ingest, which receives and verifies them before passing them to the application. The application can then use the event to start or continue the agent’s workflow.

As an agent works with more external systems, one place to handle incoming events means you don’t need separate webhook handling for each integration.

Human-in-the-Loop Agents

Some decisions genuinely need a person behind them, not just an agent’s best guess24.

Say an agent drafts a refund & needs someone to sign off before it actually goes out. It sends an event asking for approval and delivers it wherever the approver already works. This could be a Slack channel, an internal dashboard, or whatever’s already set up. The approval might not come for a day.

Once it does, the response comes back as an event too, and the workflow picks up right where it left off. Svix handles the delivery side by retrying failed deliveries and keeping a record of each attempt.

This means an approval sitting for a day doesn’t quietly vanish in transit.

Local and Ephemeral Agents

Not every agent has a public URL sitting around waiting for traffic.

One running on somebody’s laptop, or spun up for a single task and torn down right after, usually has no stable endpoint for a provider to send a webhook to. Setting one up just for this rarely makes sense & sometimes isn’t even possible depending on where the agent runs.

Polling gives the agent another option.

Events arrive through Ingest, but the agent retrieves them when it connects instead of exposing a public webhook endpoint. It reaches out on its own schedule, using an existing outbound connection.

And it pulls whatever is waiting instead of needing somewhere for events to land25.

Designing Events for Agents

Most of what an agent does internally never needs to become an event26.

Whether a task actually finishes or an approval is requested matters more than every small decision along the way.

Naming matters here too:

agent.task.completed tells anyone downstream exactly what happened. Something vague doesn’t, and now whoever’s consuming it is guessing. Svix keeps these event types in a catalog, so people integrating with your application can see which events are available & what they represent.

Keep the payload limited to what a consumer actually needs. Version it if the shape changes, so existing consumers keep working.

All of this can be built by hand. At some point, though, it stops being a quick addition and turns into its own piece of infrastructure to maintain.

Let’s see how much of this infrastructure you actually want to own, and how much makes more sense to buy…

§

Build vs Buy for Realtime Agent Infrastructure

Building a webhook endpoint yourself is not hard when it’s just your own systems talking to each other27.

One service sends a request, another receives it, and if something breaks, you notice & fix it. Yet once a customer is on the other end of this connection, the requirements start to change. Now they’re the ones who notice when a delivery fails, and they have no visibility into why or how to fix it.

A bug you’d normally get to eventually becomes a support ticket showing up today.

Handling a webhook for something internal is a tiny problem.

But handling it for a customer, reliably and at scale, is a much bigger problem. It’s worth being honest about which problem you’re actually solving before picking a path. The decision isn’t really about whether you can write a webhook endpoint.

It’s about how much of the surrounding infrastructure you want to own and maintain.

Build Path

Building your own makes sense in a few specific situations:

• Event volume is low, and only a handful of internal systems ever receive anything.

• Requirements are specialized enough for no general-purpose tool to genuinely fit.

• Reliability concerns, retries, durability, and observability are already handled by infrastructure you’re running for other reasons.

• Webhook infrastructure itself is part of the core product you’re building, not something supporting it.

None of these are edge cases nobody hits…Plenty of people genuinely sit in this exact spot, at least for a while.

When Buying Makes Sense

A simple webhook is enough for a small internal setup.

The need for managed infrastructure comes when those events become part of a customer-facing product & you need more control over delivery:

• Customers receive these webhooks, not just an internal system.

• Endpoints and integrations multiply past what’s easy to track by hand.

• Reliability & observability stop being nice extras and start being expected.

• Engineering time spent maintaining webhook infrastructure is time not spent on your actual product.

As the number of customers and integrations grows, so does the amount of infrastructure around each one.

Each customer has its own endpoint and delivery requirements. Also you need to handle delivery failures & ensure events arrive reliably.

The first webhook endpoint is easy to build.

But the work piles up when you add retries, security, replay, observability, and a customer-facing portal. The challenge grows as you keep all those pieces working together. This cost doesn’t show up on day one. It shows up gradually, as more customers sign up and more reliability work gets pushed down the backlog.

It looks like nice-to-have work and never quite gets done.

Whichever path fits, the underlying job doesn’t change. Something still has to ensure an event actually reaches the agent when it’s supposed to.

§

Closing Thoughts

A real-time agent feels almost ordinary once everything works.

It just seems to know what to do next, picking a workflow back up the moment something changes, with no one having to nudge it along.

But beneath that smoothness runs an entire delivery system quietly in the background. Every piece covered here exists to ensure an event reaches the agent every single time. No one needs to think about the machinery behind it.

At this point, you should have a clear picture of what’s actually holding this together:

• Webhooks are the mechanism, notifying the agent the moment something happens instead of making it check over and over

• Reliable delivery, security, idempotency, ordering, and observability are what turn a single HTTP request into something production can actually depend on

• Standard webhooks give senders and receivers a shared way to identify and verify events, instead of every provider inventing its own format.

• Svix runs this entire delivery layer, so building an agent doesn’t also mean building a second system just to keep it connected to the world around it.

Once you see it laid out this way, a real-time agent stops looking like one clever piece of code… It’s a model wired into a delivery system built specifically to make sure nothing it’s waiting on ever slips through.

Agents will only take on more of this over time.

They will wait for more things to happen outside the conversation. They will also react to more of them with nobody watching. And the more an agent takes on, the more its reliability depends on one thing nobody notices until it breaks.

The events it waits for must actually show up.

§

Make your agents realtime

Slow agent workflows kill the user experience & can cost you deals.

Polling adds latency at every step. Webhooks let your agents react to events as they happen, but building reliable webhook infrastructure means dealing with retries, failed deliveries, security & compliance.

If your homegrown webhooks keep breaking and waking you up at 4 AM, Svix handles the hard parts for you.

Retries, idempotency, security & compliance are built in.

Qualified startups get $12,000 in free credits.

Get started for free

If you find this newsletter valuable, share it with a friend, and subscribe if you haven’t already. There are group discounts, gift options & referral rewards available.

Subscribe now

Want to reach 250K+ tech professionals at scale? 📰

If your company wants to reach 250K+ tech professionals, advertise with me.

Thank you for supporting this newsletter.

You are now 250,001+ readers strong, very close to 251k. Let’s try to get 251k readers by 7 October. Consider sharing this letter with your friends & get rewards.

Y’all are the best.

Share

App → Svix queue → Worker → Customer endpoint

Your app → Svix → Customer’s endpoint

Agent → Start task → Wait → Completion event → Resume agent

Agent → Ask for approval → Human decides → Approval event → Agent continues

External system → Svix Ingest → Event waits → Local agent retrieves it

External system → Public URL on your laptop → Agent

• Share this letter & I’ll send you some rewards for the referrals.

AI agents can write code…

But enterprise software is where things get HARD.

Imagine you ask an AI agent to move task notifications to a new system while preserving users’ existing notification preferences.

The AI agent writes code, runs tests…everything “passes”.

But a user who turned off email notifications now receives an email when someone assigns them a task.

The code passes tests, but it does NOT behave the way you want.

A simple fix is to give the AI agent1 a better prompt:

- Keep current email behavior intact.
- Preserve existing notification preferences.
- Do not change unrelated notification flows.

But a prompt has its limitations…

As the codebase gets bigger & agent takes on extra work, it creates 3 new serious problems:

1. Understand the existing system

A prompt describes what you want.

But it does NOT automatically explain how an existing system works. The agent must understand the architecture, dependencies, business logic, feature flags, APIs, data models & old behavior in the codebase (across repos).

Without this context, the agent would make a locally “correct change” that breaks something elsewhere…

2. Build the right change

If you add extra prompt instructions, the request becomes extremely difficult to manage.

The agent must then keep track of scope, constraints, edge cases, success criteria, & dependencies across the entire task. While a vague/incomplete instruction can quietly turn into a wrong implementation.

This could multiply the mistake across many files and/or tasks…

3. Trust autonomous work

Agent autonomy = more work you trust it with.

The agent would read more code, call more tools2, change more files, and run longer workflows. So you need controls on what it can access, what it can change, where it runs, how its output gets checked, and where a human must approve the work.

Yet a “longer prompt” does NOT solve any of these problems!…

§

Spec-Driven Development

Spec-Driven Development (SDD) is a software development approach in which a specification defines intended behavior and guides planning, implementation & validation.

i.e., you define “expected” behavior, scope, constraints & success criteria before the agent starts to run.

Enterprise Agentic Software Development Lifecycle (Agentic SDLC) needs the exact same foundation:

• Full system context,

• Clear specification,

• Validation against intended result.

I’ll use Blitzy as the case study for this newsletter…

It’s an autonomous software development platform for the enterprise that turns what you want to build into a structured plan, then autonomously executes & validates it.

i.e., you provide the requirements and intent, then review key decisions.

Blitzy combines them with codebase context, creates the plan, executes the work with AI agents & validates the result.

Let’s dive in!

How Specification Guides Development

Here are 3 roles3 a specification plays in the software development lifecycle:

• Spec-first: You write the specification before implementation begins, so you know what to build.

• Spec-anchored: You keep the specification connected to the software after implementation so it can guide future changes.

• Spec-as-source: You treat the specification as the human-edited source from which you can generate and/or regenerate downstream implementation artifacts (plans, tasks, tests).

Now let’s see how SDD4 applies to our notification system…

You want to move task notifications to a new notification system. The request defines the goal, but the expected behavior still needs to be explicit.

So you define a specification:

• Objective: Move task notifications to a new system.

• Scope: Move task-assignment & task comment notifications. But leave billing notifications unchanged.

• Requirements: Preserve existing notification preferences & delivery timing. And continue using the existing email service.

• Expected behavior: Check the user’s preferences before sending notifications. If email is enabled, queue an email through the new delivery path.

• Edge cases: Define what happens if delivery fails/assignee no longer belongs to the project.

• Acceptance criteria: Task notifications use the new system while existing user preferences continue to work correctly.

Blitzy combines these requirements with its understanding5 of the application across repositories.

Then it uses requirements & codebase context to create an Agent Action Plan (AAP6).

(We’ll learn how AAP works later…)

Project Rules vs Generation Prompts

A specification defines the change itself.

Yet agents need project-level instructions7 on how to do the work…

Blitzy separates these concerns through Rules & Generation prompts:

• Rules: Project-wide instructions that apply whenever the agent works on the codebase, such as code style, testing & security requirements.

• Generation prompt: Instructions for a specific change, such as its objective, scope, constraints, technical requirements & success criteria.

Here’s an example8:

Rule:
Every new piece of business logic must include tests.

Generation prompt:
Move task notifications to the new system while preserving
existing notification behavior and user preferences.

Requirements Clarification

A BAD assumption gets exponentially expensive the further it travels…

An unclear requirement is easier to fix while the specification is still being edited. Otherwise, the agent would fill those gaps with “assumptions” and carry them into the plan & implementation.

So you should look for these details:

• Who will use the feature,

• How system should handle failures,

• What falls inside & outside the scope,

• Which existing services should change,

• What must remain backward compatible.

This lets you separate unknowns from assumptions:

• Unknown: You’re missing information to make a decision.

• Assumption: You make a guess because information is missing.

Here’s a simple clarification process for you to follow:

1. Draft the specification.

2. Identify missing/ambiguous requirements.

3. Resolve & update the specification.

4. Review the updated specification.

This process gives the agent clear requirements before it creates the implementation plan…

§

§

1. Understand the Existing System

The agent must first understand how the existing system works…

i.e., it needs the correct “system context9”.

This is where Blitzy’s Dynamic Knowledge Graph becomes helpful.

It gives agents a global, dynamic understanding of the application across repositories… this includes architecture, dependencies & code relationships.

Here’s how it works:

Full System Context

Imagine the specification says:

Task assignment should generate an in-app notification
for the assignee, unless the user has disabled that
notification type in their preferences.

So the agent needs to find:

• Which components generate notifications

• Where the system stores user preferences

• Where the system handles task assignments

• What depends on the current notification flow

• Which services & APIs connect these components

… before changing the code.

And this becomes extremely HARD in large, complex existing10 codebases with many repos. Relevant logic could span services, background jobs, event listeners, feature flags, data models & outdated documentation.

A larger context window is NOT enough… AI agent needs to identify & retrieve parts of the codebase relevant to the requested change.

Here’s how Blitzy builds system-level understanding to solve this problem:

1. Analyze codebase: Blitzy analyzes code across multiple repositories to identify architecture, code patterns, terminology & dependencies. (You can also exclude files & directories with .blitzyignore.)

2. Build a dynamic knowledge graph: Blitzy maps relationships across the codebase (relationships that span repos), including control flow, call graphs, inheritance hierarchies & module dependencies.

3. Generate a Technical Specification11: Blitzy creates a current view of the codebase. This includes its architecture, dependencies, patterns & domain terminology.

The result is a global, dynamic understanding of the application that Blitzy’s agents can reference when planning, executing & validating changes…

Code Relationships

Retrieval-Augmented Generation (RAG) retrieves relevant information & adds it to the model’s context.

In a codebase, finding relevant files & components is useful, but the agent also needs to understand how they connect!..

Consider the notification system that sends an email when someone gets assigned a task… Retrieval would find12:

• Preferences

• TaskAssignment

• NotificationService

You could represent their relationships using a dynamic knowledge graph13:

Dynamic Knowledge graph14 maps code components & relationships such as calls, dependencies, inheritance, and control flow.

Let’s look at how GraphRAG uses these relationships to retrieve context…

Graph Retrieval-Augmented Generation

“Relationships” become extremely useful when a single change affects several parts of the codebase.

Graph Retrieval-Augmented Generation (GraphRAG) uses a graph to retrieve context based on the relationships between pieces of information.

A traditional RAG would retrieve the relevant components…Each of them relevant on its own…

But graph-based retrieval shows “how they connect”.

If the agent needs to change what happens when “someone gets assigned a task”, the graph could help it understand15:

• What data it involves

• Which components call each other

• Which events flow through the system

• Which downstream components it affects

In an enterprise codebase with thousands of files across many repositories, finding these connections one file at a time is extremely HARD…

A graph gives the agent a simpler way to navigate the codebase & retrieve the context16 it needs.

Change Impact Analysis

Finding the correct code is only one part of the problem…

The agent must also understand the impact17 of changing it.

Blitzy’s agents traverse Blitzy’s Dynamic Knowledge Graph in 2 directions:

• Forward traversal18: Starts from a function/component & follows the call graph outward to understand what happens downstream.

• Reverse traversal19: Moves backward through the graph to find which parts of the system call depend on a function/component.

If the agent needs to change NotificationService, it can traverse forward to see what NotificationService calls and backward to see what calls/depends on NotificationService.

The key is understanding how the pieces connect, NOT just what exists.

§

§

2. Build the Right Change

The AI agent now needs to determine how to make the changes correctly...

i.e., it needs an “implementation plan”.

Ready for the best part?

Agent Action Plan

One BIG change can be broken into tiny tasks & dependencies.

For notification system migration:

• Update the data model,

• Update the delivery flow,

• Integrate with task events,

… and so on.

Blitzy handles this with an Agent Action Plan20 (AAP).

It uses the generation request & global codebase context from its Dynamic Knowledge Graph to construct the AAP. The Dynamic Knowledge Graph helps map dependencies & analyze the impact of the proposed change before execution begins.

Blitzy then algorithmically sequences the work from the AAP based on those dependencies. This helps agents execute the required work in a “logical order”.

You can then review & edit AAP before code generation begins.

This provides a checkpoint to catch incorrect assumptions, missing dependencies, and/or unnecessary changes before they become code.

Scope and Blast Radius

AAP defines what the agent should & should NOT change.

Blast radius describes which parts of the system a change/failure would affect. For example, changing NotificationService could affect every component that depends on it.

The specification can set scope boundaries21 for:

• Services,

• Infrastructure,

• Files & directories,

• Components that must remain unchanged.

Blitzy supports these boundaries through generation requests & repository exclusions.

This helps keep the implementation within its intended scope…

How Blitzy Agents Execute the Plan

Once AAP is approved, Blitzy’s agents execute the planned work…

For a BIG change, Blitzy uses AAP to sequence work based on “task dependencies”.

Then independent tasks run in parallel, while dependent tasks wait until their prerequisites finish…

Each agent needs the context & tools relevant to its task.

Also, agents need to stay aligned with requirements & codebase context so their changes work together across repositories.

Blitzy’s orchestration layer coordinates22 specialized agents across architecture, implementation, testing, debugging, validation & integration.

The resulting changes then get combined & checked as a complete implementation23.

Verification and Validation

AI agent can implement the plan, pass tests, but still miss a “requirement.”

So the result needs to be checked against technical requirements & intended use:

• Verification: Check if implementation conforms to its defined requirements.

• Validation: Check if resulting software satisfies its intended use & user needs.

Testing can occur at different levels:

• Unit tests: Check individual components.

• Integration tests: Check whether components work together.

• End-to-end tests: Check complete workflows.

• Acceptance tests: Check whether software meets defined requirements & user/business needs.

But automated tests only catch behaviors they actually check…

An extra review is needed to identify missing requirements and/or problems outside test coverage…

Blitzy combines human oversight with automated validation throughout the development process:

• AAP review24: You review implementation plan before code generation begins.

• Automated validation25: Blitzy checks generated code using signals such as builds, tests, benchmarks & CI checks26.

• Project Guide & code review: Blitzy generates a Project Guide27 with each pull request. This shows what got built, which requirements were addressed, and what work remains. PR Review then organizes changes into logical groups, explains why Blitzy made each change alongside the diff & surfaces relevant testing and QA results. This helps you review what changed, why it changed & how it was validated before deciding whether to merge.

i.e., passing tests alone is NOT enough…

The final implementation should be reviewed against the original requirements, scope & expected behavior.

Drift Check

As agents implement changes, code can drift from the original specification…

Blitzy prevents this drift during execution by orchestrating spec-adherence agents that continuously check generated work against approved requirements. It validates outputs at multiple levels, from individual files through end-to-end testing.

This approach helps to find:

• Incomplete work,

• Missing requirements,

• Changes outside agreed scope,

• Remaining work that needs attention.

You make the final merge decision.

You review & test the generated changes, identify remaining gaps, and decide whether to merge the pull request or request extra changes.

§

§

3. Autonomous Work

As AI agents take on more work autonomously, you need to control what they can access, what actions they can take & when humans should step in.

Onward.

Security and Governance

Start with permissions:

• Repository access: Limit which code agent can access.

• Tool access: Limit which tools & operations it can use.

• Network access: Limit which external systems it can reach.

• Least privilege: Give AI agent only the permissions required for its task.

The agent also needs to distinguish trusted instructions from retrieved content…

Code, comments & documentation can contain instruction-like text. So treat retrieved content as data, and NOT trusted instructions. This reduces prompt injection28 risk.

Plus controls must continue after deployment29… Monitoring helps detect unexpected behavior that only appears in production.

Blitzy applies several enterprise controls across its development workflow:

• Data security: Customer data is encrypted in transit & at rest. Customer code is not used for model training/stored directly.

• Isolated execution: Code generation runs in an air-gapped environment with an inbound-only VPC architecture30.

• Deployment options: Blitzy supports cloud, VPC, black-box VPC, on-premises & black-box on-premises deployments.

• Human review: You review the Technical Specification, approve the AAP before code generation & review generated changes before merge.

• QA: Multiple QA agents review generated work before delivery. Customers perform their own QA & security testing before deployment.

• Compliance: Blitzy maintains SOC 2 Type II compliance & ISO 27001 certification31.

These controls put boundaries around what autonomous agents can access, execute & ship…

Human Oversight

Human oversight defines when people need to review/intervene.

Here are two popular approaches:

• Human-in-the-loop: A person must participate at specific decision points before the agent can continue.

• Human-on-the-loop: The agent operates more autonomously while a person supervises the process & intervenes if necessary.

A development workflow can use both…

You review specifications, approve the plan, review security-sensitive changes, & make the final release decision.

Blitzy uses human review at key stages:

• Its Agent Action Plan creates a review point before code generation.

• After generation, automated checks use tests, benchmarks, builds, and other signals to assess the result.

Yet some changes need closer review because mistakes can have massive consequences:

• External system integrations

• Data persistence & migrations

• Regulatory/contractual business logic

• Authentication & session management

These areas benefit from review by engineers, architects and/or domain experts who understand the relevant system & requirements.

§

Blitzy SDD Workflow

You’ve now seen the key parts of the SDD workflow.

But how does Blitzy bring them together into one development process?

Here’s how:

1. Codebase Onboarding: Blitzy analyzes the existing codebase across multiple repositories & builds a dynamic Knowledge Graph of its architecture, dependencies & relationships.

2. Technical Specification: Blitzy generates a living Technical Specification to give you a view of its understanding of the existing codebase across repositories.

3. Requirements: You write a generation prompt to describe what you want to build/change in Blitzy’s Workspace. You define the objective, scope, technical requirements, constraints & success criteria & attach supporting files when needed.

4. Agent Action Plan: Blitzy combines the generation request with global codebase context to create an AAP. This defines the implementation approach, scope, dependencies & required code changes.

5. Plan review: You review, edit & approve AAP before code generation.

6. Multi-agent execution: Blitzy orchestrates specialized AI agents across different models to implement the plan. The agents work on different parts of the project.

7. Validation: Blitzy orchestrates spec-adherence agents during execution & validates outputs at multiple levels, from individual files through end-to-end testing.

8. Review handoff: Blitzy delivers the generated changes with a Project Guide that explains what was built & what work remains.

9. Human review: You review the results, perform any remaining tasks based on the next step guide & decide whether the pull request reaches production.

Thus the requirements remain connected to the implementation throughout the workflow… and help to prevent the original intent from getting lost…

§

Final Thoughts

Blitzy shows what an SDD workflow can enable: AI agents can take on enterprise software changes while you keep control of the outcome.

Blitzy connects these steps through codebase understanding, Agent Action Plans, multi-agent execution, validation & review.

You can reuse these ideas to build AI agents…

But implementing them at enterprise scale is hard. This is why I loved using Blitzy32, and you should try it too.

Thank you for reading.

– Neo

If you find this newsletter valuable, share it with a friend and subscribe if you haven’t already. There are group discounts, gift options, and referral rewards available.

Subscribe now

Want to reach 250K+ tech professionals at scale? 📰

If your company wants to reach 250K+ tech professionals, advertise with me.

Thank you for supporting this newsletter.

You are now 250,001+ readers strong, very close to 251k. Let’s try to get 251k readers by 29 September. Consider sharing this letter with your friends and get rewards.

Y’all are the best.

Share

References

• Blitzy - Introduction - Blitzy Documentation

• Blitzy - Tech Spec Review - Blitzy Documentation

• Blitzy - AAP Review - Blitzy Documentation

• Lewis, Patrick, et al. - Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks - arXiv:2005.11401

• GitHub Spec Kit - Spec-Driven Development - GitHub

• GitHub Spec Kit - Reference - GitHub Pages

• Microsoft Research - GraphRAG: Unlocking LLM Discovery on Narrative Private Data - Microsoft

• OWASP GenAI Security Project - OWASP Top 10 for LLM Applications 2025 - OWASP

• OWASP GenAI Security Project - OWASP Top 10 for Agentic Applications 2026 - OWASP

• Blitzy - Security - Trusted AI Software - Blitzy

• Blitzy - Generation - Blitzy Documentation

• Blitzy - Validation Checklist - Blitzy Documentation

• Blitzy - Project Guide Review - Blitzy Documentation

• GitHub Spec Kit - Constitution - GitHub

• GitHub Spec Kit - Converge Command - GitHub

• Blitzy - Quickstart - Blitzy Documentation

• Blitzy - System 2 AI Platform: Topping SWE-Bench Verified - Blitzy

• Blitzy - Blitzy Scores a Record 84.95% on SWE-Bench Pro - Blitzy Blog

• CXOTalk - Autonomous Software Development at Enterprise Scale: Inside a 1,000-Developer Pilot with Blitzy - Episode 918

• YouTube - How Agentic SDLC Turned a 5-Month Project Into 5 Days with Brian Elliott and Tom Jackson

• Blitzy - Taking the Long and Less-Traveled Road Is the Only Path to Autonomy - Blitzy Blog

• TipRanks - Blitzy Showcases QAD Case Study Highlighting 3x-5x Gains in Software Development Velocity - TipRanks

• Blitzy - Hello World! The Blitzy C Compiler Has Arrived - Blitzy Blog

• Blitzy - A Quick Blitzy Chat: 3 Codebases Take on Prompting - Blitzy Blog

• GitHub Spec Kit - June 2026 Newsletter

• Pearce, Hammond, et al. - Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions - arXiv:2108.09293

• Blitzy - Trust Center

• Share this letter & I’ll send you some rewards for the referrals.

You paste your prompt into Claude Code and wait for the answer.

i.e., one agent gets one task & you wait for it to finish.

But what if your task is more complicated than this?

Let’s say you’re building a feature that needs both backend & frontend code, connected to each other.

Did you know you can assign multiple agents with different instructions & have them work in parallel?

Let’s get started with the fundamentals…

§

[Webinar] How to stop babysitting your agents (Partner)

Agents can generate code. Getting it right for your system, team conventions, and past decisions is the hard part. You end up wasting time and tokens in correction loops.

More MCPs give agents access to information but not understanding. The teams pulling ahead use a context layer to give agents exactly what they need.

Join live on Sep 23 (FREE) to see:

• Where teams get stuck on the AI maturity curve

• How a context layer solves for quality, efficiency, and cost

• Live demo: the same coding task with and without a context layer

Register now

(Thanks to Unblocked for partnering on this newsletter.)

§

I want to introduce Gencay as the guest author.

He has been building with AI since ChatGPT first shipped in 2022.

For years, he built for clients on Upwork: 7,300+ hours of it.

Now he builds in the open and shares every piece of it on LearnAIWithMe, every Monday, Wednesday & Friday.

Not just theory for its own sake. One real build at a time, from idea to running system.

§

Here’s what you’ll find inside this newsletter:

• The two ways to run more than one Claude at once. Subagents and agent teams, and why they solve different kinds of problems.

• What a subagent actually gives you. An isolated worker with its own context window, plus the report-back model that keeps your main session clean and your token cost low.

• Why agent teams cost more, and when that’s worth it. The tradeoffs around token usage, coordination, and when tasks with real handoffs justify a full team instead of parallel subagents.

• What changes when agents need to share the same repo. Git worktrees, a shared CLAUDE.md as the coordination layer, and the enforcement that keeps three agents from stepping on each other’s files.

• Where hooks and the Writer/Reviewer pattern fit in. What they catch, when they fire, and how they compare with relying on CLAUDE.md alone when you need work validated automatically.

§

Subagents, isolated workers

A subagent1 is a second Claude, with its own context window, its own limits, and its own set of instructions2.

It works independently & returns the results. You can create it when you have a side task and/or when you want more than one specified Claude working in parallel.

Let’s say you’ll test your website, and you want to check two different things at once. One subagent tests “AI Labs,” and another tests “AI Academy” Page.

They run in parallel, each in its own window, and you read two short reports instead of one long mess.

Let me show you my website; I improve its pages often.

So after every change, I assign two agents, one to each page, using this prompt.

Create two subagents and run them in parallel.

One opens the AI Labs page in Chrome and reports layout or console problems,
the other does the same for AI Academy, and each reports its issues separately.

After pasting the prompt, it assigns two agents that run in parallel.

They're testing websites as we speak.

They took screenshots, found issues, and solved them.

Here is the final report:

I approved the fixes.

But what if I want these agents to work in sequence and pass their work to one another? Then I should use agent orchestration…

Share

§

Agent Orchestration

Agent orchestration3 turns separate Claudes into a team.

It turns your Claude into the team, spawns teammates, hands out tasks, and collects the results. And here teammates talk to each other, share tasks, and pass work.

Agent teams4 are experimental and off by default.

So if you have not enabled them yet, just tell Claude:

Add CLAUDE_CODE_EXPERIMENTAL_AGENT_TEAMS=1 to my settings.json

if it is not set already, then confirm it is active.

After you paste the prompt, Claude asks for your approval to change the file; once you approve it, you can use Claude agent orchestration.

Let’s create one, and a fun one:

Boids5 is the classic flocking algorithm, where three simple rules make hundreds of dots move like a flock of birds. One teammate researches the rules, the second verifies every claim, and the third builds only from what survives.

Paste this prompt:

Spawn three teammates working in sequence.

The first researches the boids flocking algorithm
and its three rules from the original sources,
and writes the findings to findings.md.

The second waits,
verifies every claim in the file against the sources,
and marks each one as confirmed or wrong.

The third waits for the review,
then builds a single HTML page where hundreds of boids flock live on screen,
using only the confirmed.

Now, Claude assigns the researcher agent first.

After approving it, I clicked “Research boids algorithm,” and it opened a side tab showing this agent’s instructions.

If you click on “Browsed the web, used a tool”, it’ll show you the progress.

Step 1 is finished, and the research file is being created.

And next, the verifier starts working…

Now the builder starts building the app…

Here is the full version of this website, visualizing the boids flocking algorithm:

§

§

Subagents vs orchestration

Read more