• Share this letter & I’ll send you some rewards for the referrals.

You paste your prompt into Claude Code and wait for the answer.

i.e., one agent gets one task & you wait for it to finish.

But what if your task is more complicated than this?

Let’s say you’re building a feature that needs both backend & frontend code, connected to each other.

Did you know you can assign multiple agents with different instructions & have them work in parallel?

Let’s get started with the fundamentals…

§

[Webinar] How to stop babysitting your agents (Partner)

Agents can generate code. Getting it right for your system, team conventions, and past decisions is the hard part. You end up wasting time and tokens in correction loops.

More MCPs give agents access to information but not understanding. The teams pulling ahead use a context layer to give agents exactly what they need.

Join live on Sep 23 (FREE) to see:

• Where teams get stuck on the AI maturity curve

• How a context layer solves for quality, efficiency, and cost

• Live demo: the same coding task with and without a context layer

Register now

(Thanks to Unblocked for partnering on this newsletter.)

§

I want to introduce Gencay as the guest author.

He has been building with AI since ChatGPT first shipped in 2022.

For years, he built for clients on Upwork: 7,300+ hours of it.

Now he builds in the open and shares every piece of it on LearnAIWithMe, every Monday, Wednesday & Friday.

Not just theory for its own sake. One real build at a time, from idea to running system.

§

Here’s what you’ll find inside this newsletter:

• The two ways to run more than one Claude at once. Subagents and agent teams, and why they solve different kinds of problems.

• What a subagent actually gives you. An isolated worker with its own context window, plus the report-back model that keeps your main session clean and your token cost low.

• Why agent teams cost more, and when that’s worth it. The tradeoffs around token usage, coordination, and when tasks with real handoffs justify a full team instead of parallel subagents.

• What changes when agents need to share the same repo. Git worktrees, a shared CLAUDE.md as the coordination layer, and the enforcement that keeps three agents from stepping on each other’s files.

• Where hooks and the Writer/Reviewer pattern fit in. What they catch, when they fire, and how they compare with relying on CLAUDE.md alone when you need work validated automatically.

§

Subagents, isolated workers

A subagent1 is a second Claude, with its own context window, its own limits, and its own set of instructions2.

It works independently & returns the results. You can create it when you have a side task and/or when you want more than one specified Claude working in parallel.

Let’s say you’ll test your website, and you want to check two different things at once. One subagent tests “AI Labs,” and another tests “AI Academy” Page.

They run in parallel, each in its own window, and you read two short reports instead of one long mess.

Let me show you my website; I improve its pages often.

So after every change, I assign two agents, one to each page, using this prompt.

Create two subagents and run them in parallel.

One opens the AI Labs page in Chrome and reports layout or console problems,
the other does the same for AI Academy, and each reports its issues separately.

After pasting the prompt, it assigns two agents that run in parallel.

They're testing websites as we speak.

They took screenshots, found issues, and solved them.

Here is the final report:

I approved the fixes.

But what if I want these agents to work in sequence and pass their work to one another? Then I should use agent orchestration…

Share

§

Agent Orchestration

Agent orchestration3 turns separate Claudes into a team.

It turns your Claude into the team, spawns teammates, hands out tasks, and collects the results. And here teammates talk to each other, share tasks, and pass work.

Agent teams4 are experimental and off by default.

So if you have not enabled them yet, just tell Claude:

Add CLAUDE_CODE_EXPERIMENTAL_AGENT_TEAMS=1 to my settings.json

if it is not set already, then confirm it is active.

After you paste the prompt, Claude asks for your approval to change the file; once you approve it, you can use Claude agent orchestration.

Let’s create one, and a fun one:

Boids5 is the classic flocking algorithm, where three simple rules make hundreds of dots move like a flock of birds. One teammate researches the rules, the second verifies every claim, and the third builds only from what survives.

Paste this prompt:

Spawn three teammates working in sequence.

The first researches the boids flocking algorithm
and its three rules from the original sources,
and writes the findings to findings.md.

The second waits,
verifies every claim in the file against the sources,
and marks each one as confirmed or wrong.

The third waits for the review,
then builds a single HTML page where hundreds of boids flock live on screen,
using only the confirmed.

Now, Claude assigns the researcher agent first.

After approving it, I clicked “Research boids algorithm,” and it opened a side tab showing this agent’s instructions.

If you click on “Browsed the web, used a tool”, it’ll show you the progress.

Step 1 is finished, and the research file is being created.

And next, the verifier starts working…

Now the builder starts building the app…

Here is the full version of this website, visualizing the boids flocking algorithm:

§

§

Subagents vs orchestration

Read more

• Share this letter & I’ll send you some rewards for the referrals.

Getting an AI system to work takes about ten minutes with a model API.

Yet knowing whether it still works after the next prompt change, model update, and/or retrieval tweak1 takes months of engineering that most teams never do. A chat assistant has no built-in way to detect that its answers got worse. A RAG pipeline cannot tell you that fixing one edge case broke five others. An agent cannot stop a poisoned document from steering its next action.

The system runs… But whether it runs “correctly” is a separate question.

That question has 4 parts:

1. Can you measure quality repeatably before deployment?

2. Can you stop a bad request before it reaches the model?

3. Can you reconstruct what happened when something goes wrong?

4. Can you prove system meets the obligations2 your deployment carries?

Evaluation, guardrails, security & observability each handle one of those parts.

Onward.

§

I’m happy to partner with Inngest on this newsletter.

A few weeks back, I wrote a deep dive on how to build AI agents that can recover from failures without starting over.

Read my guide to durable AI agents →

Agents, like all of us, often need to recover from failure. But they shouldn’t start from scratch.

Inngest’s durable execution checkpoints your agents every step of the way.

So your agents can pick up where they left off and avoid doubling up on token & compute spend.

Inngest offers a generous free tier and a sleek local dev server for testing.

Build durably with Inngest

§

I want to introduce Louis-François Bouchard as a guest author.

He’s a best-selling author (Building LLMs for Production), the co-founder of Towards AI, and the creator of the YouTube Channel, What’s AI, where he helps people understand AI and learn how to apply it in the real world. Through his development work with clients, content, teaching, AI training programs, and the new Mentorship program at Towards AI Academy, Louis focuses on making AI practical for builders, engineers, and curious learners alike.

At Towards AI, he and his team train AI engineers through courses and mentorship built for every stage, from beginner to advanced. That educational mission and the real-world experience building for his clients are exactly why I wanted him in this newsletter series.

§

Here’s what you’ll find inside this newsletter:

• Why an AI system working is different from knowing it still works. The trust layer around your application, and how evaluation, guardrails, security, and observability work together.

• How to measure AI quality before shipping a change. Golden test sets, grader hierarchies, LLM-as-Judge, regression testing, and the metrics that change across RAG, tools, agents, and multimodal systems.

• How to protect an AI system while it is running. Input and output guardrails, tool permissions, prompt-injection defenses, data poisoning, agent security, and red teaming.

• What you need to record when something goes wrong. End-to-end traces, safe logging, drift detection, alerts, and the feedback loop that turns production failures into future regression tests.

• How to build an evaluation process around your own failures. Error analysis, failure taxonomies, metric stacks, validity gates, and controlled experiments for deciding whether a prompt, model, or retrieval change is ready to ship.

§

Why You Cannot Skip This Layer

You can ship without evaluation, guardrails, or tracing…

But you lose the ability to tell whether the system still works once its prompts, models, data & users start changing…

Here are five properties of these systems that make that guarantee impossible without a dedicated infrastructure:

• The system is non-deterministic. The same input produces different outputs on different runs, so an exact assertion cannot distinguish a valid rephrasing from a regression.

• Its mistakes look like correct answers. A wrong answer arrives as fluently as a right one. Surface quality does not indicate correctness.

• Its regressions are silent. A prompt change that fixes one case can break five others. Without a repeatable suite, this trade gets never reported.

• Its inputs are not all trustworthy. Instructions can arrive inside a retrieved document, a fetched web page, or a tool result and steer the model. There is no complete fix, only layered mitigation.

• Its legal exposure depends on deployment. Duties attach to personal data, generated media, and high-risk decisions based on how the system gets used, not which model powers it.

How much of this layer to build depends on what the “system” decides, not on how many people use it.

An internal meeting-note helper would need a small test set & basic tracing. Yet an internal hiring screen needs much stronger controls, because a wrong answer affects someone’s career.

§

Target System Design

The trust layer has four components, each handling a different stage of the system lifecycle:

• Evaluation grades the system before deployment, answering whether the current version is good enough to ship.

• Guardrails check every request & response at runtime.

• Observability records what happened in production & feeds failures back into the evaluation set.

• Security cuts across all three & defends against external attackers.

For these four to work together, the system needs a golden test set, repeatable offline grading, runtime checks3 with explicit latency budgets, and end-to-end traces that include guardrail decisions.

The evaluation pipeline also runs in CI, so a quality regression blocks the merge in the same way a failing test does…

This newsletter covers the architecture and how to use it…

All four components assume you can already tell a good answer from a bad one.

Share

§

Part 1: Evaluation

Different AI systems need different metrics.

A RAG system needs retrieval and faithfulness checks, while an agent needs tool-use and outcome checks.

A trust layer brings these metrics into one repeatable evaluation stack that answers one question: is the system still good after the latest change?

Why Evaluation Has No Assert Statement

In software testing, a correct output is a known value.

In an AI system, “is this answer good?” depends on who is asking and which standard applies. The same answer can be correct by one rubric4 and unhelpful by another, so evaluation cannot be reduced to an equality check.

The way through is to match the grader to the question, a principle we’ll call the grader hierarchy:

• When a correct answer exists, use a deterministic check: Does the JSON parse against the schema? Does the tool call carry valid arguments? Does the code run? Does the response come back under the latency cap?

• Where quality is subjective, use a rubric or an LLM grader.

• Where the stakes are high, use human review.

The organizing rule across all three is to use the “cheapest” grader that catches the failure you care about.

From Vibe Checks to Numbers

Vibe checking is the informal starting point: try prompts, read outputs, form an impression.

It’s useful for discovering failures, but an impression cannot compare versions and/or isolate which change moved the result.

Public benchmarks help you choose a starting model… But they cannot tell you how well it would perform in your specific domain or edge cases.

A LinkedIn bot5 could correctly tell someone they were a terrible fit for a job. But the user needed to know which skills they lacked & how to improve. A correct answer is still a BAD answer if it does not help the user.

The numbers therefore have to come from a test set built around your own product.

Golden Test Sets That Stay Honest

A golden test set gives you a fixed collection of cases for evaluating your AI system.

Start with 20 to 50 cases from real failures, then grow the set to around 50 to 100.

Also split the cases into three groups:

• In-scope: Questions the system should answer.

• Out-of-scope: Questions the system should refuse.

• Adversarial: Questions designed to break the system.

You can start with synthetic cases generated from your own data.

Then replace them with real user queries as you collect production traffic, with a domain expert defining the expected behavior for each case.

Each test case should include the input, expected behavior and/or key points to check, supporting evidence, and prompt & data versions. Review each case for outdated assumptions and broken references before treating it as ground truth.

Plus a golden test set needs maintenance:

• Refresh stale cases: Add failures & queries from production as the product changes.

• Keep a holdout set: If you repeatedly tune prompts against the same tests, you can overfit to them. Keep some cases hidden from the tuning process.

Think of the holdout set as the final exam: you practice with one set of problems, then test yourself on questions you have not seen before.

Offline and Online Evaluation

Running the golden set before deployment is offline evaluation.

Every version answers the same questions under the same conditions, so the numbers are comparable across versions.

Online evaluation measures the live system on real traffic, using A/B tests, thumbs up & down, regeneration rate, and citation clicks.

Each catches what the other misses:

• Offline catches regressions before users see them, but only on the questions you thought to include.

• Online catches the cases your test set never imagined, but only after real users have hit them.

Production systems run both…

Some online signals need no grader, since a click, a thumbs-down, or a regeneration is the judgment itself. Offline scoring of anything subjective still runs through an LLM grader, and the grader needs calibrating before its scores mean anything.

LLM-as-Judge, Calibrated

A large language model (LLM)-as-Judge uses a model to grade AI outputs against a written rubric. To make those grades useful, choose a clear grading format.

For most evaluations:

• Use pass/fail: Define one clear criterion for each check. Numeric scores such as 1 to 5 are harder to calibrate because the boundary between scores could be unclear.

• Generate the critique first: Ask the judge to explain its assessment before returning the verdict. This reduces the chance the explanation simply justifies a verdict chosen first.

• Use pairwise grading for subjective comparisons: When pass/ fail is too restrictive, ask the judge to compare two answers rather than assign an absolute score.

• Grade outcomes for multi-step tasks: Check each required subgoal separately. Inspect the agent’s trajectory only when a subgoal fails. Different paths could still produce equally valid results.

So choose the format that produces the strongest agreement with your human evaluations.

Three biases need active mitigation:

• Position bias: The judge favors the answer it reads first. So swap the answer order across runs & average the results.

• Self-preference: A model favors outputs that match its own writing style. So use a different model family for the judge than the model you evaluate.

• Verbosity bias: Longer answers score higher than equally good shorter answers. So include length-matched pairs in your calibration set.

Calibration is not optional…

Before you trust the judge, run it on 30 to 50 human-labeled examples. Compare its verdicts with the human labels & compute agreement. Keep the judge blind to which system version produced each output.

Use one portion of the labeled examples to shape the rubric, then validate it on a held-out portion. Thirty to fifty labels are enough to start using the judge, but not enough to treat its scores as ground truth6.

Pick the cheapest model that passes calibration.

For a judge, consistency matters more than raw capability. Judge inference is cheap; collecting reliable human labels is the real cost.

Metric Stack

The metrics you need depend on the components inside your system.

Each component adds its own evaluation layer, so you measure only what applies.

A single model call forms the base layer… Measure:

• Correctness: Did the model produce the right answer?

• Format compliance: Did the output follow the required structure?

Retrieval-augmented generation (RAG) adds retrieval metrics:

• Hit rate: Did at least one relevant chunk appear in the top K retrieved results?

• Mean Reciprocal Rank (MRR): How high did the first relevant result rank? Higher positions receive higher scores.

• Context precision: How much of the retrieved context was relevant?

• Context recall: How much of the relevant information did retrieval find?

• Faithfulness: Did the answer stay supported by the retrieved context?

• Response relevance: Did the answer address the user’s question?

These metrics show whether RAG retrieved the right information, ranked it well, and used it to produce a relevant, supported answer.

Tool use adds another check: did the model call the correct tool with the correct arguments? This is a “deterministic” check, so it belongs in the cheapest tier of the grader hierarchy.

Agents need outcome-first evaluation… So grade the state the agent left behind:

• Did the refund get issued?

• Did the file get written?

• Did the test pass?

Path-based grading penalizes valid alternative solutions and marks down agents that still reach the correct outcome7.

Also use trajectory metrics to diagnose failures:

• Step efficiency: How many steps did the agent need?

• Loop detection: Did the agent repeat the same actions without making progress?

• Tokens per task: How many tokens did the agent spend to complete the task?

These metrics help explain a bad outcome…Yet they should not serve as release gates.

• Media generation adds prompt adherence, identity preservation, artifact rates, and unsafe content rates8.

• Voice systems add first-audio latency & interruption success.

The rule of thumb: start with the base evaluation layer, then add metrics for each component your system uses.

Evaluation as CI

The metric stack defines what to measure.

Continuous integration (CI) determines when to measure it.

Run the evaluation suite like unit tests:

1. Change the prompt/model.

2. Run the same evaluation suite.

3. Generate a scorecard.

4. Block the merge if critical scores fall below your release thresholds.

Use two evaluation suites for different goals:

• Regression suite: Contains cases the system already handles correctly. Its pass rate should stay near 100%. A failed case = new version broke something the previous version handled correctly.

• Capability suite: Contains hard cases near limits of the system’s current abilities. Its pass rate should stay below 100%, so the suite continues to expose weaknesses & measure improvements.

As the system improves, more capability tests pass.

Anthropic calls the point where nearly all tests pass eval saturation. Once a suite approaches this ceiling, add harder tasks so the evaluation continues to reveal meaningful differences between system versions.

Two rules keep the evaluation gate reliable:

1. Build most test cases from observed failures

Twenty cases from real production traces often represent actual system risks better than 100 hypothetical questions written in advance.

Yet there are 2 exceptions:

• Adversarial cases: Tests designed to expose attacks/unsafe behavior.

• Compliance cases: Tests for requirements the system must follow.

Include these before an incident occurs… Waiting for a real security and/or compliance failure is too late…

2. Run each evaluation suite three to five times per change

Model outputs vary between runs, so a two-point drop in one run could be sampling noise.

Block a release when the drop persists across repeated runs rather than reacting to a single fluctuation…

Also repeated runs produce 2 useful reliability metrics:

• pass@k: At least one of k attempts must succeed. Use this when a person reviews the result, and retries are acceptable.

• pass^k9: All k attempts must succeed. τ-bench, Sierra’s benchmark for customer-service agent tasks, introduced this metric. Use it when an agent acts without supervision.

If an unsupervised agent fails 10% of the time, the probability of at least one failure across eight tasks is roughly 57%.

A strong single-run score could therefore hide inconsistencies that make “unsupervised” deployment unreliable…

The evaluation harness measures performance on cases you anticipated & tested.

Guardrails cover another part of the problem by checking every input & output at runtime…

§

§

Part 2: Guardrails

Read more

• Share this letter & I’ll send you some rewards for the referrals.

Some of these are everyday databases, and some are quite specialized. ALL of them are super useful to software engineers building production systems.

Curious to know how many were new to you:

1. Relational,

2. Document,

3. Key value,

4. Wide column,

5. Graph,

6. Time-series,

7. Geospatial,

8. Vector,

9. Search engine,

10. In-memory,

11. Distributed SQL,

12. Embedded,

13. Columnar,

14. Data lakehouse,

15. Object storage,

16. Ledger.

For each, I’ll share:

• What it is & how it works (in simple words),

• A real-world analogy,

• Tradeoffs,

• Why it matters.

Let’s go!

§

Autonomous Software Development for the Enterprise (Partner)

Blitzy is built for large, complex software projects that other coding agents cannot handle: new feature development, large-scale refactors, scaled vulnerability remediation, and undocumented legacy systems.

Blitzy’s Sandbox lets engineers evaluate Blitzy on their own software estate, at their own pace. Eligible organizations can connect real applications, reverse-engineer up to 1 million lines of code, generate up to 25,000 lines of E2E tested code, and surface prioritized security vulnerabilities across their software estate.

Try Blitzy on your codebase

(Thanks to Blitzy for partnering on this newsletter.)

§

1. Relational

A relational database stores data in tables of rows and columns, and links those tables through shared keys. Every row follows the same shape, and keys tie the tables together.

You declare the shape upfront in Structured Query Language (SQL).

So a customer row carries an identifier, a name, and an email. Orders live in their own table and carry the customer identifier, and the database matches the two on that column in an operation called a join.

Transactions group several changes into one unit, so money moves between two accounts completely/both rows stay as they were. This guarantee is atomicity, consistency, isolation, and durability (ACID), and it is why banks and shops still run on these engines fifty years on. PostgreSQL also reaches beyond the model through extensions, adding geography and vector search to the same database.

One schema, one set of rules, and every application connecting to it gets the same answer.

Analogy

A warehouse with a strict intake form.

Every pallet carries the same label fields, so any driver finds any pallet by reading a code. Redesign the form, and you relabel the whole warehouse.

Tradeoff

Relational databases hand you correctness for free, with joins, constraints, and transactions built in.

But the fixed schema resists change, and adding a column to a large table can lock it while the change runs. Plus, write throughput hits a ceiling on one machine; lifting that ceiling means sharding by hand1.

Why it matters

Orders, payments, inventory, users, and any data with real relationships.

PostgreSQL & MySQL cover most of this ground. Pair it with in-memory caching and distributed SQL, which take over once reads and writes outgrow a single box.

2. Document

A document database stores each record as one self-contained document, typically in JavaScript Object Notation (JSON).

Two documents in the same collection2 can “differ”.

One product carries a shoe size, and the next carries a screen resolution & a warranty period. The database keeps each one whole,,, so a single read returns the entire product with every nested field attached and no join runs.

Indexes still apply to any field/path you name, including nested fields and array elements, so queries stay fast as collections grow. MongoDB & Couchbase lead here, and support transactions across several documents.

The freedom lives in the “schema”, and speed still comes from disciplined indexing.

Change the fields on one product and the rest of the collection carries on unchanged.

Analogy

A filing cabinet of case folders.

Each folder holds every page for one case, so you answer any question about it by pulling one folder. Nobody checks whether two folders hold the same kind of pages.

Tradeoff

Document stores let each record carry its own shape… This suits data nobody can pin down in advance.

But copies of the same fact spread across documents and drift apart, and relationships between documents fall to your application code. With the schema left open, malformed records reach production quietly & surface months later.

So validate the shape in the database, since the flexibility you want in year one becomes the mess you inherit in year three.

Why it matters

Catalogs, user profiles, content, and event payloads, where records vary, and you read them whole.

MongoDB and Couchbase are popular picks. Pair it with a search engine for text queries, or move to a key-value store when every read is by identifier anyway.

3. Key-value

A key-value store saves a value under a unique key & returns it in a single lookup.

The “key” is the only way in3.

Ask for one key and the store answers in one hop, with no scan & no query planner in the way. The store hashes4 each key onto one node, so adding nodes splits the keyspace and lifts capacity in a straight line. Plus, many KV stores attach a time-to-live so cached entries expire on their own.

Redis & DynamoDB dominate this space.

Analogy

A coat check.

You hand over a coat and get a numbered ticket… the ticket brings the coat back in seconds. The attendant has no other way to find every brown coat in the room.

Tradeoff

Key-value stores give you the fastest read path & scale by adding nodes.

But every lookup by something other than the key means a full pass over the data, and secondary indexes stay limited/absent. Also one “popular” key sends all its traffic to a single node, so a HOT key overloads one machine while the rest idle.

So design the key naming scheme5 before you write a line of code, because reworking it later means rewriting each caller.

Why it matters

Sessions, shopping carts, feature flags, and rate limit counters, where you know the identifier at read time…

Redis & DynamoDB are the standards. Pair it with a relational database, which holds the truth, while the key-value6 store holds the fast copy.

§

§

4. Wide-column

A wide-column store spreads rows across many nodes & each row holds only the columns it uses.

The primary key does double duty7:

• Its first part (partition key) decides which node stores the row.

• Its second part (clustering key) decides the order of rows inside the partition.

Pick both well & a query reads one partition on one node… this is how wide column stores answer in milliseconds while holding petabytes.

Writes take a fast path too: they append to a commit log & an in-memory table8 before flushing to sorted files on disk. So nothing seeks across the disk to update a row in place.

Get the key right & a read touches only one node… Get it wrong & it touches all of them…

Analogy

A stadium with numbered gates…

Your ticket names the gate, so you walk straight in while sixty thousand people move at once. Arrive without a gate number, and you circle the building.

Tradeoff

Wide-column stores take write volumes9 no single machine survives, and they stay available while nodes fail.

But you model queries before you model data, so a question nobody anticipated needs a second table and a second copy10. Plus, deletes leave markers11 behind until compaction clears them, and repair, compaction, & node replacement are real operational work.

So write down every query you need before you design the key, since the model resists new questions later.

Why it matters

Message history, activity feeds, and device telemetry at write volumes one machine will never take…

Cassandra & HBase lead here. Pair it with a time-series database,,, which gives you the same scale with far less modeling when the data is stamped with a time.

5. Graph

A graph database stores data as nodes & edges between them, then answers questions by walking those edges.

The relationship gets stored12, so following it costs almost nothing.

In a relational database, finding friends of your friends means joining a table to itself once per hop, and each hop multiplies the work. A graph database keeps a direct pointer from every node to its neighbors, so a hop is a pointer follow & depth stops mattering13.

Query languages like Cypher & Gremlin let you describe the shape you want and the engine traces it.

Analogy

An underground map…

You trace the route with your finger without reading a single timetable.

Tradeoff

Graph databases answer questions about connections, and the queries read like the question you asked.

But spreading a graph across machines is HARD, since edges cross partitions & traversal then crosses the network14. Also you need to learn a new query language, plus bulk analytics run slower here than in a columnar store.

So reach for a graph when the connections are the product.

Why it matters

Fraud rings, recommendations, social graphs, permission chains & network topology.

Neo4j & Amazon Neptune are popular picks.

Pair it with a relational database, which holds the records while the graph holds the links between them.

6. Time-series

A time-series database15 stores measurements stamped with time & organizes them by interval.

Data typically arrives in time order & old measurements rarely change afterward.

Each data point contains a timestamp, a value, and a few tags describing where it came from. New measurements get added to the latest chunk instead of updating old data across the disk,,, this makes writes much faster.

Measurements close together often look similar, so time-series databases can compress them heavily. Also retention rules automatically delete old data, while downsampling replaces detailed data with summaries. i.e., you could keep every measurement for a week, but only hourly averages for a year.

Old data shrinks and/or leaves on a schedule… so database stays within planned size.

Analogy

A ship’s logbook.

Each entry goes on the next line with the time beside it & nobody erases yesterday.

Tradeoff

Time-series databases absorb millions of points a second & answer range queries instantly.

But editing/deleting single points fights the design, so corrections are awkward. Tags with unbounded values, such as a user identifier on every reading, explode the index and exhaust memory16.

So keep tag values few & bounded.

Why it matters

Metrics, sensor readings, prices & anything you plot against time.

InfluxDB, TimescaleDB, and Prometheus lead here.

Pair it with a columnar database, which handles long-term analysis once the raw points are rolled up.

§

§

7. Geospatial

Read more

• Share this letter & I’ll send you some rewards for the referrals.

Claude Code is powerful from day one.

It reads your files, writes your code & runs your commands.

But everything it touches lives on your machine. And the real work rarely stays there. Your issues sit on GitHub, your tasks wait on Slack, and your data lives in Postgres.

This is why the natural next step is to reach them. And Claude Code gives you four doors into them. MCP servers, CLIs, plugins, and slash commands.

Let’s get started!

§

[Webinar] How to stop babysitting your agents (Partner)

Agents can generate code. Getting it right for your system, team conventions, and past decisions is the hard part. You end up wasting time and tokens in correction loops.

More MCPs give agents access to information but not understanding. The teams pulling ahead use a context layer to give agents exactly what they need.

Join live on Sep 23 (FREE) to see:

• Where teams get stuck on the AI maturity curve

• How a context layer solves for quality, efficiency, and cost

• Live demo: the same coding task with and without a context layer

Register now

(Thanks to Unblocked for partnering on this newsletter.)

§

I want to introduce Gencay as the guest author.

He has been building with AI since ChatGPT first shipped in 2022.

For years, he built for clients on Upwork: 7,300+ hours of it.

Now he builds in the open and shares every piece of it on LearnAIWithMe, every Monday, Wednesday & Friday.

Not just theory for its own sake. One real build at a time, from idea to running system.

§

Here’s what you’ll find inside this newsletter:

• The four ways Claude Code can reach beyond your local machine. MCP servers, CLIs, plugins, and slash commands, and why they solve different kinds of problems.

• What MCP actually gives Claude Code access to. External apps, services, and data sources, plus the security risk that appears when you connect untrusted tools.

• Why CLIs can sometimes be a better choice than MCP. The tradeoffs around setup, token usage, reliability, and when a direct command-line connection makes more sense.

• What changes when Claude Code can use your browser. Browser automation, authenticated sessions, and a self-correcting workflow that can inspect a page, catch visual bugs, and keep iterating.

• Where plugins and slash commands fit into the picture. What they bundle, what they can trigger, and how they compare with MCPs and CLIs when you need to choose the right connection method.

§

What MCP actually is

Model Context Protocol (MCP1) lets your AI (ChatGPT, Claude) connect to applications like Notion and/or Google Docs.

These tools make your AI feel like an assistant: instead of just answering you, it can affect the real world through connections.

Let me show you how to install your first MCP server…

§

Install your first MCP server

WARNING: Untrusted MCP servers & scraped web content can both carry prompt injection2, hidden instructions aimed at your agent.

Install only servers you trust, keep Manual mode on & read every approval before you click “yes”.

Once you trust the source:

1. Open your Claude app

2. Click “+”

3. Click “Connectors”

4. Click “Add connector”

5. Click “Browse Connectors”

Next, choose from the list.

Most people connect Google Drive & Google Docs, which are very useful.

But let’s try something different…

I connected with Firecrawl3.

To connect with it, you need an account.

You can open it here.

(It has 1000 free credits each month.)

You’d think, why would I need a scraper when Claude can already open a webpage?

But Firecrawl does more than visit it. It crawls the entire page with all its elements.

Let’s test it…

Paste this prompt:

Use Firecrawl to extract everything from this page:
https://www.tesla.com/model3

Study its sections, content hierarchy, headlines, feature blocks, and calls to action.
Then use Claude Code to build a similar one-page landing page for an AI product,
named Grammar Doctor, with original branding, copy, and visuals.

By the way, Grammar Doctor is a grammar-fixing skill I built, so it deserves a landing page.

After pasting, it asks for my approval because the session runs in Manual mode, and every action waits for a yes.

As you can see from the screenshot above, it started using Firecrawl, and that’s how it’ll scrape every element of the website.

And it saved the output inside tesla_md.txt

Next, before building, it reads this file.

After a few permission requests to create CSS files & 15 minutes, web page is ready.

Let me show you the website it creates:

If you check Tesla’s website, you’ll see it looks very similar.

You can also try doing this with a regular web search.

But the output quality won't be the same.

Web search cannot scrape every element from the page and rebuild a website that is simple, effective & similar to Tesla’s.

§

§

CLIs, the faster way in

A command-line interface (CLI) lets you connect these tools through your command line, instead of connecting via MCP.

This is the older way, since engineers were wiring tools this way long before MCP existed. But for some use cases, like “unofficial” connections, it came back.

Since Claude Code lives in the terminal, it can easily connect to these apps.

And since MCP servers can consume more tokens and be harder to install, CLIs can offer a faster way in.

Let me show you with NotebookLM4…

It has no official MCP server/CLI, but an unofficial CLI called notebooklm-py exists. And it works quite well.

To install it, paste this prompt:

Install the notebooklm-py command-line interface from

https://github.com/teng-lin/notebooklm-py

and ping me when I need to authenticate.

After you authenticate, you can now connect to your NotebookLM through Claude Code.

I have NotebookLM, created using “Boris Cherny’s 30-minute tips”5 about Claude Code.

Now, let’s use the CLI & turn it into an infographic.

Open your Claude app & paste this prompt:

Here is my Notebook: 

<paste your notebook’s link>

Use the NotebookLM CLI to create a handwritten-style infographic
summarizing these 30 minutes.

After pasting this, it connected to NotebookLM and started creating the infographic.

And now it is ready; let me show you the image:

I did not even visit https://notebooklm.google/ once.

NotebookLM created this image after my prompt inside Claude Code.

When I clicked it in NotebookLM, I saw Claude Code wrote a prompt after my description.

So NotebookLM generated the image like this.

But do you wonder how NotebookLM authentication works?

Claude opens the browser, lets me authenticate my account, and saves everything in a JSON file.

The next time it connects, it uses that file to access my account again.

Claude Code connects to the browser through browser automation6.

Now, let me show you more about how this works…

§

§

Browser automation with /chrome

Read more

One of the best ways to get ahead of most software engineers is to become good at Claude Code…

So today we’re excited to announce that doors are officially open for our brand spanking new newsletter series...

INTRODUCING: Claude Code 24/7

This 5-part newsletter series (running through the end of October) will elevate your software engineering career.

If you’ve ever thought:

“I want to understand Claude Code, but don’t know where to start.”

“I want to master Claude Code so I can get more out of it at work.”

“It’s time. I should learn how to set up subagents, hooks & scheduled automation.”

Then this is for you…

Here’s what you’ll get inside Claude Code 24/7:

• A complete setup, from your first session to a fully automated one.

• Deep dives into how MCP, subagents, agent orchestration & hooks work.

• How real projects handle context, scale, and running without you.

And here’s the best part:

You’ll get 10x the results you currently get with 1/10th of your time, energy & effort.

Subscribe now

§

Autonomous Software Development for the Enterprise (Partner)

Blitzy is built for large, complex software projects that other coding agents cannot handle: new feature development, large-scale refactors, scaled vulnerability remediation, and undocumented legacy systems.

Blitzy’s Sandbox lets engineers evaluate Blitzy on their own software estate, at their own pace. Eligible organizations can connect real applications, reverse-engineer up to 1 million lines of code, generate up to 25,000 lines of E2E tested code, and surface prioritized security vulnerabilities across their software estate.

Try Blitzy on your codebase

(Thanks to Blitzy for partnering on this newsletter.)

§

I want to introduce Gencay as the guest author.

He has been building with AI since ChatGPT first shipped in 2022.

For years, he built for clients on Upwork: 7,300+ hours of it.

Now he builds in the open and shares every piece of it on LearnAIWithMe, every Monday, Wednesday & Friday.

Not just theory for its own sake. One real build at a time, from idea to running system.

§

Claude Code is in every feed right now…

YouTube videos, Substack posts, tutorials, screenshots of an app someone built in a single prompt.

But when I sit down with clients, I keep seeing the same install.

Three commands, maybe one skill, that’s it.

So there’s a massive gap between what Claude Code can do & how most people use it.

Let’s get started!

§

Here’s what you’ll find inside this newsletter:

• What Claude Code is actually doing after you give it a task. The agentic loop behind how it gathers context, takes action, verifies its work, and uses tools to work directly with your codebase.

• The setup mistake that can give Claude far more control than you intended. How permission modes work, when to use Manual, Plan, Auto, or Accept Edits, and why bypassing permissions can become dangerous as your workflow grows.

• How to hand Claude a codebase you’ve never seen before. A practical workflow for mapping an unfamiliar repository before touching the code, plus how effort levels change how much thinking Claude spends on harder tasks.

• How to stop re-explaining your project every time you open Claude Code. What belongs in CLAUDE.md, what should stay out, and how to make your project rules and preferences survive across sessions.

• How to turn something you repeatedly do by hand into a reusable Claude Code skill. You’ll build one from scratch, give it a trigger, test it on real input, teach it from its mistakes, and make those corrections permanent so the workflow improves over time.

§

What Claude Code actually is

Claude Code1 is an agent running in your terminal2 that can do anything you could do from the command line.

When you give Claude Code a task, it collects context, takes action, and verifies the results; this is called an Agentic loop3.

But that doesn’t mean all three stages run every time you give it a task…

Let’s say you ask a question about the codebase.

In this case, gathering context could be enough, while fixing a bug would require all three stages of the agentic loop.

Models

You can change the model during the conversation.

Each model serves a different purpose.

For instance, Fable 5 is the most “intelligent” model, but you should NOT reach for it on every task4.

We’ll return to this in the next parts as we explore subagents…

Tools

An agent takes all actions through tools5.

And tools are what separate Claude Code from a simple LLM.

It reads your files, searches through them, and executes commands.

Sometimes it asks for your approval first, sometimes not, and permission modes decide which. (We will cover those later in this newsletter.)

It can search the web too.

And code intelligence means Claude sees its own mistakes right after it writes them, without running the code.

Now you know how it works.

So let’s install it & run your first session…

§

§

Install it and run your first session

To install Claude Code, you should use the terminal.

If you have never used one, here are the basics.

It takes two steps:

Step 1: Paste one of the following commands.

macOS/Linux:
curl -fsSL https://claude.ai/install.sh | bash

===

Windows(powershell):
irm https://claude.ai/install.ps1 | iex

===

Windows(CMD):
curl -fsSL https://claude.ai/install.cmd -o install.cmd
&& install.cmd && del install.cmd

Step 2: Log in

After Step 1 is finished successfully, paste the following command.

claude

Next, Claude asks you to log in, and you can authenticate with your account/API key.

Also, let’s talk about three more shortcuts before you start.

• Esc interrupts Claude mid-task, so you can redirect it without losing the work.

• Esc twice opens the rewind menu, which restores your code to an earlier point.

• Ctrl+C kills the current input when you change your mind.

Now everything is ready, but the Claude team has also released an app.

For a better experience, you can download it here.

Here’s what it looks like:

Did you notice the mistake I made?

Yes, I bypassed permissions and let Claude do anything without my approval.

It is a serious “mistake”, and it gets worse the longer you work that way.

Let’s explore the permission modes…

§

Permission modes and which one to pick

When you work with Claude, it asks for your approval between tasks.

The permission modes control how often Claude asks.

Press shift+tab or click the mode label to see the list.

At the top of the list, you’ll see your current default.

To change it, click on Settings/Claude Code.

You can toggle “Allow bypass permission mode” on & off.

If you’re still learning, leave this option off.

Here are all five options and a single-sentence explanation for each of them:

• Manual: Always ask before making any changes.

• Accept edits: Automatically accept all file edits.

• Plan: Create a plan before making any changes.

• Auto: Claude decides the permissions.

• Bypass permission: Claude runs everything without asking.

Now you know how much freedom Claude has.

So let’s put it to work on a codebase you have never seen…

§

Read an unfamiliar codebase before you touch it

A Claude project usually holds your skills and a Claude.md file6.

(We’ll go through these later in this newsletter…)

But first, let me show you mine, using VS Code, because it also shows the files.

Before publishing articles for my newsletter, I use my Claude project to test the code in them.

You can see my project name from the Claude app too.

Let’s turn the permission mode from “Manual” to “Plan” by pressing shift + tab.

(I’ve been building different projects and creating separate folders, which has made my codebase a bit messy.)

Let’s explore those files by creating an artifact7.

Here is the prompt to paste:

Read this project and map its structure.

Show every folder and the projects in it.

Return it as an artifact.

After pasting it, Claude maps the project & starts writing the plan.

Remember, we’re in “plan” mode.

After the plan, it asks for your approval.

I clicked on “Accept and auto mode,” so it continues building for me.

Here is the artifact in 6 minutes:

I know my repo, but you don’t.

If I show you the report from top to bottom, you’ll understand what was going on.

So this flow also works on a repository you’ve never opened before.

I pointed it at my own messy folder, but it works the same way with a stranger’s codebase.

Now you get the idea…

But what if the job needs “deeper thinking” than this one did?

Then you need to adjust the effort level.

Let’s explore different levels…

§

Effort levels

Effort levels adjust how much Claude thinks before doing a task.

You set it by clicking here:

Effort is directly related to token usage8.

The default level is High for every model, which is best for most tasks.

If your Claude token limit is low and the task does not require maximum intelligence, you can set it to Medium.

But if the task is critical and you do not care how many tokens it takes, the highest setting is Ultracode.

But what if you don’t want your project to forget your settings & you don’t want to repeat them in every session?

Then you should set up your Claude.md file.

Let me show you how…

§

Your project’s memory

Let’s say you have a client project/personal project.

Each day brings a new challenge, and sometimes you realize that you keep copying and pasting the same prompts and/or repeating the same settings.

Starting from scratch every time is wasted work.

With just one command/prompt, your project can gain “permanent memory” of your preferences.

To test it, I created a new folder named Your First Day with Claude Code.

The official way to create Claude.md is using this command:

/init

This will automatically create a Claude.md in your folder.

Let’s create it together only by prompting…

Create a new folder & open a new Claude Code session inside your Claude app.

Next, copy everything you have read in this newsletter so far.

And paste this prompt:

Turn this into a Claude.md file.
Assume I’m a beginner, add these as instructions,
and follow them whenever I ask you to do something.

Here it is:

Now, whenever you start a new conversation inside this project, the context stays.

i.e., your setup travels with you session after session.

But you must follow a few rules when creating a Claude.md.

Let’s talk about that…

What goes in and what stays out

The CLAUDE.md you create should not run too long/too thin.

Claude reads the entire file every time your project starts, so every line either earns its place/wastes it.

So what earns a place?

• Write down what you would otherwise re-explain.

• Build commands, naming conventions, and the rules you keep repeating.

What stays out matters just as much:

• Skip anything Claude can read from the code itself, like folder layouts and dependency lists, since they only burn context on every session.

• Long procedures stay out, too, because those belong in a skill.

And two habits from the Claude team keep the file healthy.

• Keep it under 200 lines, since longer files get followed less, not more.

• Then run /context9 now & then, and check that your file is listed under memory files. If it is missing there, Claude never saw it.

Now think about the most repetitive task you do… How would it change if you used this?

Claude.md is only half of it.

You can also use Skills alongside Claude.md, which lets you hand each job to a specialist.

Let me show you how…

§

§

Skills

Read more

• Share this letter & I’ll send you some rewards for the referrals.

Onwards ‘n downwards:

Following is the second of a premium 2-part1 newsletter series…

Some of these are foundational, and some are quite advanced. ALL of them are super useful to engineers building production-ready systems.

Curious to know how many were new to you:

26. Fuzz Testing

27. Penetration Testing

28. Injection Attack Testing

29. Encryption/TLS Testing

30. Policy/Compliance Testing

31. Reliability Testing

32. Chaos Testing

33. Circuit Breaker Testing

34. Backpressure Testing

35. Fault Injection Testing

36. Failover Testing

37. Recovery Testing

38. Idempotency Testing

39. Timeout and Retry Testing

40. Caching Testing

41. Data Integrity Testing

42. Data Consistency Testing

43. State Transition Testing

44. UI-Level API Testing

45. Developer Experience Testing

46. Documentation Testing

47. Rate Limiting Testing

48. Backward Compatibility Testing

49. Mutation Testing

50. Mock/Stub Testing

51. Synthetic Monitoring

52. Scenario/Workflow Testing

53. Component Testing

For each, I’ll share:

• What it is and how it works (in simple words)

• A real-world analogy

• Tradeoffs

• Why it matters

Let’s go!

§

Make your agents realtime (Partner)

Agents are killing your apps with compounding latency across workflows.

Your customers want their agent workflows to be efficient, so you’re losing deals if you don’t make them event-driven. Webhooks solve this issue, but implementing them is harder than it looks.

If you homebrewed your webhooks and are getting pinged at 4 in the morning for failed deliveries, this is for you.

Svix handles everything: retries, idempotency, security, and compliance. Qualified startups get $12,000 in free credits.

Get started for free

(Thanks to Svix for partnering on this newsletter.)

§

26. Fuzz Testing

Fuzz testing sends randomized, malformed, unexpected data to API endpoints to discover crashes, memory leaks, unhandled exceptions, and/or security vulnerabilities.

A human does NOT design the inputs. Instead, they’re generated algorithmically, mutated from valid inputs, pulled from known attack pattern libraries.

Tools such as AFL, Burp Suite & Restler automate this process at scale.

Analogy

A locksmith testing a new lock does not just use the correct key.

They jam random objects into the keyhole, apply unusual force, insert keys from other locks, and try every unexpected interaction to find weaknesses the designer did not anticipate.

Tradeoff

Fuzz testing finds bugs that NO human tester would think to look for.

It excels at discovering edge cases in input parsing, deserialization, & memory management. Yet results are often noisy. Many crashes are duplicates/low-severity.

Triaging fuzz results requires effort & expertise.

Why it matters

Buffer overflows, injection vulnerabilities, and crash bugs2 in production often come from inputs no developer imagined.

Fuzz testing imagines them at machine speed. Run fuzz tests continuously against input-parsing endpoints, and treat every crash as a potential security issue.

27. Penetration Testing

Penetration testing is a simulated real-world attack against the API, performed by a skilled security professional/team.

It involves creative, context-aware exploitation: chaining vulnerabilities, exploiting business logic flaws, and using social engineering vectors that tools cannot detect.

The goal is not a checklist…it’s to find out whether a motivated attacker can break in.

Analogy

A museum hires a professional thief to attempt a heist under controlled conditions.

Not to check the alarm sensors one by one, but to see whether someone clever enough can get past them all together.

The thief thinks like a thief,,, not like an alarm technician.

Tradeoff

Penetration testing provides the most realistic assessment of your API’s security posture.

It finds vulnerabilities automated tools miss by combining technical exploitation with creative thinking. Yet it’s expensive, time-consuming, and only as good as the tester.

Plus results are a snapshot…not continuous coverage.

Why it matters

Automated scans find known vulnerability patterns.

Pen testing finds the unknown ones: a business logic flaw that lets an attacker manipulate pricing, or a chained exploit that bypasses auth entirely.

So schedule pen tests before major launches & at least annually for production APIs.

28. Injection Attack Testing

Injection attack testing verifies if the API is immune to attacks when malicious code gets embedded in input data: SQL injection, NoSQL injection, command injection, LDAP injection, and header injection3.

You craft payloads designed to escape the intended input context & execute as code on the server. If any payload succeeds,,, the API has a critical vulnerability.

Analogy

A customs officer checks a package labeled “books” to make sure it contains books and not something dangerous hidden inside…

Injection testing confirms that data labeled as “user input” stays as data & never becomes executable instructions.

Tradeoff

Injection testing catches the most exploitable class of API vulnerabilities.

A single successful injection can expose an entire database or grant shell access. Yet the test surface is wide. Every input field, header, query parameter & path segment is a potential injection point.

Thoroughness requires systematic coverage.

Why it matters

Injection attacks remain among the most common and damaging API exploits.

They are well understood & well documented, and still happen because most people skip it…

So test every input vector with known injection patterns. Use parameterized queries and input sanitization as defenses, then verify those defenses hold4.

29. Encryption/TLS Testing

Encryption & TLS testing verifies if data in transit is protected.

You confirm the API enforces HTTPS, rejects HTTP connections, uses strong TLS versions, and avoids weak cipher suites5. You also verify that certificates are valid, not expired, and correctly chained.

Tools such as testssl.sh, sslyze & nmap automate these checks.

Analogy

A courier service guarantees tamper-proof delivery.

Encryption testing verifies every envelope is sealed, the seals cannot be broken without detection, and no courier is secretly delivering in transparent bags.

Tradeoff

TLS testing catches misconfigurations that leave data exposed in transit.

A single misconfigured endpoint serving HTTP can leak credentials/session tokens. Although tests are straightforward, they’re easy to forget.

Plus, TLS configurations degrade over time as certificates expire & protocols get deprecated.

Why it matters

Unencrypted API traffic is readable by anyone on the network path.

This includes credentials, tokens, PII & business data. Therefore:

• Enforce TLS 1.2+ on every endpoint.

• Automate certificate expiry monitoring.

• Run TLS configuration scans on every deployment.

§

§

30. Policy/Compliance Testing

Policy & compliance testing verifies if the API adheres to regulatory requirements and organizational security policies.

This includes GDPR data handling rules, HIPAA protections for health data, PCI-DSS requirements for payment data, and SOC 26 controls for service organizations.

You test personal data is encrypted, audit logs get generated, data retention policies are enforced & consent mechanisms work correctly.

Analogy

A restaurant does not just cook good food.

It must also pass health inspections: proper food storage temperatures, handwashing stations, pest control records. The food could taste great, but if it fails the inspection, it shuts the doors.

Compliance testing is like a health inspection for your API.

Tradeoff

Compliance testing protects the organization from regulatory fines, legal liability & loss of certifications.

It is non-negotiable for APIs handling sensitive data. Yet the tradeoff is rigidity… Compliance requirements change slowly & are often more prescriptive than practical.

Tests must be updated whenever regulations change.

Why it matters

• GDPR violation could cost up to 4% of global annual revenue.

• HIPAA breach triggers mandatory reporting & potential criminal charges.

i.e., compliance is not optional.

So map every regulatory requirement to a specific test. Automate compliance checks and run them on every release.

31. Reliability Testing

Reliability testing measures how consistently an API performs its intended function over time & under varying conditions.

You are not testing a single response.

Instead, you’re testing whether the API delivers the right response reliably across thousands of requests, network disruptions & partial system failures.

This includes uptime measurement, error rate tracking, and mean time between failures (MTBF7).

Analogy

A light switch that works 99 times out of 100 is not reliable…

You need to know the exact failure rate, the failure pattern, and how quickly the light recovers after a failure.

Reliability testing is counting every flicker.

Tradeoff

Reliability testing gives you hard numbers on system dependability.

These numbers drive SLAs, SLOs8, and architecture decisions. But it requires sustained observation over time, not a single test run.

True reliability data comes from weeks of monitoring, not minutes of testing.

Why it matters

Users tolerate slow APIs.

Yet they do not tolerate unreliable ones. An API that works perfectly 95% of the time & fails randomly 5% of the time will drive users away faster than a consistently slow one.

So define SLOs for error rate & availability… Then measure continuously & alert on degradation.

32. Chaos Testing

Chaos testing deliberately introduces failures into the system to verify if the API handles them gracefully.

You kill server instances, inject network latency, corrupt DNS responses, and fill disks, all while monitoring whether the API continues serving requests/degrades predictably.

Netflix’s Chaos Monkey is the most well-known tool in this space:

It randomly terminates production instances during business hours.

Analogy

A fire drill does not wait for a real fire:

You pull the alarm during a normal workday & observe whether people follow the evacuation plan/panic.

Chaos testing is the fire drill for your API infrastructure.

Tradeoff

Chaos testing reveals resilience gaps that no other testing approach can find.

It builds confidence that the system survives real-world failures, not just theoretical ones. Yet the risk is real. Poorly scoped chaos experiments can cause actual outages.

So start small, in staging, with a clear blast radius.

Why it matters

In distributed systems, failure is not a possibility…it is a certainty.

The question is not whether a service will go down, but whether the system handles it when it does…

Start with simple experiments: kill one instance, add 200ms latency to one dependency. Expand the scope as confidence grows.

§

§

33. Circuit Breaker Testing

Read more

§

Most people think system design is about building the “best” architecture.

It’s not... instead it’s about making the right tradeoffs for a given workload & constraints.

Here are 50 tradeoffs you should understand:

1. Latency vs throughput. Batching many requests together can reduce network, serialization & processing overhead and increase throughput1. Yet the downside is individual requests wait longer while a batch forms. So use batching when throughput matters and individual requests when low latency matters.

2. Latency vs tail latency. A system can have excellent median latency while a tiny number of slow requests create poor p95 and/or p99 latency2. Redundant/hedged requests3 can reduce these long tails, but they consume extra capacity. So use hedged requests when low tail latency matters & normal requests when conserving capacity matters.

3. Compression vs CPU usage. Compressing data reduces network bandwidth & storage requirements. Yet compression and decompression require CPU and could add processing latency4. So use compression when bandwidth & storage matters and uncompressed data when low CPU usage & processing latency matter.

4. Telemetry fidelity vs observability cost. Recording ALL distributed traces gives you information for debugging production issues. Sampling5 reduces processing and storage costs, but you’d miss occasional traces you later need. So use full tracing when maximum debugging visibility matters & use sampling when observability costs matter.

5. Cache freshness vs cache hit rate. Longer TTLs increase cache hits6 and reduce requests to the origin. But they also keep stale data cached longer. So use longer TTLs when a high cache hit rate matters & shorter TTLs when data freshness matters.

6. Local cache vs shared cache. An in-process cache avoids a network request and can provide extremely fast reads. A shared remote cache makes data easier to share across instances, but adds network latency & dependency. So use a local cache when low read latency matters, and a shared cache when many instances need shared cached data.

7. Database indexes vs cheaper writes. Database indexes7 can boost query performance. But they consume storage & add work when rows get inserted, updated, and/or deleted. So use indexes when query performance matters and keep them low when write performance matters.

8. High utilization vs capacity headroom. Running infrastructure close to maximum utilization can reduce cost per unit of work. While extra headroom8 allows the system to better absorb traffic bursts & latency spikes. So use high utilization when cost efficiency matters, and extra headroom when handling traffic bursts matters.

§

• Share this letter & I’ll send you some rewards for the referrals.

§

9. Consistency vs availability during network partitions. Replicas cannot coordinate when requests arrive during a partition. Maintaining strong consistency requires “rejecting” some operations, while remaining available replicas can temporarily diverge9. Prioritize consistency when stale/conflicting data is unacceptable, and prioritize availability when serving requests during partitions matters.

10. Cross-region consistency vs write latency10. Synchronously coordinating data across distant regions provides stronger consistency guarantees. Yet the network round trips between regions increase write and/or transaction latency. So use synchronous cross-region replication when strong consistency matters & asynchronous replication when low write latency matters.

11. Read quorum vs write quorum. Replicated systems contact many replicas during reads/writes11. This approach provides stronger guarantees, but increases latency & use extra resources. Use a larger read quorum when stronger read guarantees matter, and a larger write quorum when stronger write guarantees matter.

12. Synchronous vs asynchronous replication. Synchronous replication waits for another replica before acknowledging a write to reduce the risk of losing acknowledged data. Asynchronous replication12 gives faster writes, but replicas can lag behind. So use synchronous replication when protecting acknowledged writes matters, and asynchronous replication when faster writes matter.

13. Single-region vs multi-region deployment. Keeping data in one region typically lowers write latency13 & has less infrastructure complexity. Multi-region setup improves geographic resilience and brings reads closer to users, but replication becomes expensive & complicated. So use a single region when simplicity and low write latency matter, and multiple regions when geographic resilience matters.

14. Replica reads vs primary reads. Reading from a nearby replica reduces latency and improves read capacity. Yet if the replica is behind the primary, the application would read stale data14. Use local replica reads when read latency matters, and use primary/strongly consistent reads when data freshness matters.

15. Serializable vs weaker transaction isolation. Serializable isolation makes concurrent transactions behave as though they executed serially. Stronger isolation causes more transaction aborts & retries under contention, while weaker isolation allows more anomalies15. Use serializable isolation when preventing concurrency anomalies matters, and use weaker isolation when higher concurrency matters16.

16. Optimistic vs pessimistic concurrency control. Optimistic concurrency allows operations to proceed & check for conflicts later. This works well when conflicts are rare. Pessimistic locking prevents conflicts earlier but can increase blocking17. Use optimistic concurrency when conflicts are rare, and pessimistic concurrency when conflicts are frequent/costly18.

17. Last-write-wins vs concurrent updates. Last-write-wins automatically resolves conflicting values & keeps conflict handling simple. Version tracking & explicit merging preserve more concurrent updates, but require extra metadata and application logic19. Use last-write-wins when simple conflict resolution matters, and versioning & merging when preserving concurrent updates matters20.

18. Normalization vs denormalization. Normalized data reduces duplication and makes updates easier to stay consistent. Denormalization makes reads faster and avoids expensive joins, but you must keep duplicate data synchronized21. Use normalization when reducing duplication & consistency matter, and denormalization when faster reads matter22.

19. Relational databases vs NoSQL. Relational databases23 make it easy to query data in different ways and connect related data across tables. NoSQL databases can scale across many machines for known access patterns, but adding new query patterns can be harder. So use relational databases when flexible queries and relationships matter, and NoSQL when horizontal scaling around known access patterns matters.

20. B-tree storage vs LSM-style storage. B-trees24 organize data so reads are fast, including looking up one value/range of values. LSM trees optimize for heavy writes by writing data efficiently & reorganizing it later, but reads & background compaction can require extra work. Use B-trees when fast point & range reads matter, and LSM trees25 when high write throughput matters.

21. Leveled vs universal LSM compaction. LSM databases periodically merge stored files to clean up & reorganize data. Leveled compaction does more rewrites to keep data neatly organized, thus making reads more efficient. Universal compaction does less rewrites, making writes more efficient, but reads require checking more data. So use leveled compaction when read performance matters & universal compaction when write performance matters26.

22. Range partitioning vs hash partitioning. Range partitioning27 keeps nearby keys together; this is useful for range queries. Hashing spreads writes more evenly across partitions, but sacrifices natural key locality. Use range partitioning when range queries & key locality matter, and hash partitioning when evenly distributing writes matters28.

23. Replication vs sharding. Replication copies the same data to many nodes, thus improving redundancy & read capacity. Sharding divides “different” data across nodes. This increases aggregate storage & write capacity while making routing & cross-shard operations harder29. Use replication when redundancy & read scaling matter, and sharding when storage & write scaling matter.

24. Event sourcing vs state-based persistence. Event sourcing stores every change as an event & rebuilds the current state by replaying those events. State-based storage stores the latest state directly, thus making reads and updates simpler, but it does NOT preserve every historical change30. Use event sourcing when audit history & replay matter, and state-based storage when simpler reads & updates matter31.

25. Vertical vs horizontal scaling. Vertical scaling increases a server’s CPU, memory, and/or storage capacity without distributing the workload across many machines. Horizontal scaling adds more machines & offers better scalability, but requires the system to distribute requests/data across them32. Use vertical scaling when simplicity matters, and horizontal scaling when you need to scale beyond one machine.

26. Stateful vs stateless compute. Keeping state inside an application server makes operations simpler & faster. Yet stateless servers are easy to replace, rebalance, and autoscale because durable state lives elsewhere33. Use stateful compute to simplify operations, and stateless compute when scaling & replacement matter.

27. Monolith vs microservices. A monolith keeps calls, transactions, deployments, and debugging inside a smaller operational boundary. Microservices split the application into independent services that can be deployed and scaled separately, but add network calls, distributed data, & operational complexity. Use a monolith when simplicity matters, and microservices when independent deployment & scaling matter34.

28. Coarse-grained vs fine-grained services. Coarse-grained services handle larger parts of the application, resulting in fewer service boundaries & network calls. Fine-grained services35 handle tiny, focused functions and can be deployed & scaled independently, but require extra network communication & distributed coordination36.

29. Shared database vs database per service. A shared database makes joins & transactions between services straightforward. Giving each service its own database increases autonomy and independent evolution, but makes cross-service queries and transactions hard37. Use a shared database when cross-service joins and transactions matter, and use a database per service when service autonomy matters38.

30. Single model vs CQRS. A single model handles both reads & writes, thus keeping the application simpler. CQRS39 separates read & write models so each can be optimized and scaled independently, but adds infrastructure & synchronization between them. Use a single model40 when simplicity matters, and use CQRS when reads & writes need independent optimization/scaling.

31. Synchronous calls vs asynchronous messaging. Synchronous calls41 provide immediate success/failure and are straightforward for short operations. Queues42 decouple services and absorb traffic bursts, but introduce delayed execution, duplicate handling & workflow-state complexity.

32. Saga43 choreography vs saga orchestration. With choreography, each service reacts to events and triggers the next step without a central controller. With orchestration, a central orchestrator tells each service what to do and tracks the overall workflow. Use choreography for simpler, loosely coupled workflows, and use orchestration for complex workflows44 that need centralized coordination & recovery.

33. Two-Phase Commit (2PC) vs Saga. 2PC coordinates transactions across many services/databases so they either ALL commit or ALL roll back. But coordination can increase latency & reduce availability during failures. A Saga45 breaks the workflow into separate local transactions and uses compensating actions when a later step fails. Use 2PC when “strict” atomicity matters46 and Sagas for long-running distributed workflows where services need more autonomy.

34. Message queue vs event log. A message queue distributes tasks across workers, with each task typically processed by one worker. An event log47 stores events for a retention period, allowing many consumers to read the same events & replay them later. Use message queues48 for distributing background tasks and event logs when replay & many independent consumers matter.

35. At-least-once vs exactly-once processing. At-least-once processing retries messages after failures, so the same message would be processed more than once. Exactly-once processing prevents the same operation from producing duplicate effects, but needs extra coordination, transactions, and/or deduplication49. Use at-least-once when consumers can safely handle duplicates, and exactly-once50 when prevention of duplicate effects matters.

36. Total ordering vs partitioned ordering. Total ordering processes all events in the same order, but limits how much work can run in parallel. Partitioned ordering51 guarantees order only for events with the same partition key, allowing different partitions to process events in parallel. Use total ordering when strict ordering across all events matters, and partitioned ordering when high throughput matters52.

37. Push-based vs pull-based messaging. With push-based messaging, the broker sends messages to consumers as they arrive. With pull-based messaging, consumers request messages from the broker when they are ready53, thus giving them more control over batching & processing rate. Use push when immediate delivery matters, and pull54 when consumers need more control over how fast they process messages.

38. Batch processing vs stream processing55. Batch processing56 collects data & processes it together at scheduled intervals, thus making it efficient for BIG datasets. Stream processing processes data continuously as it arrives, providing lower-latency results but adds unbounded data, state, time, and fault recovery complexity.

39. REST vs GraphQL. REST exposes predefined endpoints that return fixed data structures. GraphQL lets clients request exactly the fields they need through a query57, but adds complexity for query execution, caching, & performance control. Use REST for simpler, resource-based APIs, and GraphQL when clients need flexible data access58.

40. Unary RPC vs streaming RPC. Unary RPC sends one request & receives one response. Streaming RPC keeps the connection open so the client and/or server can continuously send many messages59 but makes load balancing & debugging complex60. Use unary RPC for simple request-response operations, and streaming RPC61 for continuous/real-time communication.

§

§

41. Synchronous APIs vs asynchronous APIs. A synchronous API waits for the operation to finish before returning a result, but consumes connection & timeout budgets62. An asynchronous API starts the work and returns immediately, usually with an ID you can use to check its status later. Yet it requires explicit state for status, completion & cancellation. Use synchronous APIs for quick operations, and asynchronous APIs for long-running operations63 that take too long for a single request.

42. Acknowledgment latency vs durability. A messaging system can confirm a write after fewer replicas store it. This reduces latency but increases the risk of losing the message if the server fails. Waiting for more replicas before confirming the write improves durability but increases latency. Use fewer acknowledgments when low latency matters, and more acknowledgments when stronger durability matters64.

43. Retries vs load amplification. Retries can recover from temporary failures by resending a failed request. But if a service is already overloaded65, retries create even more requests and can worsen the overload. Use retries for temporary failures, and limit retries when a service gets overloaded.

44. Short vs long timeouts. Short timeouts release resources quickly when dependencies are slow. Yet setting them too aggressively can abandon requests that would have completed successfully a moment later. Use short timeouts when fast failure and resource protection matter, and long timeouts66 when operations legitimately need more time.

45. Idempotency vs implementation complexity. Idempotency67 makes retries safer because sending the same request many times produces the same intended effect as sending it once. But implementing it requires extra logic to detect & handle duplicate requests. Use idempotency when requests would be retried, and duplicate actions could cause problems. Use simpler non-idempotent operations when duplicates are harmless/impossible.

46. Aggressive vs conservative circuit breakers. An aggressive circuit breaker stops sending requests to a failing service quickly, protecting it from more traffic but potentially blocking requests during a temporary problem. A conservative circuit breaker68 waits for more failures before stopping requests, reducing unnecessary circuit openings but sending more traffic to a struggling service.

47. Load shedding vs graceful degradation. Load shedding69 rejects requests when a system is overloaded, thus freeing up capacity to handle the remaining requests successfully. Accepting more requests increases the chance of serving everyone, but during overload it can slow the entire system/cause failures. Use load shedding when system stability matters, and accept more requests when enough capacity is available.

48. Deep queues vs backpressure. Deep queues70 store more work when producers send data faster than consumers can process it, thus helping to absorb temporary traffic spikes. But if the queue gets too large, requests can sit there for a long time & become stale. Backpressure slows down/rejects producers when consumers cannot keep up, thus preventing the backlog from growing too large. Use deep queues to absorb temporary traffic spikes and backpressure to prevent large backlogs.

49. Active-active vs active-passive. Active-active runs the application in many regions at the same time, so another region can continue serving traffic if one fails. Active-passive71 keeps a secondary region on standby & switches traffic to it only when the primary region fails. This makes coordination simpler but increases failover time. Use active-active when high availability across regions matters, and active-passive when simpler multi-region operation matters.

50. Low RTO/RPO vs cost & complexity. A lower Recovery Time Objective72 (RTO) means recovering faster after a failure, while lower Recovery Point Objective (RPO) means losing less data. Achieving both requires more replication, backup infrastructure, automation & testing,,, this increases cost & complexity. Use low RTO/RPO when minimizing downtime & data loss matters, and higher RTO/RPO when reducing cost & complexity matters.

I started by telling you system design is about tradeoffs…

After 50 of them, the pattern is clear: every architecture choice gives you something & costs you something.

The skill is knowing which tradeoff your workload can afford.

“There are no solutions. There are only trade-offs.”
- Thomas Sowell

§

If you are serious about system design, you really can’t miss out on the footnotes about these 50 tradeoffs:

Get Instant Access Now

Want to reach 250K+ tech professionals at scale? 📰

If your company wants to reach 250K+ tech professionals, advertise with me.

Thank you for supporting this newsletter.

You are now 250,001+ readers strong, very close to 251k. Let’s try to get 251k readers by 7 September. Consider sharing this letter with your friends and get rewards.

Y’all are the best.

Share

Read more

• Share this letter & I’ll send you some rewards for the referrals.

Some of these are foundational, and some are quite advanced. ALL of them are super useful to engineers building production-ready systems.

Curious to know how many were new to you:

1. Smoke Testing

2. Functional Testing

3. Unit Testing

4. Integration Testing

5. End-to-end testing

6. Regression Testing

7. Acceptance Testing

8. Negative Testing

9. Boundary Testing

10. Validation Testing

11. Exploratory Testing

12. Contract Testing

13. Consumer-Driven Contract Testing

14. API Contract Mocking

15. Load Testing

16. Stress Testing

17. Performance Testing

18. Spike Testing

19. Soak/Endurance Testing

20. Scalability Testing

21. Latency Testing

22. Concurrency Testing

23. Security Testing

24. Authentication Testing

25. Authorization Testing

For each, I’ll share:

• What it is & how it works in simple words

• A real-world analogy

• Tradeoffs

• Why it matters

Let’s go!

§

Make your agents realtime (Partner)

Agents are killing your apps with compounding latency across workflows.

Your customers want their agent workflows to be efficient, so you’re losing deals if you don’t make them event-driven. Webhooks solve this issue, but implementing them is harder than it looks.

If you homebrewed your webhooks and are getting pinged at 4 in the morning for failed deliveries, this is for you.

Svix handles everything: retries, idempotency, security, and compliance. Qualified startups get $12,000 in free credits.

Get started for free

(Thanks to Svix for partnering on this newsletter.)

§

1. Smoke Testing

Smoke testing is the first sanity check you run after a new build/deployment. You pick the most critical endpoints: a health check, a login, a basic read, and confirm the API responds at all.

It runs first in your CI pipeline.

If it fails, everything else stops. No point running 500 detailed tests on a broken build. A passing smoke test only means the API responded, NOT that it responded correctly.

So you cannot rely on smoke tests alone & call it safe.

Analogy

Before a surgeon begins a 6-hour operation, they confirm the patient’s vitals are stable…Not a full diagnosis, but is the patient ready to proceed?

Smoke testing is that vital check before you invest deeper.

Tradeoff

Smoke tests are lightning fast and simple. They give instant feedback on failures before they ripple through a QA pipeline.

But simplicity is also their limitation. They are a first gate, not a full safety net.

Why it matters

Every minute a broken build spends in a QA pipeline wastes someone’s time.

Smoke testing is the cheapest guard you can put at the gate.

Run them first in every CI pipeline. If they fail, stop everything else immediately!

2. Functional Testing

Functional testing holds the API accountable to its specification. For every endpoint, you define what valid input looks like, what the response must contain, and what status code should come back. Then you verify if the API delivers it.

This includes happy paths and unhappy paths.

• What happens when required fields are missing?

• …When the auth token is invalid?

• …When the payload is structurally wrong?

Analogy

Imagine a vending machine:

Functional testing is inserting the right coins, pressing the right button, and confirming the right snack drops.

Then inserting too few coins, pressing a sold-out item, and confirming the machine rejects it cleanly without eating your money.

Tradeoff

Functional tests give you direct, verifiable confidence in your API’s behavior.

They’re the key safety net for catching broken contracts before a consumer sees them. Yet the catch is the maintenance costs. Poorly written functional tests become a tax: fragile, brittle, and constantly breaking on harmless changes.

Why it matters

Consumers build products on top of your API, trusting it behaves as documented.

Functional tests enforce trust internally every time code changes. Assert what truly matters to clients, not every incidental implementation detail.

3. Unit Testing

Unit testing isolates the pieces of your API logic: single handler, validation function, data transformer, and verifies them independently.

No real database, no real network… Just the logic, tested in a controlled environment with mocked dependencies.

Analogy

Before assembling a watch, a craftsman tests each gear wheel individually on a flat surface.

You cannot judge the full watch until you trust the components individually.

Tradeoff

Unit tests are the fastest tests you will ever write.

They run in milliseconds and give instant feedback during development, enabling fearless refactoring. Yet the risk is over-reliance.

Mocks do not always reflect reality, and a codebase with 100% unit test coverage can still fail catastrophically when components interact.

Why it matters

Unit tests form the foundation of the testing pyramid.

They catch logic bugs before they compound into harder-to-debug integration failures.

Use them for all pure logic: validators, transformers, calculators, and business rules.

4. Integration Testing

Integration testing verifies if different parts of the system work correctly “together”.

Instead of testing a single endpoint in isolation, you chain a sequence of API calls that reflect a real workflow: create a resource, read it back, update it, and delete it.

Then you confirm the entire sequence produces correct state. These tests touch realistic versions of your databases, queues, and dependent services.

Analogy

A chef does not just taste each ingredient separately…

Before serving a dish, they cook the full recipe, taste the result, and confirm the combined flavors work together.

Integration testing is that final cook-and-taste step before the plate leaves the kitchen.

Tradeoff

Integration tests catch an entire class of bugs that unit tests are structurally incapable of finding: serialization mismatches, transaction failures, auth token propagation errors.

But they are slower, need more infrastructure, and sensitive to environmental issues. One “flaky” dependency can make the entire suite unreliable.

Why it matters

In distributed systems, the bugs that cause the worst incidents rarely live inside a single function.

They live in the seams between systems. So keep a small, critical integration suite per workflow and run heavier scenarios on a nightly schedule.

5. End-to-End (E2E) Testing

End-to-end testing validates a complete user journey through the entire system, exactly as a real user would experience it.

Unlike integration tests, you hit the “real” API, which calls real services, writes to a real database, and returns a real response.

Analogy

A car manufacturer does not just test the engine on a stand and/or the brakes on a rig.

Before shipping a vehicle, a test driver takes it on the road, through traffic, over speed bumps, at highway speed, to confirm the whole car behaves as designed in the real world.

Tradeoff

E2E tests are the most realistic tests you can run, which makes them the most trustworthy signal before a release.

But they are the most expensive to maintain. Flaky environments, test data management & slow execution times make large E2E suites painful.

Why it matters

Individual services can all pass their own tests while the complete user journey still fails.

E2E tests are the last line of defense before users become your testers. Keep the suite small and focused on the most critical business journeys only.

6. Regression Testing

Regression testing reruns a set of existing tests every time new code gets merged. This confirm nothing previously working has been broken.

It’s less about testing new functionality & more about protecting what already works.

Analogy

A musician who mastered a repertoire runs through it before every performance.

This is not to learn the pieces, but to catch any drift/forgotten fingering that crept in during recent practice of new material.

Tradeoff

Regression tests are the institutional memory of your API.

They compound in value over time, catching the “should not have touched that” changes.

Without discipline, a regression suite becomes slow & unmaintainable.

Why it matters

The most common source of production bugs is not new features…but unintended side effects of changes to existing code.

Run regression tests automatically on every merge request & block merges on failures.

7. Acceptance Testing

Acceptance testing is the final verification the API delivers what was agreed upon before development began.

It tests against business criteria & user requirements, not just technical specifications.

The question is not “does it work?” but “does it do what we promised?”

Analogy

A tailor finishing a bespoke suit does not just check the stitching.

They have the client try it on against the original measurements & requirements.

The technical craft could be perfect, but if the suit does not fit the client’s needs,,, the job is not done.

Tradeoff

Acceptance tests provide the highest level of business confidence of any testing type.

They close the gap between a technical specification & a business requirement.

But they require close collaboration between technical & non-technical stakeholders. Plus, they are too slow and coarse to run on every commit.

Why it matters

Teams can spend weeks building an API that works perfectly from a technical standpoint but completely misses the actual business need…

Use acceptance tests as the final sign-off before a feature ships to production.

§

§

8. Negative Testing

Read more

• Share this letter & I’ll send you some rewards for the referrals.

Engineers still joke about not deploying on Fridays for a reason.

For a long time, deployments were one of the riskiest parts of software delivery. They were mostly manual. Engineers logged into production servers, made the required changes & restarted services.

A tiny mistake could easily cause problems.

If a deployment failed, finding the cause meant retracing every step by hand. There was hardly an audit trail, a rollback plan, or even a clear record of what had changed.

For tiny teams, this was usually enough…

Deployments were infrequent, infrastructure was relatively simple, and a single engineer often understood the entire process from start to finish.

But as software teams grew, release cycles got shorter, and deployment frequency increased. What used to happen once a month started happening weekly, then daily, and eventually many times a day. The more often teams released software, the more opportunities for human error emerged.

Tiny mistakes could bring production down:

A missed command, a forgotten environment variable, or a deployment differing slightly from the previous one was often enough.

Teams reduced this risk through runbooks, deployment checklists, and planned release windows. Those practices helped. Yet they did not solve the underlying problem. The process was still manual, and manual processes become harder to manage as systems and teams grow.

CI/CD emerged as a response to these challenges & became a fundamental part of modern software delivery.

Instead of relying on engineers to repeat the same sequence of steps, CI/CD turns those steps into an automated pipeline. Every change follows the same path. Code gets built, tested, packaged & deployed the same way every time. This reduces the risk of deployment errors and makes releases more predictable.

The goal is simple:

Move changes from a developer’s machine to production quickly and safely.

But CI/CD is not a single tool/single process…It’s a collection of systems working together:

• Source control systems trigger pipelines.

• Build stages create deployable artifacts1.

• Deployment stages promote those artifacts across environments.

• And safe deployments depend on more than automation.

They also require security checks, approval gates, rollback2 mechanisms, and monitoring.

Most engineers interact with CI/CD every day. They push code, wait for checks to pass, and watch deployments complete. What they rarely see is everything happening underneath…

Onward.

§

[Webinar] How to stop babysitting your agents (Partner)

Agents can generate code. Getting it right for your system, team conventions, and past decisions is the hard part. You end up wasting time and tokens in correction loops.

More MCPs give agents access to information, but not understanding. The teams pulling ahead use a context layer to give agents exactly what they need.

Join live on Sep 2 (FREE) to see:

• Where teams get stuck on the AI maturity curve

• How a context layer solves for quality, efficiency, and cost

• Live demo: the same coding task with and without a context layer

Register Now

(Thanks to Unblocked for partnering on this newsletter.)

§

I want to introduce Ayaan as a guest author.

He’s a Certified Kubernetes Administrator (CKA) who writes about Kubernetes, cloud infrastructure, and cloud-native systems. He builds in public and shares practical tips, projects, and lessons learned along the way.

Check out his work & socials:

• Twitter

• Github

§

Here’s what you’ll find inside this newsletter:

• What actually happens after you push code. The full path from a Git commit to the build, test, release, and deployment stages that eventually put your application in front of real users.

• Why the same release can survive staging and still break production. How artifacts, environment parity, deployment strategies, and rollback decisions determine whether a release is repeatable or becomes an incident.

• The machinery underneath every pipeline run. What runners, triggers, registries, approval gates, Pipeline as Code, secrets, and software supply-chain controls are doing while you wait for that green checkmark.

• A successful deployment can still be a failed release. Health checks, smoke tests, latency and error signals, business metrics, and SLOs that reveal what happened after real traffic reached the new version.

• Why mature CI/CD pipelines eventually lead to GitOps. Where traditional pipelines stop, how production drift appears after deployment, and the architectural problem GitOps is designed to solve next.

§

What CI/CD Actually Means

CI/CD is often used as a single term, but it refers to three distinct practices…

Each one solves a “different” problem…Treating them as one concept makes it harder to understand why pipelines are designed the way they are.

Continuous Integration

Continuous Integration (CI) is the practice of frequently merging code changes into a shared branch, often many times a day.

Each merge triggers an automated build & a test suite. If the build breaks/tests fail, the team knows immediately.

Before CI, developers often worked on separate branches for days/weeks before merging. When they finally did, the differences had grown large enough to cause serious conflict. Integration became its own painful phase at the end of development.

This is called merge hell.

CI eliminates this phase by keeping integration “continuous”.

Small, frequent merges mean smaller conflicts. Automated checks catch broken code before it reaches anyone else. The branch stays stable, and the team can always see what is actually working.

Continuous Delivery

Continuous Delivery (CD) extends CI one step further:

It ensures every change passing through the pipeline is in a releasable state. Not eventually releasable but releasable right now.

A passing test suite alone is NOT enough. The code must be built into a deployable artifact. It must also be tested in a production-like environment and pass quality and security checks. Every stage gates the next one. A failure at any point stops the change from moving forward.

The keyword here is delivery, not deployment.

The pipeline brings the change to the production door, but a human still decides when to open it. This final approval step is intentional. Some teams need it for compliance, change management, and/or business coordination.

Share

Continuous Deployment

Continuous Deployment removes the final manual step from the process.

Every change passing through the pipeline goes to production automatically. No human approval is required!

This is the furthest point on the spectrum, and not every team reaches it/needs to. It requires a high degree of confidence in the pipeline itself.

• Tests must be comprehensive.

• Rollback mechanisms must be fast & reliable.

• Monitoring must catch regressions3 quickly after release.

Most organizations operate somewhere between Delivery & Deployment rather than at either extreme.

Teams operating at this level often deploy dozens/hundreds of times a day. The pipeline becomes the gating mechanism entirely.

The distinction between Delivery and Deployment comes down to this final approval step. Delivery keeps a human in the loop before production. Deployment automates the decision entirely. Both approaches are valid.

Now we understand the terminology, let’s look at the anatomy of the pipeline itself…

§

Anatomy of a Pipeline

A pipeline is a series of automated stages that a code change must pass through before it reaches production.

Each stage is a gate. Fail one, and the change stops there. Pass all of them, and you'll have a verified, deployable artifact ready for users.

Most pipelines follow the same fundamental sequence regardless of the tool. The stages have different names across GitHub Actions, GitLab CI, and Jenkins, but the underlying structure is always the same…

Let’s walk through each stage:

1. Source Stage

Everything starts with a commit.

When you push code to a branch, a webhook4 fires and the pipeline starts automatically. No manual trigger needed. This push signals that something has changed & needs to be validated.

Git is the universal standard here.

But the branch strategy your team uses shapes how the pipeline behaves:

• Trunk-based development: Everyone commits directly to a single main branch, often many times a day. Pipelines run on every push. Problems surface fast, but this approach requires strong test coverage and feature flags5 to hide incomplete work.

• Feature branches: Developers work in isolation and merge when ready. The pipeline runs on the branch first, then again after the merge. Safer, but long-lived branches delay integration.

• GitFlow: Long-lived branches for features, releases, and hotfixes. Each type triggers different pipeline behavior. Strong structure, but slower path to production.

2. Build Stage

The build stage turns source code into something deployable.

Dependencies are resolved, code is compiled, and an artifact is produced. This artifact is the immutable output of the build. It could be a Docker image, a JAR file, or a compiled binary. The exact form depends on the stack, but the principle never changes.

Build once, deploy everywhere…

The artifact produced here moves through every environment unchanged. Nothing gets rebuilt for staging, and nothing gets rebuilt for production. Rebuilding introduces drift, and drift means what you tested is no longer what you shipped. The same artifact passing the tests in CI must be the exact artifact running in production.

Build speed matters too…A slow build is one that developers start routing around.

Here are two ways to keep it fast:

• Dependency caching: Stores resolved packages between runs. Dependencies change far less often than source code, so caching them cuts significant time off every build.

• Docker layer caching: Reuses unchanged image layers. If the base image and dependency steps haven’t changed, the build skips straight to copying the new source. The key is to order your Dockerfile from least to most frequently changed.

The build stage ends when a versioned, immutable artifact exists and is ready to be tested…

3. Test Stage

The test stage validates whether the artifact actually works.

Automated tests run against it to catch problems before they reach users. The goal is not to run every possible test. But to run the right tests in the right order so obvious problems fail fast, and expensive checks come later.

The test pyramid describes how to think about this:

• Unit tests sit at the base. They are fast, isolated, and numerous. A suite of a thousand unit tests can be completed in a minute or less. They catch logic errors at the function level and should make up the majority of your test coverage.

• Integration tests sit in the middle. They test how components interact with each other, with services, and with databases. Slower and more expensive to maintain, but they catch problems undetected by unit tests.

• End-to-end tests sit at the top. They simulate real user behavior through the full system. Use them sparingly, focused only on the most critical paths.

Fast feedback is not optional…

A pipeline taking 45 minutes to report a failure is the one developers stop trusting. They push & move on. By the time it reports back, they have lost the context to fix it. Most teams target a feedback window of 10 minutes or less for the core suite.

Here are two techniques to keep test time manageable as the codebase grows:

• Parallel jobs: Splits the suite across different runners executing at the same time. Instead of 500 tests running sequentially on one machine, 100 tests run on five machines simultaneously.

• Test sharding: Partitions the suite so each runner covers a different subset. Combined with parallel runners, this scales well even for large codebases.

Test failures are intentional gates.

They exist to stop bad changes. A pipeline not failing doesn’t mean it’s a healthy pipeline. It just means it is not catching anything…

4. Release Stage

Once the artifact is tested, it needs to be stored and versioned before it can be deployed anywhere… The artifact is pushed to a registry.

Here are some popular registries:

• Docker Hub

• Amazon Elastic Container Registry (ECR)

• Google Artifact Registry

• JFrog Artifactory

The registry is the central store from which every environment pulls.

Without it, there is no consistent source of truth for what gets deployed…

Here are two versioning practices that matter here:

• Semantic versioning6. Major, minor, and patch numbers give each artifact a meaningful identifier. Teams can see at a glance what kind of change they are deploying.

• Immutable tags. A given tag should always point to the same artifact. Never use the latest in production pipelines. latest is a mutable pointer. It moves every time you push. A deployment pinned to latest six months ago would pull a completely different image today. Pin to a specific version tag, or a content digest7.

At the end of this stage, the artifact is deployment-ready… versioned, stored, verified.

But it has not reached any environment yet…

5. Deploy Stage

The deploy stage takes the verified artifact & makes it available to users.

By this point, the artifact has already been built, tested & stored in a registry. The deploy stage takes that artifact and releases it into a target environment.

This stage often includes approval gates, deployment policies, and infrastructure-specific tasks. Some teams deploy automatically…others require manual approval before production changes can proceed.

Deployments can also involve additional work beyond simply starting a new application version:

• Database migrations8 would need to run first.

• Configuration changes would need to be applied.

• Infrastructure resources would need to be updated.

The deploy stage is where the pipeline stops validating the artifact & starts delivering it.

Pipeline Runners

The pipeline defines the work, and runners execute it.

A runner is the compute environment where pipeline jobs run. GitHub Actions runners, GitLab runners, and Jenkins agents all serve this role. When a stage is triggered, the platform schedules its jobs onto available runners. Each job runs in an isolated environment, completes its steps, and reports the result back.

Runners come in two forms:

• Platform-managed runners: These get provisioned by the CI/CD provider. You define the job, and the platform handles the infrastructure. Less control over the environment, but no infrastructure to maintain.

• Self-hosted runners: They run on your own machines. Teams use them when they need specific hardware, internal network access, and/or more control over the build environment. The tradeoff is you have to own the maintenance & scaling.

Every build, every test run, and every deployment runs on a runner somewhere.

Slow/overloaded runners are a common cause of slow pipelines, even when the rest of the system works as expected.

Each stage serves a purpose. Together, they turn a code change into a tested, versioned, and deployable release.

But a pipeline is more than a collection of stages. Teams also need a reliable way to define how those stages work & evolve over time…

§

§

Pipeline as Code

Read more

• Share this letter & I’ll send you some rewards for the referrals.

You’ve rewritten the prompt five times, but the AI model still ignores your format…

It returns clean JSON on one request but “broken” JSON on the next. The tone you asked for doesn’t last either… No matter how you write the instructions, the model won’t follow them "reliably”.

So how do you make the behavior stick?

Getting there takes a series of judgment calls, each one yours to make…

Onward.

§

[Webinar] How to stop babysitting your agents (Partner)

Agents can generate code. Getting it right for your system, team conventions, and past decisions is the hard part. You end up wasting time and tokens in correction loops.

More MCPs give agents access to information, but not understanding. The teams pulling ahead use a context layer to give agents exactly what they need.

Join live on Sep 2 (FREE) to see:

• Where teams get stuck on the AI maturity curve

• How a context layer solves for quality, efficiency, and cost

• Live demo: the same coding task with and without a context layer

Register Now

(Thanks to Unblocked for partnering on this newsletter.)

§

I want to introduce Dimple Sharma as a guest author.

Dimple is an AI engineer. Previously, she worked as a software engineer at Microsoft & Samsung.

At Microsoft, she built security infrastructure for SharePoint Online, serving millions of users. At Samsung, she developed a reinforcement-learning system for real-time video streaming, shipped in Samsung HD Displays.

She also holds two patents & a published research paper.

Today she builds applied AI systems and writes about how they work. She has a knack for taking a complex topic apart until it clicks into a clear mental model, with the precision of an engineer who has built production systems at scale.

You can follow her on:

• LinkedIn

• Substack

§

Here’s what’s inside this newsletter:

• Fine-tuning is probably not where you should start. The decision framework for when fine-tuning makes sense, when prompting/RAG is enough, and red flags that tell you to stop before spending money on training.

• LoRA changed the economics of fine-tuning. Why updating a tiny fraction of a model’s parameters can be enough to specialize an LLM, and how QLoRA pushes hardware requirements even lower.

• SFT, RLHF, DPO, ORPO, KTO, GRPO. The modern fine-tuning landscape, what these techniques are designed to teach a model, and where each one fits.

• Fine-tuning now goes far beyond better text responses. Reasoning, reliable tool use, function calling, and multimodal models can all require specialized training approaches.

• Getting a better benchmark score can hide a worse model. Production failure modes that can appear after training, including overfitting, catastrophic forgetting, and safety degradation.

• The complete path from base model to production. Model selection, dataset preparation, training, evaluation, deployment, monitoring, and continual fine-tuning, including what to watch at each stage.

§

What Fine-Tuning Is

Fine-tuning never starts from scratch…

You take a model that already works & train it a little more on a small, curated dataset of your own examples showing the exact behavior you want. Each example nudges the model’s weights1, the billions of numbers shaping its output. Run enough of them, & the behavior becomes the model’s new default.

It’s a “permanent” physical edit to the model.

But it’s NOT always the best choice…

It’s expensive in data, compute & maintenance. So don’t reach for it by reflex when a sharper prompt/retrieval step would have solved your problem for far less.

Before you train anything, learn fine-tuning’s capabilities & limits. Plus learn how it differs from prompting, retrieval & agents…

What Fine-Tuning Can and Cannot Do

Fine-tuning changes model behavior reliably…

It shapes tone, voice, response style, format compliance (JSON, templates), domain-specific reasoning patterns, response persona, and consistent structured outputs.

Consider a support team’s resolved tickets:

Feed the model thousands of (ticket, ideal response) pairs, and it stops sounding like generic internet text. Instead, it starts responding in the team’s voice, consistently. Escalation paths, tone with a difficult customer, phrasing refined over years--the model learns all of it.

The change lives in the weights, NOT in a system prompt.

Yet adding new factual knowledge is where it becomes “unreliable”…

Forcing new facts through fine-tuning can make the model hallucinate. This is called the Superficial Alignment Hypothesis2. Pretraining builds knowledge; fine-tuning shapes “how” the knowledge comes out by default.

When the gap is knowledge instead of behavior, fine-tuning is often NOT the best fix.

Fine-Tuning vs RAG vs Agents

So which tool a problem needs depends on what’s actually broken…

Imagine your model gives solid answers to some questions but hallucinates on others. It follows your format instructions one moment & ignores them the next.

So what do you fix: prompt, data, model, or workflow?

Think of your LLM as a new hire at your company:

• Prompt engineering is how you give instructions: format, tone, constraints. Like writing your new hire a detailed brief before every assignment. It’s cheap, fast, and often enough. But you can’t instruct someone to know things they were never taught.

• Retrieval-Augmented Generation (RAG) is how you supply knowledge. You pull relevant documents from a database & feed them into the prompt, so the model stays “current” without retraining. Like handing your hire the company’s latest documentation before they answer a client.

• Agents are how you enable action. You give the model access to tools, APIs, and external systems, and then let it decide how to complete a goal. Like giving your hire access to the company’s CRM, email, and databases, then trusting them to pick the right tool.

• Fine-tuning is how you teach behavior. You train the model on curated examples of what you want it to do, and the change lasts. Like sending your hire through a specialized training program--they come back with a skill baked into how they work.

Now notice what actually changed in each case…

Prompt engineering, RAG, and agents work “around” the model: they shape what goes in/what the model can access, but the model itself stays identical. Not a single weight moves.

While fine-tuning “works” on the model.

i.e., the model comes out different from what went in.

• Prompt engineering costs tokens.

• RAG adds retrieval infrastructure: a vector database, an embedding model, a pipeline.

• Agents multiply LLM calls per task and add orchestration overhead.

• Fine-tuning demands GPU compute, curated data, and iteration cycles.

Start with the cheapest option; in production, these often combine…

So should you fine-tune?

Share

§

When (and When Not) to Fine-Tune

Whether to fine-tune comes down to a few clear signals--some say stop, some say go:

Red Flags

• Prompting still works. Try using a detailed system prompt, few-shot examples3, and clear format instructions first. Only then think about fine-tuning.

• Your data changes frequently. Fine-tuning bakes a snapshot into weights. Live data goes stale at deployment; this is what RAG is for.

• You don’t have enough quality data. Quality & representativeness of training data matter more than volume. If your examples don’t cover edge cases, the count doesn’t save you.

• Your requirements keep changing. Fine-tuning cycles take days to weeks. If requirements are still shifting, the spec will have changed by the time your training run finishes.

• You need instant rollback. Fine-tuning encodes behavior. If something goes wrong, there’s no quick patch; fixing it means retraining and redeploying.

Green Flags

• Your structured output keeps breaking. The model handles familiar inputs but breaks on “edge cases” even with explicit formatting instructions. Fine-tuning bakes the format into behavior.

• You need a specific voice/persona. Prompting cannot reliably hold a distinctive tone at scale. Fine-tuning makes it the model’s default, not a request.

• Your inference costs are unsustainable. Fine-tuning a smaller open-source model on your specific task can match the same performance at a fraction of the cost. You pay for generality only when you need it.

• You have a well-defined, narrow task. When the problem is bounded (classification, extraction, structured generation4), a fine-tuned small model consistently outperforms a large general one. The narrower the task, the bigger the advantage.

• Your data can’t leave your infrastructure. GDPR, HIPAA5, and/or internal policy blocks you from sending training data to a third-party API. Self-hosted open-source fine-tuning keeps everything on-premises.

What Fine-Tuning Costs

If the green flags fit, one question remains before you commit: what are you taking on?

Fine-tuning costs money to build & saves money to run.

The build cost is higher than it looks; per-call savings are larger than the GPU bill alone suggests.

The investment pays off in three cases:

• You have high & steady inference volume.

• You’re replacing a costly frontier model API.

• Your data must stay within your infrastructure.

Task fine-tuning carries one more cost not shown above: it can “erase” the model’s built-in safety behaviors. So budget for safety evaluation as part of every training cycle.

First, the groundwork…

§

Foundations You Need First

So you’ve decided to fine-tune… but first you need to know where fine-tuning fits in a model’s lifecycle & which kind of model you start from.

From pretraining to fine-tuning

When you fine-tune a model, you’re changing one thing: numbers inside.

Those numbers are parameters: weights packed into every layer. Your text enters as tokens; attention connects them across context. Each layer transforms the representation using those weights. Fine-tuning updates them on your data6.

Everything else stays fixed…

Think of it as three stages of education:

• Pre-training is twelve years of general schooling: trillions of tokens, language, reasoning.

• Continued Pre-Training (CPT) is a specialty degree: more training on a domain corpus when the base model needs it.

• Fine-tuning is an apprenticeship: a curated dataset teaching the model exactly how to behave in your context.

Each stage builds on the previous one.

Here’s how to think about CPT & fine-tuning:

• CPT is for knowing your domain: raw text at scale, absorbed into the weights.

• Fine-tuning is for acting in your domain: labeled examples teaching specific task behavior, format & style.

CPT demands far more data & compute; fine-tuning is cheaper & faster.

Strong domain systems often sequence both.

Base Models vs Chat Models

Before you fine-tune, pick the right starting point…

• Base model is a raw next-token predictor: it has read everything but held no job. Ask it a question, and it would continue writing the question.

• Chat model (aka instruct) is a base plus supervised fine-tuning (SFT) and alignment. It knows how to respond, follow instructions & converse.

So start from base for full control; and start from chat to preserve instruction-following & specialize on top.

Chat models carry one piece of hidden structure: a chat template.

These models are built around a specific conversation format: control tokens marking the system, user, and assistant roles. The format varies by model family: Llama 3, Mistral, and Gemma each use different tokens.

Use the wrong template & the model loses track of who’s speaking…and responses degrade without warning7.

So how’s it actually done?…

§

Read more

• Share this letter & I’ll send you some rewards for the referrals.

You know that feeling when a job runs for ten minutes, gets closer to the end, and then fails?

You check the logs & one API timed out!

So you run it again…

And now you wait another ten minutes for the “same work” to start all over again.

This is annoying enough for a tiny background job. But imagine the job was a “long-running” AI agent workflow1. It searched through dozens of documents, called an LLM several times, extracted the necessary information, and prepared most of the final answer.

And then all the work is gone because one step failed near the end.

It could get worse…

What if the agent had already sent an email before it failed? Or updated a database? Running the whole workflow again could “repeat” those actions as well.

Plus, an AI agent could call a model, search a database, call an external API, wait for someone to approve something, and then continue later. The longer the workflow gets,,, the more painful it becomes to start over whenever something goes wrong.

So workflow needs to “remember” what has already happened…

If the first five steps succeeded, but the sixth failed, then you want to pick up from step six. (i.e., you don’t want to run the first five steps again!)

This is the problem durable execution is designed to solve…

It tracks progress as the workflow runs, so a failure does NOT have to erase everything before it.

But how does it actually work? What happens when a workflow needs to retry a failed step, wait for a person, or recover after the process running it disappears?

I’ll use Inngest as the case study for this newsletter.

(It’s a durable execution platform for background functions, workflows, and AI agents.)

Onward.

§

What Is an AI Agent

You already know what an AI chatbot looks like…

You probably use it every day to ask questions, write something, summarize a document, and/or help you solve a problem. In a typical interaction, you ask for something & the model gives you an answer. Period.

An AI agent takes this idea one step further…

Instead of stopping after one response, it keeps working toward a goal, deciding what it needs to do next as the work unfolds. Give a research agent a topic, and it can work through the research before coming back with the final answer.

The catch is this work does NOT always finish in one simple request.

It can take several steps, involve external systems, wait for something to happen, and/or fail midway. And once an AI agent starts working like a real workflow, simply retrying the request is NOT enough.

So let’s see what happens when these workflows have to deal with “real” failures & how to make them reliable…

§

Why AI Agents Need Durable Execution

A simple & self-contained background job is “easy” to retry from the start.

Yet a single agent run chains together LLM calls, tool calls, database reads, and external API requests. And any of those can fail on its own.

Let’s look at what actually makes “agents” different…

What Makes Agent Workflows Different

Here’s what:

1. Individual steps are slow and expensive

A single LLM call can take several seconds & cost real money.

Repeating five completed steps to get back to the one that failed is never a rounding error. Instead, wasted time & tokens on work already done.

2. Outputs are not deterministic

LLM outputs can vary between executions.

Rerunning a completed step does not “guarantee” the same result, so restarting from the beginning can change the workflow’s outcome.

3. Some steps create real side effects

If the agent already sent an email, updated a record, and/or charged a card before the failure, restarting the workflow will repeat those actions…

A duplicate charge is NOT a retry problem, but a correctness problem.

4. Some steps wait on people

A workflow pausing for editor approval would stay paused for hours/days.

And holding a worker process open the entire time does not “scale”. A basic job queue is NOT designed to track where a multi-step workflow was waiting & resume it later.

When an Agent Fails Mid-Workflow

Think of a research agent:

It retrieves sources, extracts evidence, generates a draft, waits for editor approval, & publishes the result.

Say the model provider times out during the drafting step:

Retrieval & extraction already finished. In a system with NO memory of progress, the entire workflow must restart. And the completed steps have to rerun for no reason.

So if any “side effects” fired before the failure, those repeat too…

Reliability is not just a property of the model/agent framework. It also depends on the execution layer running underneath them.

Now we understand what makes agent workflows different; let’s look at where the traditional queue & worker model starts to fall short…

§

I’m happy to partner with Inngest on this newsletter.

Agents, like all of us, often need to recover from failure. But they shouldn’t start from scratch.

Inngest’s durable execution checkpoints your agents every step of the way.

So your agents can pick up where they left off and avoid doubling up on token & compute spend.

Inngest offers a generous free tier and a sleek local dev server for testing.

Build durably with Inngest

§

Why Traditional Job Queues Are NOT Enough

The basic idea is simple:

An event creates a job, the job enters a queue, and a worker processes it. If something goes wrong, the job gets retried…

For simple, self-contained work, such as sending an email/resizing an image, retrying the entire job from the beginning is reasonable.

Yet the problem becomes DIFFICULT when the job represents a long-running workflow. Instead of one piece of work, you now have several steps, each depending on results from earlier steps.

AI agent workflows “amplify” this problem2 because a single run could involve many LLM calls, tool calls, external APIs, intermediate results & side effects…

Queue vs Workflow

This is where the difference between a queue & a workflow matters:

• A queue mainly focuses on getting work to a worker & making sure failed work gets another chance.

• A workflow needs to remember what happened along the way, including which steps finished, which ones failed, and what needs to happen next.

BullMQ, Celery & SQS3 give you useful building blocks for queues.

You get workers, retries, rate limiting, job dependencies, and parent-child flows. But once you need checkpointing, resumability, human-in-the-loop coordination, and/or a complete history of every step, you’ve more work to do…

You can build those pieces yourself,,, but now the queue is no longer the whole solution. You’re starting to build a workflow system around it…

So what does it take to make these workflows recover properly?

Let’s take a look…

§

§

What Durable Execution Actually Means

Durable execution is a simple idea with a ton of work happening underneath it…

The system remembers what has already been completed. So when something fails, execution can resume from the last successful step instead of starting from zero.

Let’s dive in!

Checkpointing

A checkpoint records a “completed” unit of work.

When a step finishes successfully, it saves the result. If the process crashes, restarts, or a deployment occurs mid-workflow, the result is still available when execution continues…

The workflow can then reuse the completed result instead of running the same step again.

Each step can also be retried independently, so a later failure does NOT undo the work already completed.

Inngest memoizes4 completed steps in the function’s stored state and skips on later executions.

Resumability

You can resume a workflow after an “interruption” with checkpointing.

The workflow does not need to stay tied to the same process/machine while it runs. If the process disappears, execution can continue using the progress already saved.

This matters even more for long-running tasks…

A workflow could run for hours, pause for an external event, or survive a deployment mid-execution. The workflow keeps its progress even while the process running it changes.

i.e., the workflow is the full job; the process is just what runs it.

Why Retries Are NOT Enough

Retries & checkpoints solve different problems…

A retry reruns a failed task, while a checkpoint remembers completed tasks.

Think of a research agent:

It already retrieved the documents & extracted useful information. Then times out on the drafting step …

With a retry, the whole workflow must run again from the beginning, repeating the retrieval & extraction work.

With durable execution, those completed steps get saved. And their results can be reused, so only the failed drafting step needs to run again.

For an agent workflow, this can save a lot of time & reduce LLM usage costs. This is the difference between retrying a job & recovering a workflow.

Now we understand what durable execution means…But what does it take to build it yourself?

Let’s keep going!

§

How to Build Durable Execution Yourself

You need a way to track progress & recover from failures.

The workflow must handle retries and long waits without causing duplicate actions. Also, it needs traffic controls & history to explain what happened.

Let’s get into it!

1. Persisting Workflow State

You need a place to store the “workflow’s progress”.

Plus, you need to save the “result” outside the process running the workflow when a step finishes. Otherwise, a crash/deployment can take the progress with it.

You also need to connect each result to the right workflow & step, so execution knows where to continue later…

2. Retries and Recovery

Once progress is persisted, the next problem is handling failures…

Some failures are worth retrying, while others are not. So you need retry rules, backoff between attempts, and a defined path once those attempts run out.

And when a retry occurs, workflow needs to know which work has already finished.

3. Idempotency

Then there is the problem of doing something twice.

Imagine an agent sends an email & fails immediately afterward. When the step runs again, the application needs to know whether sending the email again is “safe”.

Idempotency keys, upserts, existence checks, and deterministic identifiers can help. The exact approach depends on the system, but protecting the side effect is still the application’s job.

4. Waiting and Scheduling

Long-running workflows should wait somewhere.

An agent could finish a draft & sit for days while an editor decides what to do. Keeping a worker alive during the entire wait “wastes” resources. Instead, the workflow should save its state & release the compute. It resumes when the event arrives.

Also it needs a defined path if the approval never comes, arrives too late, and/or gets rejected.

5. Flow Control

A workflow recovering correctly can cause problems when too much work arrives at once…

A single customer could trigger thousands of agent runs, consuming shared capacity & hitting an LLM provider’s limits. The resulting failures can then create even more retries.

Here are the necessary controls:

1. Concurrency: Controls how much work can run at the “same time”, so one workflow/customer cannot consume all available capacity.

2. Throttling: Spreads work over time when a sudden burst would overwhelm downstream services.

3. Rate limiting: Keeps requests within the limits imposed by external providers.

4. Per-tenant isolation: Stops one customer’s traffic from consuming capacity needed by others.

6. Observability

Finally, you need enough history to understand what happened when something goes wrong…

A basic queue can tell you a job failed. A long-running workflow needs more context around the steps, retries, results & failures.

Without this history, you have to piece together what happened from queue logs, application logs & tracing tools.

Now let’s see how Inngest handles all of this…so you don’t have to build this yourself…

§

§

What Inngest Is

Inngest is a durable execution platform for background functions, workflows & AI agents.

It sits between your application logic & infrastructure running underneath it. Your model handles generation, while your agent code handles tools & workflow logic.

And Inngest takes care of the execution around them, including retries, pauses, recovery & observability.

Your functions still run on your own compute & can run in environments such as serverless platforms.

i.e., Inngest provides the execution layer without requiring you to build all the workflow infrastructure yourself.

Ready for the best part?

step.run() as the Core Primitive

Everything in Inngest starts with step.run()5.

It wraps a unit of work & gives Inngest a point where it can save the result. Once the step finishes, its result is stored & can be reused if the workflow runs again.

So if a later step fails, Inngest does NOT need to rerun earlier steps. Instead, completed results can be reused & execution can continue from the step that still needs to run.

Your workflow logic stays in your application code. And Inngest handles the execution around it, while step.run() gives the runtime a clear “boundary” for each piece of work.

Long-Running and Waiting Workflows

Not every workflow finishes in one go.

Some need to pause for a while before they can continue…

• step.sleep()6 lets a workflow wait until a specific time.

• step.waitForEvent()7 lets it wait for an event, such as an editor approving a draft or a payment webhook arriving.

The crucial part is what happens during the wait…

A worker does NOT need to sit there doing nothing. The workflow can pause, keep its state, and continue when the required time/event arrives.

This becomes extremely useful for workflows involving human approval and/or external systems, where the next step would not happen for hours/days.

Durable Functions and Durable Endpoints

Inngest supports two ways to run these workflows based on how they are triggered:

• Durable functions run “asynchronously” in the background. They can start from an event, schedule, or webhook and continue until the workflow finishes. This fits research agents, ingestion pipelines, and other work where the result does not need to come back immediately.

• Durable endpoints are designed for “user-facing calls” where the response needs to come back to the user, while the steps inside the request still need durable execution.

The entry point is different, but both use the same underlying execution model.

Now let’s see how Inngest compares with other ways to build durable workflows…

§

BullMQ vs Temporal vs Inngest

Here are three ways to solve the same problem (along with their tradeoffs):

BullMQ: Queue-First

BullMQ makes sense if you need fine-grained control over background work.

Also it works well when your team already has the expertise to operate the required infrastructure. You get queues, workers, retries, rate limiting & job dependencies out of the box. Your team controls how those pieces are configured & how the workflow is built around them.

Yet the tradeoff is the “operational” work.

Someone has to own the infrastructure, tune the system & deal with the cost when a configuration does not behave as expected in production.

So this fits well for organizations with a dedicated infrastructure team seeking that level of control.

Temporal: Workflow-First

Temporal takes a more workflow-centric approach…

Its biggest advantage shows up when workflows become very complex & depends on/call many other workflows. A dedicated workflow model can make a large web of dependencies easier to keep straight.

Yet the tradeoff is “complexity”.

Temporal is more involved for developers & operators. The extra machinery makes sense when workflows actually need it.

Inngest: Application-Code-First

Inngest takes a simpler approach…

The workflow stays in your application code, with steps as the basic units of durable work. Completed steps are saved & reused during recovery, while failed steps can retry independently.

Your functions run on your existing compute, while Inngest handles the orchestration around them. You get durable execution without building the workflow infrastructure yourself.

So you take on less infrastructure & workflow complexity.

Yet the tradeoff is less low-level control than BullMQ.

Choosing Between Them

There is NO one right choice.

• BullMQ works well when you have the infrastructure team & want to control the system yourself.

• Temporal becomes more compelling as the workflow graph grows more complex.

• Inngest fits when you want durable execution around application code. It avoids much of the infrastructure & workflow complexity.

So the question is not which tool has the most features….Instead, how much control & complexity your workflow actually needs….

Let’s keep going!

§

Don’t let agent failures blow the budget (Partner)

Agents, like all of us, often need to recover from failure. But they shouldn’t start from scratch.

Inngest’s durable execution checkpoints your agents every step of the way.

So your agents can pick up where they left off and avoid doubling up on token & compute spend.

Build durably with Inngest - includes a generous free tier and a sleek local dev server.

§

Under the Hood: How Inngest Executes a Function

Every Inngest function follows the same basic execution pattern:

A trigger starts the run, steps execute one by one, their results get saved, and later executions reuse the work already completed.

Once you see this flow, the recovery behavior becomes much easier to understand…

Step 1: Starting an Execution

A run can start from an event, a schedule, or a webhook.

When the trigger fires, Inngest invokes your function over HTTP. Your function still runs on your own compute. You could be using a serverless platform, a container, or a traditional server.

There is NO separate worker process sitting there waiting for your function.

Inngest coordinates the execution, while your existing application environment runs the code.

Step 2: Discovering Completed Steps

The interesting part starts when the function should rerun...

The function begins from the top, but Inngest already knows which steps have been completed. Before running a step, the SDK checks its saved state.

If the step has a recorded result, the SDK returns this result instead of running the code again.

Each step has its own ID, and Inngest uses the step ID to identify its stored state.

SDK hashes the ID & use it to look up the previous result. This is also why completed steps can be reused across later executions & deployments.

So the function would run from the start again, but the completed work does not.

Step 3: Executing New Steps

Any step without a saved result still needs to run.

Inngest executes each step separately & records the result when it finishes. The result then becomes available to the next part of the workflow.

If the function runs again later, Inngest can use the recorded result instead of re-executing the same step. This lets a workflow move forward without repeating everything it has already finished.

Step 4: Checkpointing

Inngest can checkpoint progress as the function runs instead of waiting for the full workflow to finish.

With checkpointing enabled, the SDK can execute steps immediately and send progress back to Inngest as those steps complete. This reduces extra latency between steps while still recording the workflow’s progress.

This is especially useful for AI workflows, where a function could contain many small steps and even tiny extra latency between each one can add up.

If a step fails & needs to retry, Inngest can fall back to its standard orchestration path to handle the failure safely. So you get faster execution when things go well, without losing recovery behavior when they don’t.

Step 5: Waiting Without Holding Compute

Some workflows reach a point where they simply need to wait…

• step.sleep() can pause execution until a specific time.

• While step.waitForEvent() waits for an external event, such as an editor approving a draft or another system confirming a payment.

Also you can configure a timeout for an event wait.

Your function does NOT need to keep a worker process running just to sit there while waiting. Inngest tracks the workflow, and when the required time/event arrives, execution can continue with the recorded progress.

So a workflow can start on one machine, pause for hours/days, and continue later without keeping compute occupied the entire time.

Everything looks pretty smooth so far…But real workflows rarely stay this way.

Let’s see what happens when something goes wrong…

§

How Inngest Handles Failures

Once a workflow runs for a while, things can go wrong in different ways.

A retry is useful in some cases but not in others…

A temporary failure can usually be retried, while a process crash requires the workflow to recover its previous progress. Duplicate execution brings its own problems, and some workflows simply need to wait for a person before they can continue.

There is also one more case to consider:

The workflow can finish without technical errors and still produce a BAD result.

So instead of treating each failure the same way, let’s look at what each one requires...

Transient and Permanent Failures

An LLM request could time out, or the provider could return a 429 HTTP status because you have hit a rate limit.

In either case, the problem could disappear on the next attempt, so retrying the step makes sense.

Inngest retries steps independently.

i.e., a failed drafting step can run again without repeating the retrieval & extraction steps.

If you configure retries: 58, the step gets five retries after its initial attempt, while the completed steps keep their recorded results.

But some errors won't fix themselves…

If the request is invalid/API key is wrong, running the same step again will most likely produce the same error. Inngest lets you mark an error as non-retriable, so it skips the remaining attempts & moves the run toward its failure path.

Once all retries have been used, onFailure can handle the failed run.

It can send an alert, notify a user, trigger a fallback, or pass the problem to another part of the system.

Infrastructure Failures

Retries are NO help when the process running the workflow disappears…

Suppose several steps have already completed but the process restarted before the next step finished. When the function reruns, it needs to know which work has already been done.

Inngest keeps the results of completed steps; so those results can be reused when execution starts again. The workflow can then continue from its saved progress instead of rerunning all the earlier steps.

So even though the process itself did not survive, the work it had already completed does.

Duplicate Execution

Recovery can also create a different problem:

If the same operation runs again, you need to make sure it does NOT accidentally perform the same action “twice”.

A webhook could arrive more than once, or a function could get re-triggered after a failure. Inngest provides event IDs & function-level idempotency keys to prevent duplicate function executions for the same operation.

Yet the external action still needs its own protection.

If the workflow sends emails and/or writes to a database, you’d need an idempotency key, an upsert, an existence check, or a deterministic identifier so the operation remains safe to reattempt.

Inngest can help prevent the function from running twice, but “your” application still needs to make its side effects safe.

Human Delays

Think of a research agent…

Once the draft is ready, an editor would need to approve it before publishing. Approval could arrive in a few minutes/days, so keeping compute running the entire time wouldn't make much sense.

Instead, the workflow can wait for the required event with step.waitForEvent(). Also you can set a timeout for cases where the event never arrives. Once the matching event arrives, the workflow continues with its previous progress intact.

From there, you can handle the result normally…

An approval can continue the workflow, a rejection can stop it, and a timeout can trigger an escalation/fallback.

Disconnected Clients

Sometimes the workflow is fine, but the connection to the user breaks.

Token streaming is a good example:

An agent can be in the middle of a model turn/tool call when the browser closes or the connection drops. The agent can keep running in the background, while the browser simply reconnects later & catches up with the stream.

This separates the agent’s execution from the connection delivering its output.

i.e., the browser can disappear without taking the workflow with it, and when the user comes back, the result is still there.

Outcome Failures

The workflow can finish successfully but still give you a BAD result.

An agent could choose the wrong tool, miss an important source, or produce an answer a reviewer rejects. Technically, everything worked because every step completed without an error.

But completing the workflow is NOT the same as producing a good outcome.

The execution system can tell you whether the work finished… yet it cannot decide by itself whether the result was actually good…

For this, you need scoring and evaluation; let’s look at it next…

§

§

Other Pieces That Matter in Production

Let’s dive in!

Handling Traffic and Concurrency

A workflow recovering correctly can cause problems if too much work arrives at once.

A burst from one customer can fill the queue, hit a shared LLM provider’s rate limit, and “slow down” everyone else. So the system needs ways to control how much work runs & how quickly new work starts.

Here are the flow-control tools used by Inngest:

1. Concurrency: Controls how much work can run at the same time. If a database struggles beyond 20 concurrent writes, a concurrency limit keeps the workflow from overwhelming it.

2. Throttling: Controls how quickly new work starts. If an API allows 100 requests per minute, throttling spreads the work out instead of sending everything at once.

3. Rate limiting: Instead of holding excess work for later, it skips events once the configured limit is reached. This is pretty useful when some events are better dropped than processed later.

4. Debouncing: Helps when many events arrive within a short period, but only the final state matters. Instead of running the function for every event, the system waits until the burst settles before starting it.

5. Priority: Lets more important work move ahead when capacity is limited. A critical workflow does not have to sit behind a large amount of lower-priority work.

These controls sit alongside the execution…so traffic management does not become another layer around the workflow.

Seeing What Happened

Once something fails, you need to see what happened before you can fix it.

A basic queue usually gives you the job & its final status. For a long-running workflow,,, you need more. You need to see where the run spent its time, which steps completed, what retried & where execution stopped.

Inngest automatically traces function runs.

The trace shows step timing, queue delays, retries, and the data going into & coming out of each step. You can follow the whole workflow in one place instead of piecing it together from queue logs, application logs & separate tracing tools.

And when you fix the problem, you can replay the affected runs instead of rebuilding them manually. Inngest lets you select runs by time range and status and replay them in bulk.

Plus, replays are spread over time so a large recovery does NOT overwhelm the application.

Execution Success vs Outcome Success

Execution tells you whether the workflow finished…but it does NOT tell you whether the result was good.

Inngest lets you attach scores to a run/step.

This keeps user feedback, human review, or business outcomes connected to the execution that produced them. The score can also arrive after the workflow has finished. This is extremely useful when a result's quality only becomes clear later.

It gives you two different things to look at:

• You can see whether the workflow completed.

• And you can see whether it actually did the right thing.

But you still need to handle a few pieces yourself.

Let’s have a look at what they are…

§

What You Still Have to Handle

Here are some responsibilities that still belong to your application:

Idempotent Side Effects

A step can rerun after an error, so the code inside it needs to be “safe”.

For example, inserting a new user twice can create two records if the first write succeeded, but the response never reached the application. An upsert/deterministic identifier can make the second attempt harmless.

The same applies to emails, payments, API calls, or/and other external actions.

Inngest can control when a step runs again, but the application still needs to decide what happens when the external operation is attempted again.

Good Step Boundaries and Stable Step Identities

How you split the workflow into steps is still a design decision…

A step should represent a meaningful “unit of work”. Put too much inside one step, and a retry could repeat more work than necessary. Split everything into tiny steps, and the workflow becomes harder to reason about.

Step IDs matter for the same reason…

Inngest uses them to memoize completed work, so changing an existing step’s ID makes Inngest treat it as a different step. The SDK also uses a counter when the same step runs inside a loop, so the ID can stay stable across iterations.

The execution layer can preserve the state, but it cannot decide where your workflow should draw its boundaries.

Retry and Failure Policies

Inngest handles retries automatically,,, but the application still needs to decide when a retry makes sense.

A network timeout is usually worth another attempt, but an invalid API key is not. You can mark permanent errors as non-retriable, while onFailure9 gives the workflow a place to go once its retries are exhausted. The default is four retries after the initial attempt, and you can configure this per function.

The right response after failure depends on the workflow:

Sometimes you need an alert, and sometimes you need a fallback. In other cases, the user needs to be told that the operation could not finish.

Data Consistency Across External Systems

Durable execution keeps the workflow’s progress.

Yet it does NOT combine several external systems into one transaction.

Suppose a workflow records a payment & then creates an order. If the payment succeeds but the process fails before the order gets created, retrying the workflow does not automatically undo the payment.

The application still needs a way to make the two operations work “safely” together. Idempotency, compensating actions, and careful ordering can help based on the systems involved.

This becomes highly crucial when a workflow touches many external systems.

The execution layer can keep the workflow moving, but the application still owns the consistency of the data it changes.

Let’s put all of this into one workflow & see how it works together…

§

Putting It All Together: A Durable Research Agent

Let’s see what all of it looks like in a real workflow using a research agent:

Workflow Starts

The agent first retrieves its sources & extracts the evidence it needs.

Each part runs as its own step, and once a step finishes, its result is saved.

So when retrieval & extraction are done, the workflow already has two completed pieces of work it can reuse later.

Failure and Recovery

The agent moves on to drafting, but the model provider times out halfway through the step…

Yet the workflow does NOT start over. The drafting step retries, while the retrieval & extraction results stay untouched. Once the draft succeeds, its result is saved & the workflow can move on.

This is where checkpointing becomes very useful…

The workflow does NOT need to remember everything in the worker process because the completed work is already persisted.

Waiting for Human Input

The draft now needs editor approval…

So the workflow calls step.waitForEvent() and pauses.

Nothing needs to keep running while the editor decides. Two days later, the approval event arrives, and the workflow continues with the publishing step.

The same workflow has now survived a model failure & a two-day wait without losing its progress.

Execution History and Replay

Once publishing finishes, the run still has a complete execution history.

You can see how the workflow progressed, where it retried, and how long each part took.

If the final answer turns out to be poor, a score can be attached to the run later. And if a fix needs to be tested against previous executions, the affected runs can be replayed instead of rebuilding them manually.

By the end, the workflow has done more than simply finish.

It has recovered from a failure, waited for a human without holding compute, preserved its progress, and left behind a history of what happened along the way.

§

Closing Thoughts

Reliable AI agents need more than a model that produces good answers…

Once an agent runs long workflows, calls external tools, waits for people, and handles real traffic, the execution layer becomes just as important.

Here are the key takeaways for you:

• Durable execution saves progress. Completed steps can be reused after a failure instead of rerunning the whole workflow.

• Retries need protection. Idempotency, backoff & clear failure paths keep retries from creating bigger problems.

• Waiting is part of the workflow. Human approvals and external events can take hours/days without keeping compute running.

• Flow control is part of reliability. Concurrency, throttling, and rate limits keep one traffic spike from affecting everyone else.

• Execution history matters. Knowing what ran, what failed, and what retried makes debugging much easier.

• A successful run is not always a good run. Scoring and evaluation help measure result quality.

BullMQ gives you queue primitives. Temporal gives you a workflow model. Inngest puts durable execution around the application code you already write.

For AI agents, that application-centric approach is what makes Inngest particularly compelling.

§

Don’t let agent failures blow the budget (Partner)

Agents, like all of us, often need to recover from failure. But they shouldn’t start from scratch.

Inngest’s durable execution checkpoints your agents every step of the way.

So your agents can pick up where they left off and avoid doubling up on token & compute spend.

Inngest offers a generous free tier and a sleek local dev server for testing.

Build durably with Inngest

If you find this newsletter valuable, share it with a friend, and subscribe if you haven’t already. There are group discounts, gift options, and referral rewards available.

Subscribe now

Want to reach 241K+ tech professionals at scale? 📰

If your company wants to reach 241K+ tech professionals, partner with me.

Thank you for supporting this newsletter.

You are now 240,001+ readers strong, very close to 241k. Let’s try to get 241k readers by 27 August. Consider sharing this letter with your friends and get rewards.

Y’all are the best.

Share

References

• Inngest Documentation

• How Inngest Functions Execute

• Inngest Steps

• Step Execution and step.run()

• Error Handling and Retries

• Handling Idempotency

• Durable Endpoints

• Waiting for Events

• Concurrency and Flow Control

• Throttling

• Observability and Metrics

• Scoring Function Runs

• Durable Token Streaming

• Durable Endpoints Streaming

• Share this letter & I’ll send you some rewards for the referrals.

I used to treat large language models like “search engines”…

I wrote a tiny prompt, pressed Enter, and expected a good answer.

But the model often ignored my constraints and/or solved a different problem. I didn’t realize until now that my prompt left too much open to “interpretation.”

A prompt tells the task, provides context, defines constraints, and makes the expected outcome clear. When the prompt is CLEAR, the model has less to guess.

But prompts are only one part of the story…

Modern AI systems do much more than respond to prompts. They retrieve documents, call external tools, remember useful information, and validate the outputs before responding.

They work well because they deliver the right “context” at the right time.

Onward.

[Webinar] Can you prove AI is working?

AI is in your engineering workflow. While the token spend shows it, the throughput doesn’t. The human is very much still in the loop, and that’s a context problem.

Join live on Aug 19 (FREE) to learn:

• The 4 metrics to measure where AI gains leak out before production.

• The 8 stages of context maturity, the specific walls capping your metrics, and a free tool to pinpoint where your team is.

• Why more MCPs and bigger context windows aren’t enough, and what it takes to get real value from your agents.

Register Now

(Thanks to Unblocked for partnering on this newsletter.)

I want to introduce Tina as the guest author.

She’s a data engineer specializing in AI, machine learning, and large language models. She explains the why behind AI and data engineering concepts. Her work has been featured through Medium’s Boost program, and she also publishes the AI Cartographer.

You can follow her on:

• Medium

• Substack

• LinkedIn

• X

Here’s what’s inside this newsletter:

• Why good prompts still produce weak answers. How tokenization, context windows, attention, and in-context learning shape every response, and why relevance matters more than stuffing more text into the prompt.

• The prompting techniques behind reliable outputs. What zero-shot, few-shot, chain-of-thought, ReAct, reflection, structured outputs, and function calling each solve, and when adding more reasoning only increases cost.

• Prompting is only one layer of the system. How context engineering, Retrieval-Augmented Generation, memory, reranking, compression, and long-context design determine what the model knows before it answers.

• Every AI workflow has a failure mode. Why tool loops, retrieval overload, prompt injection, stale memory, invalid JSON, benchmark gaming, and weak evaluation can break an application even when the prompt looks correct.

• Eight prompts and a production checklist you can reuse. Classify, extract, summarize, ground answers in documents, control tool use, improve reasoning, revise drafts, and review customer replies before shipping them into a real system.

Why Prompt Engineering Matters

If you ask a language model to “write something,” the model has to make several decisions on its own…

Should the response be a story, an email, or a poem? Who is the audience? How long should the response be?

Prompt engineering is the process of designing prompts to help a language model produce better responses.

A language model cannot ask follow-up questions.

And the response depends entirely on the information available in its context window. So clear prompts typically lead to better results.

Still, a “good” prompt has its limitations:

When the required information is missing, no prompt can fill the gap. Modern AI systems also rely on retrieval, memory, tool use, and safety mechanisms to produce reliable responses.

Before learning how to write better prompts, let’s understand how a language model reads a prompt…

Share

How Large Language Models Read a Prompt

A prompt does NOT go directly into a language model.

Instead, text gets broken into smaller pieces before processing begins. This representation makes it easier for the model to work with text of any length/language.

A token is the basic unit of text the language model processes, representing a word, part of a word, punctuation mark, or other text fragment. The model then works with a sequence of tokens instead of complete words/sentences.

Those tokens become part of the model’s context window1, its workspace for the current request.

It contains everything the model can use to generate a response, such as the system prompt, instructions, examples, retrieved documents, conversation history, and your current prompt. As the model generates new tokens, they also become part of the same context2. Yet the context window has a “fixed” size, so the prompt & the response share the same token budget.

The model does NOT treat every token equally…

It decides which part(s) of the context deserve the most focus before generating the next token. This process is called attention. i.e., the model focuses more on the information most useful for predicting the next token.

Imagine listening to several conversations at once…

You hear every voice, but you pay more attention to the most important one.

You’d expect more context could improve the response. But this is NOT always true. Once the model has the information it needs, extra details can compete for attention instead of helping.

i.e., relevance matters more than quantity.

Let’s return to this idea later to discuss Retrieval-Augmented Generation (RAG) and long-context systems…

Share

What is In-Context Learning

Spend a little time using different AI assistants, and you’ll notice something…

We often upload images to Gemini, ask ChatGPT general questions, and use Claude for coding. You’d also notice something else. After rewriting the same prompt a few times, many people try a different approach… They add an example of what they want, and the response often “improves” immediately.

The model has NOT learned a new skill.

Instead, it uses the examples in the prompt to figure out the pattern and applies it to the next input. Larger instruction-tuned models usually do this more reliably than smaller ones.

This ability is called in-context learning.

One explanation is the model recognizes patterns similar to those encountered during training. Another comes from Bayesian inference3, where each example provides another clue about the current task.

Yet this works only if the examples represent the task well.

If a customer support prompt contains only straightforward complaints & praise, the model will struggle with a review like “The package finally arrived. Better late than never.” The prompt never showed sarcasm, so the model has to infer the missing pattern…

i.e., a few carefully “chosen” examples usually work better than many similar ones.

Next, let’s look at how to design effective prompts…

Share

Design Effective Prompts

Many people treat a prompt as one long block of text…

When the response isn't what they expected, they rewrite the entire prompt & hope for a better result. Instead, think of a prompt as a “collection of components” and not ONE long instruction.

Each component has a specific purpose.

Some tell the model what to do, while others provide background information, examples, retrieved documents, conversation history, and/or the desired output format.

You do NOT need every component for each prompt.

A simple question only requires clear instructions, while a more complex application combines several components to give the model the context it needs. This way of thinking also makes it easier to improve the prompts.

And when a prompt produces poor results, the issue is often a missing/unclear component rather than the entire prompt. i.e., don’t rewrite everything when a problem occurs, but identify the component & then refine it.

Let’s take an example…You want to generate a project status update for your team…

You could write:

Write a project update.

The model can certainly generate a response, but it has to guess the project, the audience, the level of detail, and the desired format…

Now check this:

Write a weekly project update for the engineering team.
Summarize the progress of the authentication service,
mention that API integration is complete,
database migration is in progress, and testing begins next week.
Keep the tone professional and format the response as three bullet points.

The second prompt is better because it provides the components the model needs: a clear instruction, relevant context, specific information to include, and the expected output format.

Prompt Diagnosis Matrix

Some tasks need more than clear instructions…

Here’s a quick diagnostic guide:

So far, we have focused on the building blocks of a prompt…

The next question is: How many examples should you include?

Prompting Strategies and Reasoning Techniques

You ask a model to classify customer reviews as positive, negative, or neutral.

Sometimes a simple instruction is enough. But for trickier tasks, especially when you care about a specific style, format, or way of reasoning, just telling the model what to do is NOT sufficient. In those cases, giving one or more examples usually helps it perform more reliably.

The amount of guidance you provide gives rise to different prompting strategies…

Here are the common prompting strategies based on the number of examples they use:

• Zero-shot prompting: you provide only the instruction and the task.

• One-shot prompting: you add a single example to show the expected behavior.

• Few-shot prompting: you include several examples so the model can recognize the underlying pattern.

• Many-shot prompting: extends few-shot prompting by using dozens or even hundreds of examples.

It could be tempting to keep adding examples when the model makes mistakes…

Yet more examples are NOT always better. Every example consumes part of the context window. A small set of diverse examples often works better than a long list of similar ones.

Plus, a tiny instruction such as “Let’s think step by step” can improve performance on many reasoning tasks.

You do not need to provide examples. Instead, you ask the model to work through the problem step by step before answering. This technique is called zero-shot chain-of-thought prompting.

Different tasks benefit from different prompting strategies.

A prompt that works well for factual questions may struggle with complex reasoning.

Let’s explore the most widely used techniques and when to apply each one in coming sections…

Share

ReAct and Tool: Augmented Reasoning

Ask ChatGPT, “What was your company’s revenue for last quarter?“

The answer already exists, but it’s stored in the company’s database and NOT in the model’s memory. If ChatGPT can access the database, it first retrieves the relevant information, then uses it to answer the question.

Good reasoning is only part of the solution…The model also needs the right information. Without it, even perfect reasoning cannot produce the correct answer…

This is the idea behind Reason & Act (ReAct).

The model can decide when to use an external tool, retrieve the information it needs, and then continue reasoning with the new evidence.

This is similar to how we solve problems:

When you realize you’re missing a piece of information, you stop, look it up, and then continue. ReAct follows the same pattern… The model alternates between reasoning and using tools, with each result becoming part of the context for the next step.

Language models can use different tools for different tasks:

They could search the web, retrieve documents from a company database, perform calculations, and/or interact with external systems.

So the model is no longer limited to the information already in its context by combining these tools with reasoning…

It can gather what it needs before producing an answer…

And this typically leads to accurate & reliable responses.

Worked Example: Stopping a Tool Use Loop From Repeating Itself

Imagine you’re building an assistant with access to a search tool…

Your first prompt could look like this:

BEFORE

You have access to a search tool.
Use it to answer the user’s question.

This seems reasonable…the model knows it has a tool & knows its objective.

But nothing tells it when to stop searching or how to decide whether a search was “successful”. If the search results are incomplete/irrelevant, the model would repeat the same query several times or continue searching unnecessarily.

A better prompt adds explicit decision rules:

AFTER

You have access to a search tool.
Use it when the answer requires current 
or specific information you do not already have.
Before each search, briefly state what you still need to find out.
After each search result, state explicitly whether it answered
what you needed or not; if it did not, try a different query
rather than repeating the same one. You may search up to 3 times.

If you still cannot answer after 3 searches, say so directly instead of guessing.

Notice what changed…

The revised prompt doesn't simply tell the model to use a tool. It defines when to use the tool, how to evaluate the result, when to try a different strategy, and when to stop.

These tiny additions reduce unnecessary tool calls & make the model's behavior more predictable.

Verification and Reflection Loops

Read more

• Share this letter & I’ll send you some rewards for the referrals.

The smartest AI models in the world cannot decide, mid-answer, they need more information and go get it…

They can reason at length, draft legal memos, and score in the 99th percentile on standardized tests. But every response they produce comes from what is already in the context window.

They have no mechanism to act on the world, observe the results, and adjust.

For most tasks, this is enough. For any task where the answer requires searching, reading, discarding dead ends, and circling in over many steps, a passive model is the wrong architecture.

An agent gives the model this ability…

Instead of generating a response from a single prompt, the model decides what to do next, calls a tool, reads the result, and keeps going until it has enough. The model stops being something you call and becomes something that runs.

This loop is the entire idea behind agents, and it is how Perplexity Deep Research, ChatGPT agent mode, Claude Code, and Cursor all work under the surface.

Those products handle general use cases well.

While building your own is worth it when you need to control the runtime: which tools the model can call, how the loop is structured, when it stops, and how you measure its success.

Onward.