• Share this letter & I’ll send you some rewards for the referrals.

The smartest AI models in the world cannot decide, mid-answer, they need more information and go get it…

They can reason at length, draft legal memos, and score in the 99th percentile on standardized tests. But every response they produce comes from what is already in the context window.

They have no mechanism to act on the world, observe the results, and adjust.

For most tasks, this is enough. For any task where the answer requires searching, reading, discarding dead ends, and circling in over many steps, a passive model is the wrong architecture.

An agent gives the model this ability…

Instead of generating a response from a single prompt, the model decides what to do next, calls a tool, reads the result, and keeps going until it has enough. The model stops being something you call and becomes something that runs.

This loop is the entire idea behind agents, and it is how Perplexity Deep Research, ChatGPT agent mode, Claude Code, and Cursor all work under the surface.

Those products handle general use cases well.

While building your own is worth it when you need to control the runtime: which tools the model can call, how the loop is structured, when it stops, and how you measure its success.

Onward.

[Webinar] Can you prove AI is working?

AI is in your engineering workflow. While the token spend shows it, the throughput doesn’t. The human is very much still in the loop, and that’s a context problem.

Join live on Aug 19 (FREE) to learn:

• The 4 metrics to measure where AI gains leak out before production.

• The 8 stages of context maturity, the specific walls capping your metrics, and a free tool to pinpoint where your team is.

• Why more MCPs and bigger context windows aren’t enough, and what it takes to get real value from your agents.

Register Now

(Thanks to Unblocked for partnering on this newsletter.)

This newsletter builds a research agent from scratch:

A complex question goes in… The system breaks it into focused sub-questions, calls tools in a loop, reads the sources worth reading, and then returns a structured report with citations.

Think about how a person actually answers a HARD question.

They do not produce the answer in one pass. They search for something, read what comes back, realize it raises a new question, search again, discard a dead end, and circle in on an answer over many steps.

The model we built in the last newsletter cannot do any of this.

Its knowledge is frozen at training; its memory ends at the context window, and every response comes from what it already knows. For explaining a concept or drafting a document, answering in one pass is fine.

For a question where the answer has to be found rather than recalled, one pass is the wrong shape entirely.

I want to reintroduce Louis-François Bouchard as the author of this newsletter.

He’s a best-selling author (Building LLMs for Production), the co-founder of Towards AI, and the creator of the YouTube Channel, What’s AI, where he helps people understand AI and learn how to apply it in the real world.

Through his development work with clients and his content, teaching, and AI training programs on the Towards AI Academy, Louis focuses on making AI practical for builders, engineers, and curious learners alike.

At Towards AI, he and his team train AI engineers through courses built for every stage, from beginner to advanced. That educational mission and the real-world experience building for his clients are exactly why I wanted him in this newsletter series.

Inside this newsletter, you’ll get:

• The agent loop. The think-act-observe cycle that makes an agent an agent, and why most agents fail not at reasoning but at knowing when to stop.

• Tool design. Why the model’s success depends almost entirely on how a tool is described, how to make errors recoverable, and why quality drops once you pass about 20 tools.

• MCP and the tool ecosystem. How one protocol lets any model talk to any tool, when a ready-made server is enough, and when writing your own is worth it.

• Multi-agent patterns. How orchestrator-worker and parallel-and-synthesize patterns work, and what the cost multiplier looks like in practice.

• Evaluating agents. Why scoring the final answer is not enough, and how to catch the failure where the agent gets it right in 50 tool calls when 5 would have done.

• Practical build. A small research agent in plain Python: an MCP server that exposes search and fetch, an agent loop with native function calling, a deep-research flow that decomposes, fans out, synthesizes, and reviews, and a binary judge.

Why Build Your Own Agent?

For many tasks, a hosted agent/agent mode is a good choice.

ChatGPT Agent Mode, Perplexity Deep Research, Claude Code for coding tasks, Manus, and Devin already give you the convenience of an agent: a model that reasons across steps, calls tools, and keeps going until it has enough.

For generic web research, coding in a local repo, or a single person using it interactively without audit/cost constraints, a hosted product will usually be faster to deploy than anything you would build yourself.

The case for building your own comes down to what hosted products cannot give you: ownership of the runtime.

Let’s dive in!

• Tool control. Hosted agents do not know your customer relationship management system (CRM), your warehouse, or your internal services. They cannot read your pipeline tracker, write to your ticketing system, or trigger a deployment. Your own agent exposes only the tools you allow, with the scopes you set. A literature research agent that searches PubMed first, ranks results by impact factor, and checks for retractions is a different artifact from a generic web searcher. The value is in the actions only your tools can take, and the safety model follows directly from that: a tool that does not exist in your runtime cannot be called.

• Workflow control. With your own agent, you decide the sequence, stopping rules, and escalation path. Always search these sources first. Require a citation after every claim. Cap the loop at ten turns. Route low-confidence answers to a human reviewer instead of letting the model guess. Hosted agents ship with whatever defaults the vendor chose, and you cannot change them.

• Audit and evaluation. Every tool call, every observation, and every model decision can be written to a trajectory log you own and can replay. You can run your own evaluation on this trajectory, not just on the final answer. Regulated industries need that trail by default, and your own agent emits it. A hosted agent gives you, at best, what the vendor decides to expose.

• Cost and latency. A scoped agent over your own tools typically spends fewer tokens per task than a general-purpose agent, because it does not need to discover which sources to trust on every run. Tight retrieval loops over your own infrastructure also avoid the multi-second round-trip that external web calls introduce. The fastest tool is one you control.

Also, building your own agent does not mean building the primitives from scratch.

Native tool calling, the loop itself, and context handling are now standardized across providers, and frameworks such as LangGraph and OpenAI Agents SDK ship them. So what you build is the tool layer, workflow, and evaluation.

The loop is the runtime; your application is the set of tools, prompts, and stopping rules you run on it.

If the task is generic, data is public, and a hosted product already meets the bar, building your own is the wrong investment…

Share

What We Are Building

The system we’re building is a research agent.

A user types a question, the model breaks it into focused sub-questions, calls web search and page fetch in a loop to gather sources, and writes a structured report with citations when it has enough.

A typical input is a question like “Summarize the recent literature on LLM agent evaluation, focused on trajectory metrics,” and it returns a 600 to 1,200-word report with two/three sections, 8 to 15 inline citations, and a confidence rating per section. A run takes 10 to 20 tool calls and 2 to 4 minutes, almost all of it spent reading fetched pages rather than on the model’s own reasoning.

These numbers are design targets, not measurements from a specific run.

The build runs under a few fixed constraints:

The loop stops on a completion signal or at an iteration cap of ten turns. Every claim in the final report has to cite a retrieved source, which a reviewer checks at the end of the run. Token-budget caps and a low-confidence escalation path are production guardrails we describe but do not implement (in Part 6 walkthrough).

Every run writes a trajectory log of model calls, tool calls, observations, and stop reasons.

Building this system means designing several layers that depend on one another.

The model picks the next action. The harness is everything that makes the action useful: the loop that runs the model in a cycle, the tools it can call, the protocol it uses to reach external services, the memory it carries between turns, and the evaluation that tells you whether any of it worked. Each layer has its own failure modes, and none of them work in isolation.

This newsletter works through five questions in order:

• How the agent loop runs and when it stops,

• How to design tools the model uses correctly,

• When to use MCP and what it provides,

• When one agent is not enough and how memory changes the design,

• And how to measure whether the system works without letting it run up an unbounded bill.

The loop comes first because nothing else in the system matters if it does not terminate.

Let’s start!

Share

Part 1: The Agent Loop

The agent loop is the core of the system.

Everything else--tools, memory, multi-agent patterns, evaluation--sits on top of it.

Getting the loop right means understanding what it is, how each turn runs, and most importantly, how it stops.

What is an Agent

Anthropic’s engineering team describes an agent as an LLM that uses tools in a loop, autonomously, until it has accomplished a task.

There is a model that decides what to do next, a set of tools it can call, and a loop that runs the model, executes the tool call, feeds the result back, and repeats.

• Remove the tools, and you have a chat assistant.

• Remove the loop, and you have a one-shot function call, like a chatbot that queries a weather API once and reports back with no iteration.

• Remove the model, and you have a workflow.

The easiest way to see where the agent sits is to plot it on a line.

At one end is a single LLM call. You ask a question; the model thinks once, and returns an answer. There is no opportunity to gather new information/change course.

In the middle is a workflow. Your code fixed the steps in advance. For example: search the web, fetch the top result, then summarize it. Every task follows the same sequence, whether it is the correct sequence or not.

At the other end is an agent. Instead of following a fixed path, the model decides what to do next. It could search the web, read a page, search again with a better query, or stop and answer when it has enough information.

The key difference is who controls the next step…

In a workflow, your code decides. In an agent, the model decides based on what it has learned so far.

How One Run Executes

Each turn of the loop has three steps:

1. Model thinks and emits either a tool call or a final answer.

2. The code acts, running the tool the model picked.

3. Result is observed by being appended to the context for the next turn.

This is the think-act-observe cycle, and it runs until one of the stop conditions trips.

A typical research run takes 10 to 20 turns before the model decides it has enough to answer.

The cycle itself is fixed… Yet how the model organizes its thinking inside each turn is not, and this choice is the loop pattern.

Loop Patterns

Here are three named patterns that cover most real systems:

• ReAct1 (Yao et al., 2022) interleaves a reasoning trace with each tool call (”I need X, so I search Y”) and is the “most common” shape for prompted agents.

• Plan-and-Execute2 (LangChain, 2023, building on Wang et al.’s Plan-and-Solve) separates planning from execution: a first call returns a short-step plan, a second loop executes it, and the agent replans on failure. It’s better for “long tasks” where wandering off-plan is expensive.

• Reflection3 (Shinn et al.’s Reflexion, 2023) adds a critic step: the agent reads the draft answer, lists problems, and feeds them back. It earns its keep when single-pass quality plateaus.

A reasonable starting point is a prompted ReAct loop.

Move to Plan-and-Execute when exploration is expensive, and add Reflection when quality plateaus. Reasoning models like the o-series and Claude with extended thinking are an orthogonal choice that move the planning inside the model; try one when planning quality is the bottleneck.

Whichever pattern you pick, the loop has to stop.

This is the part most people get wrong.

Stopping the Loop

Most agent failures are termination failures.

The model keeps searching, refining, and searching, never deciding it has enough. Without a hard stop, one stuck task burns $50 of tokens.

A system needs three stop conditions4, and the loop fires on whichever trips first:

• Completion signal: the model emits a final answer with no tool call.

• Iteration cap: a hard limit of 15–25 turns for research, and fewer for simple tasks.

• Budget cap: maximum number of tokens or dollars per task that returns partial results plus a flag when it trips.

When either cap consistently fires on a given question type, it usually signals that a tool is broken or the question is mismatched with the agent’s design.

Before reaching for an agent at all, it is worth asking whether the task actually needs one. Most tasks are workflows with known steps. An agent is the “right” choice when the path cannot be determined until you see the data.

If the steps are known up front, hard-coding them is faster, cheaper, and easier to debug…

Part 2: Designing Tools the Model Can Use

Read more

• Share this letter & I’ll send you some rewards for the referrals.

System design can feel overwhelming, but a small set of patterns carries most of the weight in real-world systems.

In this newsletter, I’ll cover the 10 most important system design patterns to solve common problems.

For each, I’ll share:

• What problem it solves,

• How it works,

• Tradeoffs.

Let’s start!

§

Superblocks just launched something big for enterprise AI (Partner)

Until now, CIOs and CISOs had two choices:

• Block vibe coding completely.

• Or open the floodgates and risk sensitive company data.

Neither is a good option!

INTRODUCING SUPERBLOCKS 3.0

The secure, governed way to vibe code enterprise software…

Business teams can now build production-ready internal apps from a single prompt.

• Unlike consumer tools such as Replit & Lovable, every app runs inside your own AWS account and VPC, so your data “never” leaves your private cloud.

• Instead of relying on one big AI agent, Superblocks orchestrates specialized agents across the build, routes simpler work to lower-cost open-source models, and cuts token costs by up to 30%.

• Before anything ships, a swarm of security agents scans every deployment for business logic flaws, chained authentication bypasses, and supply-chain vulnerabilities.

Company login, role-based permissions, audit logs, encryption, and governance are enforced automatically across every app.

So everyone builds and nothing ships ungoverned.

TRY SUPERBLOCKS 3.0 TODAY

(Thanks to Superblocks for partnering on this letter.)

§

1. Caching

Imagine the “same” API route gets called a thousand times per second.

And each call hits the database for the result.

i.e., every read will:

• access disk storage,

• lock tables during reads,

• consume connection pool slots.

But all this work just to return data that has NOT changed since the last request…

You can use in-memory caching to solve this problem by storing frequently accessed data in memory.

Your app checks the “cache” first before querying the database. Cache hits return data from memory, while cache misses query the database & then store the result for later.

Advantages

You get extremely fast response times because memory access is 100x faster than disk.

Plus the database handles fewer requests since only cache misses reach it. Also, this lets you scale “read” capacity without adding more database replicas.

Disadvantages

But cache invalidation becomes difficult because it's hard to know exactly when cached data must be removed/refreshed.

Also you need to keep the cache & database in “sync” to avoid stale data. Plus, memory is significantly more expensive than disk storage.

This increases infrastructure costs.

Best practices

So cache only the data that can tolerate some staleness, such as product catalogs & user profiles.

Besides, set an appropriate Time-to-Live (TTL) based on how often the data changes. Also use the cache-aside pattern1 to keep the cache populated only when data is requested.

Caching gives you fast reads for data that doesn’t change often.

But it only helps when the same data gets requested over & over. So what about systems where users read “different” data?

§

2. Scaling reads

Your query performance looks good,,, but your database is “struggling”.

The problem is NOT slow queries; instead, it’s the volume. Thousands of users hit your API at the same time…

Every query runs fast, but every request still:

• Uses CPU on the database server

• Consumes a database connection

• Waits in line when the server gets overloaded

To overcome slow reads, you can add indexes and tune query execution plans.

This lets you get the most out of your existing servers. Then, when your servers reach their limits, add read replicas…

Your application sends writes to the primary database and routes reads to replicas…While the replicas stay synchronized through replication.

Advantages

You can scale reads simply by adding more replicas instead of upgrading a single database server.

Also, you can reduce latency by placing replicas closer to your users. Plus distributing reads across many servers prevents one database from becoming a bottleneck.

Disadvantages

Yet replication is NOT instant; replicas need time to copy changes from the primary.

Also, read-after-write requests may receive stale data if they hit a replica too soon. Plus, running replicas adds operational complexity because they must be monitored, maintained, and kept healthy.

Best practices

So optimize your existing database before adding more infrastructure.

Also, use read replicas only when eventually consistent reads are acceptable. Besides, route critical read-after-write operations to the primary database to guarantee fresh data.

Read replicas are great for high-read traffic applications where some stale data is acceptable. But how do you manage high-write traffic?

Share

§

3. Scaling writes

Your database handles every write,,, but eventually:

• You’ve already optimized inserts & added batching,

• You cannot keep upgrading to a larger database instance,

• CPU and IOPS2 (Input/Output Operations Per Second) has reached their limits.

You can solve this through sharding.

i.e., distribute writes across many database nodes…

You choose a partition key, and a hash function routes each record to a specific shard. Think of sharding like splitting a large phone book by region, where every region has its own book.

Yet each shard stores only a “subset” of the entire data…

So it’s important to choose a good partition key:

• User IDs: many of them, evenly distributed

• Geographic regions: related data stays together for better local queries

• Time windows: great for time-series data where older shards become read-only

This lets you scale writes...

You can add shards to increase capacity & reduce latency by placing shards closer to users.

Advantages

You can scale write capacity simply by adding more shards instead of relying on a single database server.

Plus, each shard handles only part of the total traffic, which reduces the load on individual machines. Also, placing shards closer to users can reduce write latency.

Disadvantages

But resharding is difficult because moving data between shards requires downtime and/or complex migrations.

Also, changing your partition key later requires relocating large amounts of data (this is expensive!). Plus, queries spanning many shards require extra coordination to combine the results.

Best practices

So choose partition keys with many distinct values to distribute traffic evenly.

Also monitor for hot partitions and rebalance shards if necessary. Besides, design your schema to minimize queries that must access many shards.

Sharding distributes writes across different machines. But what about distributing them across time?..

§

4. Message queues

Imagine your API receives a burst of 10,000 requests in one minute.

Each request triggers database writes. And this could make your system performance BAD. Some requests succeed, while others fail/time out!

Then the problems cascade:

• One slow service blocks the entire request chain

• Services must scale together even with different workloads

• Retry logic makes it worse, as clients repeatedly hit the same overloaded endpoint

You can solve this problem using message queues such as Amazon SQS, RabbitMQ, or Kafka.

They buffer messages between services. So when one service slows down, it has less impact on the others.

Here’s how a message queue works:

1. Producers send messages to the queue

2. The queue stores messages until they’re ready to be processed

3. Consumers retrieve messages at their own pace, process them, & then acknowledge completion

So instead of processing requests immediately, your API places them in a message queue.

Background services then process the messages asynchronously…

Advantages

Message queues “absorb traffic spikes” because messages get stored until workers can process them.

Plus, they decouple services, so each one can scale independently. Besides they set the foundation for event-driven architectures where services communicate through events instead of direct API calls.

Disadvantages

But in asynchronous processing, clients do NOT immediately know whether a request has completed successfully.

Also, message queues add latency because work happens later instead of during the request. Plus, they introduce another piece of infrastructure that must be deployed, monitored & maintained.

Best practices

So make consumers idempotent3 because messages could be delivered more than once.

Also use dead letter queues4 to isolate failed messages. Besides, monitor queue depth because growing backlogs are often an early sign of system problems.

Message queues make asynchronous processing possible. But what does asynchronous processing actually look like from a user’s perspective?

§

§

5. Asynchronous processing

Read more

• Share this letter & I’ll send you some rewards for the referrals.

§

I shipped this new feature:

→ Read support ticket,
→ Choose the right tag,
→ Then route the request to right person.

The simplest way to build the feature is to call a frontier model through an API…

Yet this approach doesn’t scale because of API costs, latency, and data sensitivity risks. Plus, the model isn't solving a HARD reasoning problem. Instead, it's making the same decision a thousand times a day.

So I tried a “boring” alternative approach:

Fine-tune a 3 billion parameter model on my desk in 74 minutes.

On one legal-labeling task, it scored 81.7% -- ahead of Claude Sonnet 4.6 at 77% and GPT-5.5 at 76.7%.

i.e., the small model is NOT smarter than a frontier model. But a specialist and does one narrow task extremely well.

Onward.

§

[Webinar] Can you prove AI is working?

AI is in your engineering workflow. While the token spend shows it, the throughput doesn’t. The human is very much still in the loop, and that’s a context problem.

Join live on Aug 19 (FREE) to learn:

• The 4 metrics to measure where AI gains leak out before production.

• The 8 stages of context maturity, the specific walls capping your metrics, and a free tool to pinpoint where your team is.

• Why more MCPs and bigger context windows aren’t enough, and what it takes to get real value from your agents.

Register Now

(Thanks to Unblocked for partnering on this newsletter.)

§

I want to introduce Kukeshajanth Kodeswaran as the guest author.

He’s the co-founder of modalis and Director of Engineering at Launchpad.AI, where he’s shipped AI systems running in production at Nike, Mr. Cooper, and Levi’s.

If you want to learn to build production-grade AI agents, I highly recommend his live Maven bootcamp: Become an Agentic AI Engineer and Master AI Agents for Business. He teaches it with Dr. Ryan Ahmed - he also co-authored the #1 Claude course on Udemy, and the Maven cohorts are rated 4.9.

(You’ll also get an exclusive 20% discount when you use the link.)

§

Here’s what’s inside this newsletter:

• Why a tiny AI can outperform a large one. The narrow class of problems where a 3B model can beat frontier models, where it falls short, and why size alone doesn’t decide the winner.

• The specialist’s toolbox. Retrieval-Augmented Generation (RAG), Low-Rank Adaptation (LoRA), distillation, and quantization; what each technique solves and when each belongs in your stack.

• Production architecture quietly replacing one-model-fits-all. Where small models handle routine work, where frontier models still earn their cost, and why routing between them changes economics.

• A real benchmark from start to finish. A fine-tuned SmolLM3 model, a legal contract dataset, head-to-head results against GPT-5.5 and Claude Sonnet 4.6, and what the numbers actually mean.

• Decision framework for your next AI project. Which workloads deserve a vertical small LLM, which should stay with a frontier model, and where you’ll spend time and money without seeing better results.

§

What Makes a Small AI Different?

The idea comes down to two words: small & vertical.

A small language model usually has between 1 and 8 billion parameters. i.e., it’s small enough to run on hardware you already own. A 3 billion-parameter model can run on a gaming GPU, and many models in this range even run on a modern laptop. But frontier models are much larger. They run in cloud data centers, and you access them through an API.

A vertical model is trained for “one” domain. Instead of becoming good at every task, it becomes extremely good at one. For example, banking support, contract review, medical Q&A.

Think of it like hiring people:

A frontier model is a general practitioner who knows a little about many subjects. A vertical small model is like a heart surgeon. You wouldn’t ask a surgeon to file your taxes, and you wouldn’t ask a general practitioner to perform heart surgery. Each is built for a different job.

The same idea applies to AI.

If your company workflow keeps making the same decision over & over, a vertical small model can outperform a much larger general-purpose model on that specific task.

IMPORTANT: small doesn’t mean weak… it just means “focused”.

§

Why Companies Build Their Own Small Models

If frontier models are so capable, why do companies build their own small models?

Here’s why:

1. Cost

A frontier model is inexpensive1 for a few requests.

The problem appears when your application handles millions of them every day. Suppose every support ticket, insurance claim, or contract goes through AI. Even a small cost per request adds up quickly.

A self-hosted small model, or an inexpensive hosted one (such as Groq), can reduce inference costs by 25-60 times for these repetitive tasks.

For a high-volume system, this can save thousands or even millions of dollars each year.

2. Speed

Latency matters just as much.

When you call a frontier model, every request travels across the internet to a remote data center before the model can respond. The network trip often takes longer than the model’s actual computation2.

A small 3B model running on your local server avoids this extra trip.

A background check company moved its high-volume background check system from a hosted frontier model to a fine-tuned small model running locally.

This reduced their response times from several seconds to less than a second3.

3. Data Privacy

Sometimes the decision has nothing to do with cost or speed.

Many banks, hospitals, government agencies, and law firms cannot send sensitive customer data to an external AI service because of legal, regulatory, and/or company requirements.

In these cases, running an open model on their own infrastructure lets them keep the data inside their own environment while still using AI.

For these organizations, the question isn’t, “Which model is cheaper?” Instead, “Which model lets us use AI without our data leaving the building?”

Share

§

How Small Model Works

Building a small model is usually much simpler than most people expect…

You don’t train a model from scratch. Instead, you start with a small open base model and improve it using one/more of these techniques4:

1. Retrieval-Augmented Generation (RAG)

The model searches your company’s documents before answering a question.

Instead of relying only on what it learned during training, it also uses the latest information from your knowledge base.

For “most” applications, this is enough. No extra training required!

2. Fine-Tuning

If the model keeps making the same type of decision, you can teach it using “examples” from your data.

The most common approach is Low-Rank Adaptation (LoRA). Instead of retraining all billions of model parameters, LoRA trains only a small set of new parameters while leaving the original model unchanged.

This makes fine-tuning cheaper and faster. A model that normally requires many GPUs can be fine-tuned on a single GPU in about an hour.

(More on this later…)

3. Distillation

A large, powerful model generates high-quality answers.

Those answers then become training data for a smaller model. So the smaller model learns to produce similar results at a fraction of the cost5.

4. Quantization

After the model is trained, you can shrink it by storing its parameters with lower numerical precision, so it fits on smaller hardware.

For example, converting a 7-billion-parameter model from 16-bit to 4-bit precision reduces its size from about 14 GB to around 4.5 GB, with little loss in accuracy.

Putting It Together

A typical production setup looks like this:

1. A small model receives the request.

2. RAG provides the latest company information.

3. LoRA fine-tune teaches the model your specific task.

4. Quantization reduces the hardware needed to run it.

5. If the request is too difficult, it’s forwarded to a larger frontier model.

This approach gives you the speed and low cost of a small model while still having a powerful model available for the few requests that need it.

§

Where Small Models Fall Short

Small models are powerful, but they’re not the right tool for every job:

1. They struggle with complex tasks

A specialized model works best if it repeats the same type of decision over and over.

Give it a large, open-ended task, like designing a software system or writing a complex database migration, and its performance drops quickly.

For example, on real-world coding benchmarks, fine-tuned small coding models score around 20% to 36%, while today’s frontier models score closer to 80% to 88%.

2. They can still hallucinate

A specialized model is not automatically more accurate.

Legal AI systems built specifically for lawyers still invent facts. Popular legal research tools, such as Lexis+ AI and Westlaw AI6, produced hallucinations in 17% to 33% of their answers.

Plus models tuned specifically for medicine gave hallucination-free answers only 51% of the time, while general models hit 77%.

Specialized does not always mean more reliable.

So when a vendor says a domain model is more reliable because it’s specialized, ask for the numbers and evaluate a model on your own data.

3. Fine-tuning has tradeoffs

Fine-tuning teaches a model new skills, but it can also weaken existing ones.

The model would become better at your task while becoming worse at general knowledge/reasoning. Plus, fine-tuning can unintentionally reduce a model’s built-in safety behavior, even when the training data itself is harmless.

Every time you fine-tune a model, you must test it again before putting it into production.

4. Benchmarks don’t tell the entire story

A small model can have a good benchmark score and still perform poorly for your use case.

Sometimes models get a high score because they have already seen similar benchmark questions during training. When researchers evaluate them on completely “new data”, the scores can drop significantly.

So it’s very important to benchmark your own and perform well on the real tasks your users care about

§

How to Combine Small and Large Models

The goal is not to make a small model do everything.

Instead, the goal is to use the correct model for each request.

Most production systems run both a small model and a frontier model together. The small model handles the (90%) common requests, while the frontier model takes over when the task (10%) is too difficult.

Let’s dive in!

1. Model Routing

The simplest approach is model routing.

Every request goes to the (cheap) small model first. If the model is confident, it returns its answer immediately. If its confidence falls below a threshold, the request is forwarded to a frontier model instead.

Open-source model “routers” follow this approach.

They can achieve about 95% of the quality of a frontier model while reducing inference costs by roughly 85%.

2. Speculative Decoding

Another optimization technique is speculative decoding.

The small model generates a draft response first. Instead of generating the entire answer from scratch, the frontier model verifies the draft and corrects it if necessary.

Because checking a draft is faster than writing one from scratch, speculative decoding can improve speed by around 3-4 times while producing the same final output.

3. On-Device AI

The same idea also appears on mobile devices.

Apple Intelligence uses a (3B) small model running directly on the device for everyday tasks. Only requests that need more reasoning/external knowledge are sent to larger models in the cloud.

This reduces latency, lowers cloud costs, and keeps more user data on the device.

4. Measure Before You Ship

None of these techniques matter if you don’t evaluate your model properly.

Don’t rely on public benchmarks alone. Build your own held-out test set7 using real-world examples from your app, and ensure the model has never seen those examples during training.

This is the only reliable way to know whether your model is improving or simply performing well on familiar benchmarks.

(I followed the same process for the experiment later in this newsletter, and I’ll show you exactly how...)

§

§

Where Small Models Work Best

Small models work best when the job is simple, repetitive, and happens thousands or millions of times.

Instead of solving a new problem every time, they make the same type of decision again & again.

Let’s look at some use cases:

1. Support Tickets

A customer sends a support message.

The model decides which team should handle it and forwards it to the right place. The task is always the same; the volume is high, and the potential answers are limited. This is exactly the kind of job where a small model can outperform a large model.

A model with one billion parameters (or less) beat GPT-5 on this task.

2. Contract Review

Many companies use AI to find specific information in contracts.

For example:

• Find the termination clause.

• Extract the liability limit.

• Identify the renewal date.

The model isn’t trying to understand every detail of the contract.

Instead, it’s looking for the same pieces of information in documents that all follow a similar structure. This is another task where a specialized small model can outperform GPT-5.

(I’ll use the example later in this newsletter…)

3. Regulated Industries

Small models are also a good fit for industries such as healthcare, finance, and law.

In many cases, sensitive customer data cannot be sent to an external AI service because of legal, regulatory, and/or company requirements. Running a small model on your own infrastructure keeps the data inside your environment.

These applications also focus on a narrow domain.

The model doesn’t need to know everything on the internet. It only needs to perform well on the specific tasks your organization asks it to do every day.

§

Why Run Your Own Model

Running your own small model isn’t just about saving money.

It also gives you more control over how your AI system works.

Let’s dive in!

Read more

• Share this letter & I’ll send you some rewards for the referrals.

How do you speed up your API?

Every request requires work…Parse the payload, query the database, package the response, then send it back. i.e., each step takes time…

As traffic increases, these tiny delays add up:

• Servers struggle under load,

• Users wait longer for responses,

• Costs increase as you add more machines.

The solution is NOT to write faster code.

Instead, build smarter systems that do less work, handle more traffic, and stay fast under pressure.

Let’s dive into proven ways to build high-performance APIs…

§

§

Here’s what’s inside this newsletter:

• Why latency grows even when your code is efficient. The real bottlenecks hiding in databases, networks, and downstream services, and why scaling an API is mostly about reducing unnecessary work.

• The architecture patterns behind high-performance APIs. What caching, asynchronous processing, connection pooling, load balancing, rate limiting, and circuit breakers each solve, and why production systems combine them instead of relying on one optimization.

• Every optimization has a failure mode. Why stale caches, pool exhaustion, retry storms, payload bloat, and poorly tuned autoscaling can reduce reliability while trying to improve performance.

• Your API is only as fast as its slowest dependency. Why databases, external services, network round trips, and inefficient payloads usually dominate end-to-end request latency, long before your application code becomes the bottleneck.

• Nine prompts to review your API architecture. Audit latency bottlenecks, database access, caching strategy, scaling readiness, observability, and resilience to identify high-impact performance improvements before traffic exposes weak points.

§

1. Caching and compression

If responses didn’t change, you should STOP recomputing them.

This is the core idea behind caching: store responses so you can reuse them later. This skips expensive operations such as database queries, data manipulation, and/or API calls.

Here’s how it works:

1. Your API receives a request X for the first time

   • It does the usual steps: parse, query the database, package it into a response Y, and send the response

   • Right before/after sending, it stores Y somewhere in-memory; Y is now a cached response to request X

2. Your API receives a request X for the second time

   • It checks the cache first and sees that the response Y is already there, so it returns it; this is a cache hit

   • It’s a lot faster because there were NO database queries, API calls, or data manipulation

3. But what happens when the response to X is no longer Y?

   • If the cache serves outdated data, your users see stale results (this is bad)

   • Your API needs to remove the stale data and update it with a fresh copy; this is called cache invalidation

   • There are different techniques for invalidation: time-to-live (TTL), manual removal, cache-control headers

Caching is useful for data that does NOT change often.

Think of requests like profile lookups or product catalog searches. Popular implementations are in-memory caches (Redis, Memcached) and edge caches (Fastly, Cloudflare).

Downsides of caching

• Stale data: cache can serve outdated responses if invalidation is not managed well

• Complexity: invalidation logic adds extra logic when to remove/refresh entries

• Hard to debug: behavior depends on timing, not just code logic

Compression explained

If the responses are too BIG, then make them smaller.

This is the core idea behind compression: shrink the payload before sending it over the network. It removes redundancy, such as repeated characters or formatting. A smaller, but equal result travels faster & uses less bandwidth.

Here is how it works:

1. Your API and the client agree on which compression format to use

   • This process often happens through the Accept-Encoding and Content-Encoding HTTP headers

   • For example, a client requests Accept-Encoding: gzip, and the API replies with Content-Encoding: gzip

2. Your API prepares the response data Y

   • Before sending, it runs Y through a compression algorithm

   • This makes Y smaller, so it uses less network bandwidth & gets sent faster

3. The smaller, compressed version of Y gets sent over the network

4. Client receives this compressed version of Y

   • Before using it, the client runs the response through a decompression algorithm

   • This makes it grow to its original size again

Compression works well for large JSON/binary responses.

Some popular compression formats are Gzip and Brotli. Other options are using Protobuf or Avro formats that reduce payload size by design.

Downsides of compression

• CPU cost: each encode/decode cycle consumes processing power

• Only helpful for large payloads: best used when responses exceed 1KB

• Diminishing returns: for small responses, CPU cost could outweigh network savings

Takeaway

• Caching avoids recomputation and saves processing time, but adds memory cost and invalidation complexity.

• Compression reduces payload size and network latency, but increases CPU usage.

Both speed your API when used selectively & measured against their tradeoffs.

§

2. Load balancing and autoscaling

How do you keep your API fast as demand grows?

Every request your API gets is a “job”.

As traffic grows, one machine can’t do it all. The system slows down, queues form, and users wait. Adding more servers helps, but only if you can share the work and manage them properly.

This is what load balancing & autoscaling do.

Share

Load balancing explained

If one server is busy, you should route new requests to other servers.

That’s the key idea behind load balancing. It spreads incoming traffic across many servers so NO single one gets stuck doing all the work. It’s like a traffic cop directing cars to open lanes.

Here’s how it works:

1. Your API receives a request

2. The load balancer sits in front of all servers and decides where it goes

3. It checks which servers are healthy & how busy they are

4. Load balancer sends the request to the best option

5. That server handles it and returns the response

Load balancing makes your system faster and more reliable.

If one server fails or gets overwhelmed, others take over. Load balancing is great for web APIs, microservices, and high-traffic systems. You rarely build one yourself, but use existing solutions.

Some popular implementations are AWS ELB, HAProxy, and Nginx.

Downsides of load balancing

• Setup overhead: adds configuration & management work

• Single point of failure: can become one without “redundancy”

• Session problems: stateful sessions break when requests go to different servers

• Extra complexity: sticky sessions/shared storage fix sessions, but add complexity

Autoscaling explained

If servers can’t keep up, you shouldn’t add more manually; let the system do it for you.

This is the idea behind autoscaling. It adds/removes servers automatically when the load changes. It’s like calling in extra chefs when a restaurant gets a rush of orders, then sending them home when it quiets down.

Here’s how it works:

1. System tracks metrics such as CPU, memory, and/or request rate

2. When traffic gets high, it adds more servers based on pre-defined rules

3. New servers join & start taking requests

4. When traffic drops, extra servers get shut down

5. The system stays stable & doesn’t waste money

Autoscaling is common in systems with unpredictable workloads & traffic spikes.

Popular tools are Kubernetes HPA & AWS Auto Scaling Groups.

Downsides of autoscaling

• Rule sensitivity: triggering too early wastes resources, while triggering too late hurts performance

• Startup delays: slow server startup affects performance during scaling

• Configuration work: requires tuning trigger rules & thresholds

Takeaway

Load balancing spreads work across servers & prevents overloading.

Autoscaling adds/removes servers in the system and manages system capacity. Together, they make APIs faster, stable, and ready for traffic changes.

§

3. Asynchronous processing

How do you keep your API responsive when some tasks take a long time?

Tasks like sending emails, reading files, or calling services spend a lot of time waiting for input/output (IO).

During the wait, the server isn’t using the CPU efficiently. If your API waits for these long tasks to finish, users experience slow response time. Asynchronous processing solves this problem. It lets the API respond immediately while heavy tasks run in the background.

Here’s how it works:

1. API receives a request that triggers an IO-bound task

2. Instead of processing it right away, it puts a message in a queue

3. The API responds to the client immediately with a job ID/status

4. Background workers read from the queue and process tasks independently

5. Workers notify the client when the job is done (OPTIONAL)

It’s like placing an order at a restaurant and receiving a buzzer:

You can keep doing other things while the kitchen finishes your meal, and the restaurant keeps serving new customers.

Asynchronous processing is ideal for high-traffic systems, unpredictable workloads, or slow IO tasks. It decouples request handling from heavy work, keeping APIs fast and responsive.

Popular tools are RabbitMQ, Kafka & Celery.

Downsides of asynchronous processing

• Increased complexity: must manage queues, workers, retries, and error tracking

• Harder debugging: execution happens outside the main request flow

• Failure handling: workers that fail need to retry/re-queue tasks

• Limited CPU benefit: offers little help for CPU-bound work where waiting isn’t the problem

Takeaway

Asynchronous processing keeps APIs responsive by offloading slow IO tasks to background workers.

It trades simplicity for scalability, making systems faster under load but harder to monitor and debug.

§

Read more

• Share this letter & I’ll send you some rewards for the referrals.

Some of these microservice patterns are foundational, and some are quite advanced. ALL of them are super useful to software engineers building scalable systems.

Curious to know how many were new to you:

1. API Gateway,

2. Database per Service,

3. Circuit Breaker,

4. Event-Driven Messaging,

5. Service Discovery,

6. Retry with Backoff,

7. Decompose by Business Capability,

8. Saga,

9. Health Check API,

10. Distributed Tracing,

11. Service Instance per Container,

12. API Composition,

13. Transactional Outbox,

14. Remote Procedure Invocation (REST/gRPC),

15. Backends for Frontends (BFF),

16. Bulkhead,

17. Idempotent Consumer,

18. Strangler Fig,

19. CQRS,

20. Sidecar/Service Mesh,

21. Anti-Corruption Layer,

22. Event Sourcing.

For each, I’ll share:

• What it is in plain English & how it works,

• A real-world analogy (if I found one),

• Tradeoffs,

• Why it matters.

Let’s go!

§

The most dangerous app in your company was built by someone in finance last Tuesday (Partner)

The number of vibe-coded apps in enterprise environments has skyrocketed.

Yet most vibe-coded apps cannot handle customer data securely, comply with your internal workflows, or pass production processes.

i.e., they “break” your company standards.

So your apps become an IT problem when you ask yourself these questions:

• Where does the data live & who can access the app?

• Where are the audit logs & how to enforce company security policies?

• Can this safely reach production?

INTRODUCING SUPERBLOCKS…

(Instead of cleaning up later, it makes IT the foundation from day one.)

• Apps deploy inside your own AWS account and VPC, so sensitive data stays under your control.

• Company login, role-based permissions, audit logs, and governance get enforced automatically across all apps.

• Every deployment gets scanned for vulnerabilities before it reaches production.

So you build production-ready apps instead of demos that need to be rebuilt later…and your business moves faster.

If you’re serious about enterprise AI app development, try Superblocks right now:

TRY SUPERBLOCKS NOW

(Thanks to Superblocks for partnering on this post.)

§

1. API Gateway

An application programming interface (API) gateway is a single entry point for your services.

A client must “never” call services directly.

The gateway gives them a single address and forwards each request to the correct service. Plus, it centralizes the cross-cutting concerns: authenticates requests, enforces rate limits, terminates Transport Layer Security (TLS), caches responses, and translates protocols.

i.e., your services stay focused on business logic instead of repeating cross-cutting logic across the codebase.

Analogy

A hotel front desk routes guest requests.

You never walk into the kitchen or the laundry room yourself. Instead, you ask the desk, and they route your request to the right department.

i.e., one point of contact hides a dozen moving parts.

Tradeoff

A gateway simplifies clients1, centralizes security and observability, and lets you reshape services behind a stable interface.

But it adds a network hop, and could become a single point of failure if you run only one instance. Plus, it can become a development bottleneck when every team must change it, and it risks becoming a mini-monolith if you add business logic inside.

So run it behind a load balancer for high availability and keep it thin.

Why it matters

Use an API gateway when many services sit behind one public API, mobile and web apps need a single endpoint, or you want to handle authentication and rate limits in one place.

It can also help you move clients from a monolith to microservices without breaking the existing API…

Some popular implementations are Kong, NGINX, Envoy, AWS API Gateway, and Apigee.

TIP: Pair it with the Backends for Frontends2 pattern (more on this in concept 15) when different clients, such as web and mobile apps, need different APIs.

2. Database per Service

Each service has a private database that NO other service directly touches.

A service keeps its own data and exposes data only through its API.

Other services ask for what they need instead of directly reaching into the tables. This keeps services loosely coupled: you can change a schema, switch between databases, or add an index, and nothing breaks…

i.e., database becomes an implementation detail hidden behind the service boundary.

Analogy

A shop with its own stockroom that only its staff enter.

Customers ask at the counter, and the staff fetches what’s needed. Nobody wanders into the back.

Because one team controls the stockroom, it stays organized, and changes surprise nobody.

Tradeoff

Private databases give each team independence, isolate failures, and let each service pick the right storage for its job.

But queries that span services get harder, you lose cross-table joins & foreign keys, and keeping data consistent across services needs patterns such as Saga, API Composition, and Transactional Outbox3.

A shared database looks easier on day one but silently couples every service over time.

Why it matters

Every microservices system has independent deploys, or services with very different data shapes & access patterns.

This pattern makes microservices independent. So treat a “shared database” as a red flag.

Popular data stores are PostgreSQL, MySQL, MongoDB, and DynamoDB.

Share

3. Circuit Breaker

A circuit breaker temporarily stops calls to a failing service, giving it time to recover and protecting callers from cascading failures.

The breaker wraps a remote call & watch failures:

• When calls succeed, it stays closed and traffic flows.

• When failures cross a threshold, it opens, and every call fails fast without touching the failed service.

• After a cool-down, it goes half-open and lets one trial call through.

• If this succeeds, it closes again; if not, it opens & waits longer.

This turns a slow, resource-draining timeout into an instant, cheap rejection.

Analogy

An electrical fuse trips when too much current flows.

Instead of letting a fault burn down the house, the fuse cuts the circuit. Once the problem is fixed, you reset it and power returns.

The breaker does the same for a failing dependency, cutting it off before it drags everything down.

Tradeoff

They add error-handling paths and need careful tuning of thresholds & timeouts.

Plus, they can trip on a “brief” blip and reject good traffic, and require monitoring so you notice an open circuit.

Why it matters

Calls to a remote/third-party dependency must degrade gracefully, or one slow service must NOT take down the whole request path.

Resilience4j (Java), Polly (.NET), and Envoy proxy implement breakers, and a service mesh (more on this in concept 20) such as Istio can apply them without code changes.

4. Event-Driven Messaging

Services communicate by publishing & consuming events through a “message broker” instead of calling each other directly & wait.

A direct call couples the caller to the callee in time: the caller blocks for a reply, and a failure on either side breaks the request.

Event-driven messaging breaks this link. A service publishes an event to a broker and moves on, and any number of consumers can subscribe & react on their own schedule.

This publish-and-subscribe style also lets you add new consumers later without touching the producer.

Analogy

A phone call versus a group channel:

A phone call needs both people free at the same instant. A message posted to a group channel waits until each reader is ready, and everyone interested sees it.

New people can join the group channel later and still get the updates.

Tradeoff

It trades simple request-reply for eventual consistency.

Plus, debugging gets harder across an event chain, and there’s a need to handle duplicate & out-of-order delivery.

Why it matters

Workflows where steps can happen asynchronously, fanning one event out to many services, decoupling producers from consumers, or smoothing traffic spikes with a buffer.

Apache Kafka, RabbitMQ, AWS SNS and SQS, and Google Pub/Sub are popular broker implementations.

TIP: Pair it with the Idempotent Consumer pattern4 (more on this in concept 17), because at-least-once delivery means duplicates will happen.

Share

5. Service Discovery

Service discovery lets services find each other by name through a registry, which updates itself as instances start, stop & move.

In microservices architecture, instances come and go, and their addresses change constantly. i.e., hard-coding a location breaks the moment a service moves.

Each instance registers its address with a service registry when it starts and de-registers when it stops. Callers query the registry by name to get a current, healthy address.

Here are 2 ways to implement service discovery:

• client-side, where the caller queries the registry and load-balances itself,

• server-side, where a load balancer or the platform does it.

Analogy

A contacts app instead of numbers scribbled in pen…

When a friend changes their number, the app updates and you can still reach them. The registry keeps every service’s current contact details, so callers always reach them.

Tradeoff

• It adds infrastructure,

• Makes the registry a critical dependency,

• Needs health checks to stay accurate, and a stale cached lookup can still send traffic to a dead instance.

Why it matters

Fixed addresses are NOT practical for auto-scaling fleets, containerized deployments with shifting IPs, blue-green and rolling deploys5.

Consul, etcd, and Netflix’s Eureka are classic registries, while Kubernetes builds discovery in through DNS and Services.

Most service meshes6 include it, so you often get discovery for free at scale.

6. Retry with Backoff

Retry with backoff reattempts a failed call a few times with a growing delay, turning brief glitches into successes without hammering a struggling service.

Networks blip, and a call that fails once often succeeds on the next try.

A naive retry loop can stampede a service that’s already down. Backoff fixes this by waiting longer between each attempt, usually “doubling” the delay, and jitter adds randomness so many clients don’t retry in lockstep.

A retry limit and a timeout cap the total effort so a stuck call gives up instead of hanging forever.

Analogy

Redialing a busy phone line.

You wait a moment and try again, not forty times in a row, and you give up after a few rings rather than holding the line all day.

Each wait is a little longer so you’re not jamming the line.

Tradeoff

Retrying a non-idempotent call can double-charge or double-write; aggressive retries can amplify an outage into a retry storm, and extra attempts raise tail latency7 for the request.

TIP: Only retry idempotent operations, always add jitter, and pair retries with a circuit breaker.

Why it matters

Calls to flaky networks/third-party APIs, transient database errors, or any remote operation that’s safe to repeat.

Resilience4j & Polly provide retry with exponential backoff and jitter out of the box.

Timeout, retry with backoff, and circuit breaker together are standard resilience mechanisms for remote calls.

7. Decompose by Business Capability

Split a system into services along the natural lines of the business, using business capabilities and domain-driven design8 to draw the boundaries.

The HARD part of microservices is deciding where one service ends and the next begins…

Here are two methods to guide you:

• Decomposing by business capability splits the system by what the organization does, such as orders, payments, or shipping. Then gives each capability a service.

• Decomposing by subdomain uses domain-driven design (DDD) to model the business and carve it into subdomains.

Each becomes a bounded context: a boundary inside which every term carries one precise meaning.

Boundaries drawn this way follow the business, which shifts far more slowly than the code.

Analogy

Think of a company with different departments:

Sales handles selling, Shipping handles deliveries, and Payments handles billing. Each department owns its own work. Even the same word can mean different things. For example, a "customer" means an order and delivery address to Shipping, but payment details & invoices to Payments.

A bounded context keeps each meaning separate, so teams don't interfere with each other.

Tradeoff

Choosing the right service boundaries needs a good understanding of the business.

If you split services the wrong way, they end up constantly calling each other. While changing those boundaries later becomes difficult because each service already owns its own data and responsibilities.

Why it matters

Use this pattern when you’re building a new microservices system, breaking a monolith into smaller services, or giving each team ownership of a specific business area.

§

§

Read more

• Share this letter & I’ll send you some rewards for the referrals.

Every task used to start the same way for me, and you know this feeling:

You sit down to ship something small, and before you write a line, you ask, “Where is this documented?”

The context I needed was never in one place…

So I would dig through scattered READMEs, a stale runbook, an ADR1, and a lost Slack thread, then ask a team member. And when none of it worked, just guess.

None of it was the work itself, and over time, I realized it came down to three problems stacked on top of each other:

• First, the time. I burned hours hunting through docs and GitHub pages before I even started, just trying to find where this structure was explained.

• Then comes trust. So let’s say my AI agent finds something. How do I trust it when AI can hallucinate2?

• And the last one is the scariest: the knowledge never written down anywhere, the kind that lives in one senior engineer’s head and leaves the day they do.

These are real problems, and they cost real time and trust on every team.

The good news is you can solve all three with a single tool called NotebookLM.

So let me show you what it actually is…

[Webinar] 8 levels of context maturity in AI-native engineering (Partner)

AI is in your engineering workflow. While the token spend shows it, the throughput doesn’t. The human is very much still in the loop, and that’s a context problem.

Join live July 23 (FREE) to learn:

• The 8 levels of context maturity: where most teams are stuck and what the ceiling looks like at each stage

• Why more MCPs, rules, and skills provide agents access but not understanding

• How leading engineering teams are using a context engine to make the most of their agents

Register now

(Thanks, Unblocked, for partnering on this newsletter.)

I want to introduce Gencay as the guest author.

He has been building with AI since ChatGPT first shipped in 2022.

For years, he built for clients on Upwork: 7,300+ hours of it.

Now he builds in the open and shares every piece of it on LearnAIWithMe, every Monday, Wednesday, and Friday.

Not just theory for its own sake. One real build at a time, from idea to running system.

Here’s what’s inside this newsletter:

What NotebookLM Actually Is

NotebookLM is a research tool from Google that answers your questions using a set of documents you provide.

The set you give it is called a notebook, just a workspace where you collect the files you want to read, like a folder you can ask questions to.

Picture a sharp new hire on their first day:

They are smart, and they want to help, but they do not know your system yet, so you hand them a stack of your real docs and tell them to work from those. This single constraint is the whole point.

They will not wing it from general knowledge, and when they tell you something, they can point to the page they got it from.

NotebookLM works differently from general chat models like ChatGPT/Claude. Those usually come from broad model knowledge.

Now you might ask, why not just bolt a RAG pipeline3 onto ChatGPT or Claude and be done with it?

Yes, you can, and sometimes it is the right call. But NotebookLM gives you grounding and citations out of the box with nothing to build or maintain.

(I’ll cover the real numbers on this later.)

Under the hood, the core mechanism behind NotebookLM is retrieval and grounding rather than open-ended model memory.

Here’s the workflow:

1. You ask a question in plain language.

2. NotebookLM searches your sources and pulls only the passages that match it.

3. Those passages, and nothing from the open web, go to Gemini4 as the context.

4. Gemini writes the answer from those passages.

5. Each claim comes back with a citation to the exact line it came from.

The technique has a name: retrieval-augmented generation (RAG).

It also leans on the notebook’s chat history while it answers, so a follow-up still makes sense without you repeating the setup.

Plus, it runs on Google’s Gemini models, which Google keeps upgrading, and it can hold a lot at once.

Each source can run up to about five hundred thousand words, and the chat works across a context window of a million tokens. So you can load real documentation instead of a handful of pages.

The one thing to keep in mind is scope.

Each notebook only knows what is inside it. It does not pull from your other notebooks. The boundary sounds like a limitation until you realize it is exactly what makes NotebookLM safe to put underneath an agent.

Enough about what it is… Let’s set one up.

Share

How to set up NotebookLM?

1. Visit here,

2. Click on “Try NotebookLM”,

3. Then click “Create new notebook”.

You have a few ways to feed it.

A web search, file upload, websites, a Google Drive connection, or pasted text.

Pick whichever matches where your material already lives.

For this notebook, I used websites.

I clicked “Websites” and pasted the Backstage links I had pulled together. The architecture decision records, the core docs, and the pull requests.

Here are the exact links I added:

https://backstage.io/docs/architecture-decisions/adrs-adr001/

https://backstage.io/docs/architecture-decisions/adrs-adr002/

https://backstage.io/docs/architecture-decisions/adrs-adr003/

https://backstage.io/docs/architecture-decisions/adrs-adr004/

https://backstage.io/docs/architecture-decisions/adrs-adr005/

https://backstage.io/docs/architecture-decisions/adrs-adr006/

https://backstage.io/docs/architecture-decisions/adrs-adr007/

https://backstage.io/docs/architecture-decisions/adrs-adr008/

https://backstage.io/docs/architecture-decisions/adrs-adr009/

https://backstage.io/docs/architecture-decisions/

https://backstage.io/docs/overview/architecture-overview/

https://backstage.io/docs/backend-system/architecture/index/

https://backstage.io/docs/features/techdocs/architecture/

https://backstage.io/docs/deployment/

https://backstage.io/docs/getting-started/

https://backstage.io/docs/overview/what-is-backstage/

https://github.com/backstage/backstage/pulls

https://github.com/backstage/backstage/tree/master/docs/architecture-decisions

A couple of minutes later, the notebook gets trained and is ready to answer.

From here, the notebook is ready, with every Studio output on the right waiting to build from your sources.

So the notebook is loaded.

But why trust a word it says? This comes down to one idea…

Grounding, in plain language

Here is what grounding looks like in practice:

I asked the notebook about the latest AI model Anthropic released, something nowhere in my sources, and here is its response:

To make the boundary clear, let’s customize the tone, using this prompt:

If the answer is not explicitly supported by the provided sources, respond only with:

Not in source.

I asked the same question after it, and here is the answer:

So I asked the same question again, and this time it just said: “Not in source.” One line in the prompt forced it to stay inside what I had uploaded instead of reaching for what it already knew.

Share

How citations work

If grounding5 is the promise, then citations are the receipt to prove it.

NotebookLM does hallucinate less, and there is a study to point at now:

Researchers ran ChatGPT, Gemini, and NotebookLM over the same 300-document reporting task. NotebookLM made something up in roughly 13% of its answers, against about 40% for the other two on the same sources.

Here is the research report.

Let me show you the notebook we will use for the rest of this newsletter.

It holds Backstage’s Architecture Decision Records. Backstage is Spotify’s open-source developer portal, and its decision log is public. So it makes a clean stand-in for a real team’s knowledge base.

Here is the prompt:

Summarize the key architecture decisions recorded in these sources.
For each decision, cite the specific Architecture Decision Record it comes from.

Here is the answer it returned, and the part worth watching is the citations:

Look at the numbers in the answer…

Each one is a citation. Click one, and NotebookLM opens the source and highlights the line the claim came from.

That is the difference between an answer you hope is right and an answer you can check in two seconds.

So that is what it does well. Just as important is what it does not.

What NotebookLM is not?

A PDF reader will happily open your PDF, but it just shows you the page and stops there.

NotebookLM reads across every source at once and tells you exactly where it looked, so it does much more than a plain reader would.

Also, it is not a full agent running your code. It reads your sources, grounds its answers in them, and cites them, but the actual building still happens in your own tools.

On its own, NotebookLM is a read layer, and a very good one at that. So keep this frame in mind, and you will reach for it at the right time.

You have seen what it is and what it is not, so the next question is how you actually set one up…

Engineering Use Case: NotebookLM as a Knowledge Layer for Code

This is the part I actually run…

The rest of the newsletter explains NotebookLM. This section puts it to work.

The setup is one notebook, loaded with a real engineering knowledge base, and a Claude Code agent querying it.

I am using the Backstage notebook from earlier, since its decision records, docs, and merged pull requests are all public. I loaded them into a single notebook and pointed my agent at it.

How I connected NotebookLM and Claude Code in the first place

We connect NotebookLM and Claude Code through a command-line interface. (It is not official.)

Here is the link: https://github.com/teng-lin/notebooklm-py

You can follow the setup steps on GitHub, or you can just drop this prompt into Claude Code:

Install the notebooklm-py command-line interface from
https://github.com/teng-lin/notebooklm-py
and ping me when I need to authenticate.

It opens the browser. You authenticate once.

From then on, the agent can reach your notebooks.

Now, let me show you how it reaches the Backstage notebook…

Use this prompt:

Using the notebooklm-py command-line interface, connect to my Backstage notebook
and ask it to summarize ADR003 through ADR007
so I can see the decisions at a glance.
Return the answer with the citation for each ADR.

Backstage notebook link: (paste link here)

First, it authenticates, then it summarizes ADR003 through ADR007.

Here it is:

Imagine if accessing the right document while building was this easy. How much time would you save?

Share

How agent queries on every Pull Request

The goal here is simple:

Just catch a change breaking a documented decision before a human spends time reviewing it.

So, on every pull request, the agent reads the diff and then asks the notebook one question via the command-line interface. “Does this change follow our recorded decisions?”

The notebook answers from the architecture decision records, with a citation, and the agent flags or passes on the answer.

Here is the prompt the agent runs:

Using the notebooklm-py command-line interface, query my Backstage notebook.

A pull request adds a new catalog entity kind called “Datastore”.

Its YAML descriptor has no api Version field and no metadata.name.

Based on the architecture decision records and the default catalog file format,
does this change follow our documented conventions?

Answer only from the sources and cite the specific record.

Backstage notebook link: (paste link here)

Now the agent has something concrete to check against, so here is what it flagged:

Look at what came back.

The agent did not just say “this looks wrong.” Instead, it flagged three separate problems and tied each one to a specific record.

It holds no opinion about your architecture. So it asked the notebook, and the notebook answered from Backstage’s real records.

That is what turns architecture review from tribal knowledge stuck in one person’s head into a check that the agent can run on every change.

A concrete example, answered with citations.

The PR check above is the whole idea in one shot. A change comes in, the agent asks the notebook, the notebook answers with the exact records the change breaks.

Take the strongest of the three flags, the Datastore kind. To see what the agent is checking against, ask the record directly:

Using the notebooklm-py command-line interface, query my Backstage notebook.

What does ADR005 decide about catalog core entities,
and would a new “Datastore” entity kind fit those decisions?

Quote the relevant lines and cite the source.

Backstage notebook link: (paste link here)

Here is what the record gave back:

Look at what the record itself says. The agent pulled ADR005 straight from the source, hash and all, and quoted it verbatim.

The catalog stands on three core entities: Components, APIs, and Resources. A datastore is not a fourth kind. It is already a Resource, the same bucket Backstage uses for Bigtable databases and S3. So the proposed change was never going to fit, and the record said so in its own words.

Now the design-doc check. The same question works before a review meeting, when the change is still a proposal:

Using the notebooklm-py command-line interface,
query my Backstage notebook:(paste the link here)

Here is a new design doc proposing a dedicated “Datastore” entity kind for databases.

Does this conflict with any existing architecture decision?

Cite the record it conflicts with.

Here is what came back this time.

Same answer, different door. The first query asked what ADR005 decides.

This one asks whether a new design conflicts with it:

Both land on the same record, the same source hash, the same verbatim lines.

One reads it as a PR check, the other as a design review before the meeting. The notebook does not change its story between the two, because it is reading, not guessing.

Detailed workflows

The same loop ends up covering four different jobs, so it is one notebook and one query pattern paying off in four places:

• PR review. The agent checks the diff against the records and tells you whether the change follows the decisions you already made. Say a new teammate opens a PR that quietly reintroduces a pattern the team dropped a year ago. A human reviewer might not remember the old decision, but the notebook does. So the agent flags it with the exact record, before anyone burns a review cycle on it. This is the Datastore case from earlier, running on autopilot.

• Architecture and design review. The agent runs the design doc against the decision log before the meeting, so a conflict surfaces while the design is still a proposal. You drop in a proposal for a new caching layer. The agent points out that it contradicts a trade-off the team settled on two quarters ago, and it provides the citation. The meeting then starts from there instead of relitigating it live, which is usually where the hour goes.

• New feature spec. The agent pulls the records touching the area you are working in, so the spec starts from what the team has already decided. Before you write a line of the spec for a new auth flow, the agent surfaces the history for you. Every past decision, constraint, and rejected approach in this corner of the system. You stop proposing things the team already ruled out, and the reviewers stop typing the same objections they typed last time.

• Debugging against past incidents. The agent queries old PRs and notes for the same symptom to determine whether the team has encountered this failure before. A service starts timing out under load, so you describe the symptom. The agent finds the same failure from eight months ago, along with the fix that actually worked. Instead of rediscovering it from scratch at 2 am, you are reading the answer someone already paid for once.

Every one of them is really the same move: you ask the notebook, you get a cited answer, and you act on it.

The code layer is where I live, but the same notebook quietly does more.

Let me show you…

Alternative Use Cases for Engineers

Once you wrap it in a Claude routine6, most of it runs without you.

Let me show one, then point at the rest…

The one I run on a schedule: weekly PR reporting

A Claude routine queries the notebook on a schedule.

Every Friday, it checks the week’s merged PRs against our recorded decisions and sends a short report. Which changes followed the records, which drifted, and the citation for each call.

Here is the prompt the routine runs:

Build a routine that runs every Friday at 5pm.

Use the notebooklm-py command-line interface to query my Backstage notebook:
(paste the link here)

Review the merged pull requests in the sources against the architecture decision records.

For each PR, say whether it follows our recorded decisions or drifts from them,
and cite the specific ADR.

Send me a report with one line per PR
and a single summary line for the team lead at the end.

Next, Claude Code built the routine.

Now let’s head to the routine inside Claude Code.

Click on the one you just set up.

Click on “Run now” to test it.

Here is what the first run produced.

The first run came back empty.

The notebook had no merged PRs inside it, only open ones, so there was nothing to check against the records.

It did not invent a result. Instead, it told me the sources held no merged PRs and stopped.

This empty result is reassuring because it shows the workflow tells you when there is nothing to report, rather than inventing something to fill the gap.

So I added a real merged PR as a source and ran it again.

Here is the result:

From here, you can have Claude send this report wherever you want. Slack, email, or any tool you already use.

You have seen what it does. Now here is where it wins and where it does not…

Let’s keep going!

Read more

• Share this letter & I’ll send you some rewards for the referrals.

A general-purpose AI model, even with access to clinical notes, still does not write like a clinician…

Give the model access to legal documents, and it still does not draft like a lawyer…

The vocabulary is familiar, but the format and conventions are NOT present by default.

Ask a foundation model to write up a patient visit, and you get:

“The patient is a 70-year-old man with a history of hypertension
and hyperlipidemia who presents to the emergency department
complaining of substernal chest pain, pressure-like,
radiating to the left arm and jaw, with nausea, diaphoresis,
and shortness of breath.”

While a clinician writes:

“70 y/o M w/ h/o HTN, HLD presents to ED c/o substernal CP.
Pressure-like, radiates to L arm and jaw. + nausea, diaphoresis, SOB.”

Same content…

One is a paragraph; the other is a chart entry. The general model knows the medicine; it doesn’t know how clinicians write it down.

Fine-tuning is how you get them there.

Rather than prompting the model into a style on every call, you train those patterns directly into the model’s weights so they become its defaults. Fine-tuning is how tools like Harvey write like junior associates at large law firms.

• Glass Health produces clinical notes in the format clinicians use.

• BloombergGPT reasons over financial data the way analysts expect.

• GitHub Copilot responds fast enough to feel like autocomplete.

Each of these is a general-purpose model trained in a narrow domain.

This newsletter covers how fine-tuning works, when it is the right choice over prompting or RAG, and how to build it.

Let’s dive in!

[Webinar] 8 levels of context maturity in AI-native engineering (Partner)

AI is in your engineering workflow. While the token spend shows it, the throughput doesn’t. The human is very much still in the loop, and that’s a context problem.

Join live July 23 (FREE) to learn:

• The 8 levels of context maturity: where most teams are stuck and what the ceiling looks like at each stage

• Why more MCPs, rules, and skills provide agents access but not understanding

• How leading engineering teams are using a context engine to make the most of their agents

Register now

(Thanks, Unblocked, for partnering on this newsletter.)

I want to reintroduce Louis-François Bouchard as the author of this newsletter.

He’s a best-selling author (Building LLMs for Production), the co-founder of Towards AI, and the creator of the YouTube Channel, What’s AI, where he helps people understand AI and learn how to apply it in the real world.

Through his development work with clients and his content, teaching, and AI training programs on the Towards AI Academy, Louis focuses on making AI practical for builders, engineers, and curious learners alike.

At Towards AI, he and his team train AI engineers through courses built for every stage, from beginner to advanced. That educational mission and the real-world experience building for his clients are exactly why I wanted him in this newsletter series.

Here’s what’s inside this newsletter:

• What fine-tuning change in a model? Behavior, style, and domain reasoning patterns.

• Low-Rank Adaptation (LoRA) and Quantized Low-Rank Adaptation (QLoRA). The methods that make fine-tuning practical by modifying less than 1% of a model’s weights.

• Data quality. Instruction-response pairs, dataset size, synthetic data, and the gap between training distribution and production.

• The decision framework. When prompting is enough, when RAG is the right addition, when fine-tuning is warranted, and when all three are needed together.

• Alignment beyond Supervised Fine-Tuning (SFT). Reinforcement Learning from Human Feedback (RLHF), Direct Preference Optimization (DPO), model merging, and distillation.

• A hands-on comparison. Three configurations for the same clinical SOAP note task: base model, fine-tuned model, and frontier model ceiling.

Golden members get all letters like these!…

Subscribe now

Why Build Your Own Domain Expert AI?

Models such as GPT-5.5, Claude Opus 4.7, and Gemini 2.5 Pro are foundation models: Large Language Models (LLMs) trained on broad, internet-scale text to perform many tasks reasonably well rather than one task very well.

Their defaults are calibrated for a wide audience, not a specific domain.

For a domain product, this means out-of-the-box quality is just the starting point, and specialist users typically expect more…

• Domain vocabulary and reasoning. A foundation model recognizes terms such as “Transient Ischemic Attack” and “force majeure,” but recognition is not the same as application. It does not consistently apply the National Institutes of Health (NIH) Stroke Scale, the standard scoring system clinicians use to grade stroke severity. It does not draft a force majeure clause the way a specific law firm drafts it. Fine-tuning bakes those specialized patterns into the model’s weights so they appear by default, without requiring them to be prompted on every call.

• Tone and style consistency. Every domain has a house style. Brand voice for marketing, a formal style for legal memos, a structured format for clinical chart notes. Harvey reads like a junior associate at a large law firm because the underlying model has been customized for legal drafting work. Prompting can imitate a style for a turn or two, but the model usually drifts back to its default voice within a few turns.

• Latency and model size. A fine-tuned 7-billion-parameter (7B) model often beats a foundation model on a narrow task. On a single Graphics Processing Unit (GPU), a fine-tuned 7B model can produce its first output token in under 300 milliseconds. This metric is called Time-To-First-Token (TTFT). For a product like code autocomplete, where suggestions must appear before the user types the next keystroke, this latency is the difference between a usable and an unusable product. GitHub Copilot is fine-tuned for exactly this reason.

• Cost at scale. Above a few hundred thousand queries per month (lower against a flagship-plus-RAG configuration, higher against a mini-class API), a self-hosted fine-tuned model is cheaper than paying for foundation-model Application Programming Interface (API) calls. Below this volume, the work required to build the dataset and run training usually costs more than a year’s worth of foundation-model API bills.

• Data privacy. Some training data cannot leave your infrastructure. Patient charts are regulated under the Health Insurance Portability and Accountability Act (HIPAA), the attorney-client privilege protects legal documents, and customer records may be regulated under the General Data Protection Regulation (GDPR) or System and Organization Controls 2 (SOC 2). None of this data can be sent to a third-party API for processing. Open-weight fine-tuning addresses this constraint by keeping the model, training data, and inference traffic within your own Virtual Private Cloud (VPC), behind your own authentication, and with your own audit trail.

Before deciding to fine-tune, you must confirm prompting and Retrieval-Augmented Generation (RAG) do NOT meet your quality bar.

Most people overestimate how much fine-tuning will buy them over a well-engineered prompt combined with retrieval. Fine-tuning is the right answer when one of the five reasons above applies. But it is the wrong answer when you are reaching for it because the prompting felt inconvenient.

Once you have decided to fine-tune, three design questions determine the system: which base model to start with, which dataset to build on, and whether to pair fine-tuning with retrieval.

What We Are Building

The system has four components:

• Base model is the pretrained Large Language Model (LLM) the fine-tuning pipeline starts from.

• Dataset is a set of instruction-response pairs capturing the behavior the model should produce by default.

• Fine-tuning pipeline uses that dataset to shift the base model’s behavior toward the target domain.

• API endpoint serves the resulting model, paired with the retrieval layer from newsletter 3, for use cases where the domain also requires fresh facts.

Requirements

• System must produce outputs in the domain style and format without per-query examples, and apply domain reasoning without Chain-of-Thought prompting on every call.

• It must refuse out-of-scope questions in a calibrated way rather than attempting an answer.

• The entire training run must be reproducible, with a locked dataset, locked random seeds, and a versioned model card.

• A production ship would also verify that general capability regresses by less than 5% on Massive Multi-task Language Understanding, Professional version (MMLU-Pro) or MT-Bench, and that inference cost is at least 50% below the foundation-model baseline. Both gates are described in Parts 3 and 4 but are not run in Part 5’s walkthrough.

• One advanced feature covered later: preference tuning with Direct Preference Optimization (DPO), used to correct behaviors that Supervised Fine-Tuning (SFT) cannot reach reliably.

What this system does not do

It does not call tools or take actions, which is the focus of a future newsletter.

Also, it does not guarantee factual freshness, because the model’s training data is fixed at the point of fine-tuning. For current facts, RAG is still the right layer to add.

Deployment

Part 5 builds the open-weight path and documents the OpenAI fine-tuning path as a hosted alternative.

The open-weight path uses a base model such as Llama, Qwen, or Mistral with LoRA adapters served on vLLM1. The hosted path uses the OpenAI fine-tuning API.

Model selection

The right starting point is the smallest base model that already understands the domain.

Here are three criteria to look for:

1. Run a 20-question domain probe on the base model; if it cannot answer roughly half correctly, fine-tuning will not recover it, because fine-tuning shifts defaults and does not install knowledge the model never had.

2. Confirm fine-tuning support: on the open side, Llama, Qwen, and Mistral all support it; on the hosted side, OpenAI API currently exposes GPT-4.1 and GPT-4.1-mini for fine-tuning, since the GPT-5.x reasoning tier is not available for SFT at the time of writing.

3. Check license fit: Llama community license, Apache 2.0, and the OpenAI Terms of Service each impose different constraints on where the model can be hosted, which determines whether the model can run where your data lives.

Start in the 7B-8B parameter range and scale up only if the domain probe requires it.

Model choice is the last decision you make before training begins.

The next part covers what training actually does to the model you picked…

Part 1: How Fine-Tuning Changes a Model

Pretraining runs next-token prediction over trillions of tokens of text scraped from the internet, books, and code repositories.

By the end of the process, the model has compressed a broad representation of human language and knowledge into its weights. This is the base model: capable across many tasks, specialized in none.

Fine-tuning starts from that base and runs the same training process again, but on a much smaller dataset, typically a few thousand examples, chosen specifically to push the model’s behavior in one direction. The model is not learning language at this stage. Instead, it is learning which of its existing patterns to apply by default.

i.e., fine-tune for behavior, retrieve for knowledge.

Fine-tuning works by reinforcing patterns across many examples until they override the base model’s defaults. A reasoning pattern or output format repeated across thousands of training examples will stick.

A fact that appears once or twice will not, because the model has no reason to weigh it more than the vast body of knowledge it acquired during pretraining.

What Gets Updated During Training

Fine-tuning adjusts every parameter in the model slightly, one example at a time, until its behavior shifts toward what the dataset shows.

The primary cost here is memory:

A 7B model has 7 billion numbers stored on the GPU. Training requires storing a second number alongside each parameter to track how it should change, roughly doubling memory usage to 28 GB.

The optimizer adds a few more bookkeeping values per parameter on top of that, pushing the total GPU memory to 60-80 GB.

Consumer GPUs have 16 to 24 GB of memory, which is why a full fine-tune of a 7B model used to require a cluster and cost tens of thousands of dollars. Parameter-efficient methods (covered later) reduce the requirement to a single 16 GB GPU and bring the cost down to around $10 per run.

The size of the GPU bill forces the choice of training regime; the next decision a team has to make…

Three Training Regimes

There are three ways to fine-tune a model, each using a different kind of dataset:

• Continued pretraining runs the same kind of training the base model already underwent, but on raw text from a single domain rather than the open internet. There are no labels; the model just reads a lot of domain text and gets better at predicting the next word in the domain. The dataset is large, ranging from hundreds of millions to billions of words, and the point of the exercise is to teach the model the field’s vocabulary. BloombergGPT started this way with financial text, and Med-PaLM (a medical fine-tuning of Google’s PaLM, Pathways Language Model) started this way with medical text. You should reach for continued pretraining only when the base model barely understands your domain.

• Supervised Fine-Tuning (SFT), introduced in newsletter 1, trains on a list of questions paired with the answers the model must produce. The model learns to imitate those answers. Typically, the datasets contain 1,000 to 50,000 of these question-answer pairs. SFT is the right starting point for almost every fine-tuning project, because most domain problems are about style, format, and reasoning patterns rather than missing knowledge.

• Preference tuning, which includes both Reinforcement Learning from Human Feedback (RLHF) and DPO, trains on comparisons. For each question, the dataset contains two possible answers and a label indicating which is better. The model learns to produce answers closer to the preferred one. Datasets are smaller, typically 500 to 5,000 pairs. Preference tuning is used to correct behaviors that SFT cannot reliably teach: calibrated refusals, a consistent tone, and output formats that hold up even when users push back.

The practical order is to start with SFT, ship it, identify the specific behaviors that still need work, and then add DPO to address those cases.

Continued pretraining enters the picture “only” if SFT stops improving and the base model clearly does not understand the domain vocabulary.

SFT handles the bulk of the work in a typical project, and the question it raises immediately is how to run it without an 80 GB GPU. The answer is a class of methods called Parameter-Efficient Fine-Tuning (PEFT), which trains only a small fraction of the model’s parameters rather than all of them.

LoRA is the most widely used method in this class…

Low-Rank Adaptation (LoRA)

LoRA leaves the base model’s weights completely frozen and instead trains a small side network alongside it.

The side network is built from two small matrices, conventionally labeled A and B. At inference time, the output of the side network is added on top of the frozen base model’s output.

Only the side network is trained; the base model never changes.

This works because the change required to turn a general model into a domain expert is mathematically small.

You do not need to adjust every parameter in the model; you only need to apply a targeted correction to the parameters that matter. The side network learns the correction. Its capacity is controlled by a setting called the rank, typically 8 or 16, which determines how much the side network can express.

In those settings, LoRA trains fewer than 1% of the parameters full fine-tuning would touch and recovers 90-95% of the quality on most tasks. GPU memory drops by an order of magnitude because the optimizer only tracks the small side network, and the risk of degrading the base model remains low because its weights never change.

Yet LoRA has two trade-offs against full fine-tuning:

• First, on most tasks, it matches full fine-tuning in quality, but not always at parity. On hard tasks, full fine-tuning is, on average, a few percentage points stronger.

• Second, the side network has a fixed capacity ceiling set by the rank: a rank-16 adapter can only encode so much change to the base model. For large domain shifts, such as moving a general model toward dense medical or legal reasoning, LoRA hits this ceiling and stops improving with more data.

Full fine-tuning has no such ceiling, but it incurs the highest GPU cost and carries the greatest risk of the model forgetting its general capabilities.

It is worth considering only when LoRA has plateaued, and the GPU budget allows it.

Reminder: this is a teaser of the subscriber-only newsletter series, exclusive to my golden members.

When you upgrade, you’ll get:

• Simple breakdown of real-world architectures

• Frameworks you can plug into your work or business

• Proven systems behind ChatGPT, Perplexity, and Copilot

Unlock Full Access

Ready for the best part?

Quantized Low-Rank Adaptation (QLoRA)

Read more

• Share this letter & I’ll send you some rewards for the referrals.

In the early days, running apps was simple.

Most applications ran on a single server, and managing them was quite easy. Then microservices became popular. Instead of a single app on a single server, companies were running hundreds of apps across hundreds of servers.

But managing this became extremely difficult.

Engineers had to decide where apps should run, what to do when apps crash, how to handle more traffic, and how to update apps without downtime. Most people handled this with scripts and manual work. It worked for a while, but as systems grew, system started breaking more often, and downtime became more common.

Google faced this problem on a massive scale.

So they built an internal system called Borg. Borg was used to manage apps across thousands of servers. It automates scheduling, scaling, and self-healing across many servers. Years later, Kubernetes (k8s) was built based on many of the same ideas as Borg and got released as an open-source system. Since then, it has become the standard platform for running distributed systems and microservices.

Kubernetes could look simple from the outside, but within the cluster, many components constantly work to keep your apps running.

Let’s look at what is happening behind the scenes…

The most dangerous app in your company was built by someone in finance last Tuesday (Partner)

The number of vibe-coded apps in enterprise environments has skyrocketed.

Yet most vibe-coded apps cannot handle customer data securely, comply with your internal workflows, or pass production processes.

i.e., they “break” your company standards.

So your apps become an IT problem when you ask yourself these questions:

• Where does the data live & who can access the app?

• Where are the audit logs & how to enforce company security policies?

• Can this safely reach production?

INTRODUCING SUPERBLOCKS…

(Instead of cleaning up later, it makes IT the foundation from day one.)

• Apps deploy inside your own AWS account and VPC, so sensitive data stays under your control.

• Company login, role-based permissions, audit logs, and governance get enforced automatically across all apps.

• Every deployment gets scanned for vulnerabilities before it reaches production.

So you build production-ready apps instead of demos that need to be rebuilt later…and your business moves faster.

If you’re serious about enterprise AI app development, try Superblocks right now:

TRY SUPERBLOCKS NOW

(Thanks to Superblocks for partnering on this post.)

I want to introduce Ayaan as a guest author.

He’s a Certified Kubernetes Administrator (CKA) who writes about Kubernetes, cloud infrastructure, and cloud-native systems. He builds in public and shares practical tips, projects, and lessons learned along the way.

Check out his work & socials:

• Twitter

• Github

Here’s what you’ll find inside this newsletter:

• The mental model that makes Kubernetes click. The reconciliation loop, what it is, why it matters, and why most Kubernetes confusion comes from not understanding it early enough.

• A complete breakdown of the Kubernetes architecture. Every component in the control plane and worker nodes, what each one does, how they talk to each other, and what breaks when any one of them goes wrong.

• From kubectl apply to a running pod, step by step. The exact sequence of events inside the cluster from the moment you hit enter to the moment your app is live.

• Every core Kubernetes object explained. Pods, deployments, daemonsets, statefulsets, services, ingress, secrets, what each one does, when to use it, and when people reach for the wrong one.

• What Kubernetes genuinely gets right and where it gets complicated. An honest look at the trade-offs most explainers skip over, including etcd as a single point of failure, the complexity floor, and the hidden cost nobody talks about.

• When NOT to use Kubernetes. The answer most posts won’t give you.

Most Kubernetes posts teach you how to use it…

This newsletter is about how it actually works, because that’s the part that makes everything else make sense.

Kubernetes Architecture Overview

To understand Kubernetes, we first need to look at its architecture.

Then we can dive into the individual components…

Kubernetes architecture is a distributed system:

It doesn’t run on just one server. Instead, it operates on a group of servers that work together, known as a Kubernetes Cluster. These servers could be virtual machines or bare metal servers. The components of Kubernetes are distributed across these servers over a single network.

At the highest level, the Kubernetes cluster consists of two key components: control plane and worker nodes.

We will go through each of these components in detail later...

Control Plane Node

Think of the control plane as the brain of the Kubernetes cluster.

It manages global decisions for the cluster. Plus, it schedules apps, checks system health, and ensures the cluster remains in the desired state.

The control plane contains these components:

• kube-apiserver

• etcd

• kube-scheduler

• kube-controller-manager

• Cloud-controller-manager

Worker Node

Imagine the worker node as the muscle of the Kubernetes cluster.

It’s responsible for running workloads (containerized applications) in the cluster. And it contains these components:

• kubelet

• kube-proxy

• Container runtime

i.e., control plane decides what runs where; worker nodes run the containers.

Kubernetes continuously checks and fixes the system. This ensures everything runs as expected…

A cluster can have many control plane nodes and worker nodes; this usually depends on the number of workloads to be deployed.

In production systems, it’s typical to have many control plane and worker nodes. This setup keeps the system stable. It also ensures applications run smoothly, even if some machines fail or are unavailable.

Plus, Kubernetes has a strong, decentralized structure. It automatically moves workloads to healthy nodes upon node failure. This keeps your system running smoothly.

This makes Kubernetes the foundation of a reliable and self-healing distributed system.

Desired State and Actual State

One of the most important ideas in Kubernetes architecture is the concept of desired state and actual state.

When we deploy an application to Kubernetes, we don’t start it in the cluster ourselves. Instead, we tell Kubernetes what we want the system to look like; for example, we could say we want three replicas of an application running.

This description of what we want is called the desired state.

The actual state is what is currently running in the cluster.

Kubernetes always checks the desired state against the actual state. If it finds a difference, it automatically tries to fix it.

For example, if one app crashes or a node fails, the actual state no longer matches the desired state. Kubernetes detects this and automatically creates a new app on healthy nodes to bring the system back to the desired state. The ongoing task of checking and fixing the system is called the reconciliation loop.

This is a key concept in how Kubernetes operates.

Now we understand Kubernetes architecture, let’s learn the control plane and worker nodes…

Share

Kubernetes Control Plane Core Components

We already saw how the control plane makes global decisions for the cluster, but it does NOT do that as a single component.

Instead, control plane is divided into five components that work together to manage the entire cluster state.

These components have their own specific roles:

• One component is the central hub for all cluster communication.

• Another stores the cluster data.

• One makes scheduling decisions.

• Another continuously monitors cluster health and keeps it in the desired state.

• And the last one integrates Kubernetes with your cloud provider to manage resources such as load balancers, routes, and storage volumes.

Let’s look at each of the control plane components and the concepts behind each:

1. Kube-Apiserver

The kube-apiserver is central to the Kubernetes control plane.

It serves as the key hub of the cluster, exposing the Kubernetes API. Almost every operation in the Kubernetes cluster goes through the API server. The other cluster components don’t talk directly with each other. Instead, they use the kube-apiserver to read and update the cluster state. This lets the parts work independently but still connect through a central system. Also, when you interact with Kubernetes, you do not interact with the Kubernetes nodes and their components directly.

Instead, you interact with it via kube-apiserver.

It handles your requests and updates the cluster state.

All messages between the API server and other components are encrypted using TLS1. This keeps unauthorized users out and protects communication within the cluster.

i.e., API server is central communication hub of Kubernetes; without it, the cluster cannot function.

Key Responsibilities of Kube-Apiserver

1. Exposes cluster API endpoint: It’s the only component in the control plane that the outside world can talk to. When you run a kubectl command (kubectl is the command-line tool to interact with Kubernetes), it sends a request to the API server over HTTPS. When your app gets deployed in the cluster, it’s talking to the API server.

2. Single entry point for all operations: All requests to the cluster go through the API server. This includes requests from you and from internal Kubernetes components. Components do NOT communicate directly with each other. They talk only to the API server. Because of this, API server always has a complete picture of what is happening in the cluster.

3. Authentication and Authorization: It checks who is making the request. Then, it confirms if they have permission to take the action. If either check fails, the request stops right there.

4. Notifies the rest of the system when something changes: Once a change is saved, API server notifies every interested component.

5. Stores cluster state in etcd: Kubernetes keeps all cluster details in etcd. This is a fast, lightweight database made for storing cluster state. Nothing reads or writes to it directly except the API server. Every change has to go through it first, this keeps the data clean and consistent.

Since the API server stores all cluster state in etcd, etcd becomes the source of truth for the entire Kubernetes cluster.

Let’s now look at etcd and understand its role in the control plane…

2. Etcd

etcd is a distributed key-value store (distributed database) that Kubernetes uses to store the entire cluster data.

It keeps track of what is running, what should run, and how each Kubernetes component is set up. Since it stores all data about the cluster, it is also called the single source of truth for the Kubernetes cluster.

Kubernetes API server stores all information ever received in etcd.

The scheduler and controller manager monitor the API server for changes. They help keep the cluster in its desired state. The API server reads data from etcd. This makes etcd the central storage system for the entire cluster.

etcd is a key part of the Kubernetes control plane.

It’s crucial to back it up regularly in production settings. If etcd data is lost, the cluster loses its state. So Kubernetes won’t know what applications are running, what nodes exist, or what the system’s desired state is.

i.e., etcd is the database that stores everything about the cluster.

Key Responsibilities of Etcd

• Stores cluster state: All Kubernetes objects are stored in etcd. This includes Pods, Deployments, Services, Nodes, ConfigMaps, and Secrets.

• Acts as the source of truth: The entire cluster state is stored in etcd, and all the components in the cluster rely on this data to make decisions.

• Enables cluster recovery: Backups of etcd can be used to restore the entire cluster state in case of failure.

Also, etcd is the only StatefulSet component in the control plane.

i.e., etcd must be carefully managed and backed up, as it contains the entire cluster state.

Now we know how Kubernetes stores its cluster state; let’s explore how it decides where to run applications…

3. Kube-Scheduler

The kube-scheduler decides which node will run a pod.

A pod is the smallest unit you can deploy in Kubernetes. It acts as a wrapper for one or more containers. Whenever a pod is created, it remains in the cluster without a node being assigned to it.

The scheduler always checks for these pods.

It assigns them to the right node based on available resources and pod setup.

Let’s learn how the scheduler works:

1. Watches for unassigned Pods: The kube-scheduler monitors the API server. It checks for pods that haven’t been assigned to any node after creation.

2. Gets the list of available nodes: It gets the information about all worker nodes and their available resources.

3. Filters Nodes: The scheduler removes nodes that don’t meet the pod's requirements. This includes:

   • Not enough CPU or memory

   • Not enough GPU

   • Node is under maintenance

   • Pod needs a GPU, but the node lacks one

4. Scores for remaining nodes: After removing some nodes, it scores each remaining node. It considers resource availability, scheduling policies, load balancing, and so on. Then, it selects the node with the highest score from the filtered list.

5. Updates the Pod information: The scheduler updates the Pod object via the API server and assigns the selected node.

i.e., scheduler does not run the pod itself; it only decides which node the pod should run on.

Once the scheduler assigns a Pod to a node, kubelet on that node is responsible for actually running the container.

4. Kube-Controller-Manager

The kube-controller-manager is part of the control plane.

It monitors the cluster and ensures the actual state matches the desired state. It runs multiple controllers, each handling a specific task in the cluster. These controllers monitor the cluster’s state using the Kubernetes API.

If they find a mismatch between the desired and actual state, they work to correct it.

To understand this better, let’s look at an example:

Imagine an application on Kubernetes set to run three replicas. This is the cluster’s desired state. But one of the replicas crashes, and the actual state becomes 2 instead of 3. The controller will detect this difference and will automatically create a new replica of the application to make it three again. This is how the controller continuously monitors the cluster state and matches it with the desired state.

Kubernetes is self-healing and highly automated because it continuously monitors the cluster. It resolves differences between the desired and actual states.

Here is a list of important built-in Kubernetes controllers:

• Node Controller: Detects node failures and responds when nodes become unavailable.

• Replication Controller: Ensures the correct number of Pod replicas are running.

• Deployment Controller: Handles rolling updates and ensures Deployments maintain the desired state.

• Endpoint Controller: Updates endpoints for Services when Pods change.

• Namespace Controller: Manages namespace lifecycle and resource cleanup.

• Service Account Controller: Creates and manages service accounts and tokens.

• Job Controller: Ensures jobs are completed successfully.

All the controllers in Kubernetes work as a control loop.

i.e., it continuously monitors the cluster state via the API server and compares the actual state with the desired state. Kubernetes has a control loop feature that’s key to its design.

This allows for self-healing, auto-scaling, and rolling updates.

5. Cloud Controller Manager (CCM)

The kube-controller-manager manages internal cluster tasks.

But Kubernetes must also connect to cloud providers for resources.

This includes:

• Load balancers for services

• Storage volumes for persistent data

• Virtual machines for nodes

The cloud controller manager handles this.

The cloud controller manager helps Kubernetes stay cloud-agnostic. It does this by keeping cloud-specific logic separate from the main Kubernetes components. Kubernetes can run on various cloud providers, such as AWS, Google Cloud, and Azure. It can also work in on-premises environments. Best of all, there’s no need to change the core Kubernetes code.

The cloud controller manager connects Kubernetes to the cloud provider’s infrastructure.

We have looked at the control plane components that manage the cluster and make decisions. But the control plane does not run applications itself. The actual applications run on the worker nodes.

So let’s look at the worker node components and see how they work…

Read more

• Share this post & I’ll send you some rewards for the referrals.

You have more than one AI agent working on the same project:

A few coding agents in your terminal, a chatbot in your browser, an agent deployed on AWS.

Each keeps its “own” memory.

i.e., your terminal agent doesn’t know what you told the browser chatbot 10 minutes ago. And the browser chatbot doesn’t know what’s happening in your editor.

And you copy decisions between chats and carry context1 by hand…

So a single “shared memory” every agent can read from & write to is missing…

I’ll use Omnigraph as the case study for this newsletter.

(It’s an open-source graph engine built for solving this exact same problem.)

Here’s what’s inside this newsletter:

• Why shared folders and vector databases fail. The hidden problem that appears the moment many AI agents write to the same memory.

• Why graph-shaped memory wins. How nodes, edges, and schemas let agents build knowledge instead of disconnected text.

• Object storage without database transactions. How Omnigraph achieves atomic writes on Amazon S3.

• Git for AI memory. Why branches, commits, merges, and policy checks stop agents from overwriting each other or turning mistakes into shared truth.

• One query, three retrieval methods. How graph traversal, keyword search, and semantic search combine to assemble better context for every AI agent.

• When graph memory is the wrong tool. The workloads where a traditional database, a graph database, or a vector database is still the better choice.

Onward.

Why traditional memory breaks

One “shared” memory every AI agent can read from and write to sounds simple.

Yet the obvious ways to build it don’t work well…

Let’s start with two simple approaches:

1. Shared document folder

Every agent stores text in notion, google drive, confluence, or a wiki.

Yet the folder has NO structure an agent can reason over…it’s just a pile of text.

If you ask, “Which deploy touched this service, or who shipped it?”, the agent can answer it off a single file just fine. But it becomes extremely difficult when the answer is spread across 200+ files…

Then agent doesn’t know deploy, service, and person are different concepts. Plus, it has to read everything to be sure it didn’t miss the one file that mattered... or fall back to the classic habit of hallucination2.

A folder stores text,,, NOT knowledge.

2. Vector database

Ask the vector database3 a question using retrieval-augmented generation4 (RAG).

It returns the chunks of text that most closely match your question…This is useful for “find me the passage about X.”

But a vector database holds NO relationships or types.

It can’t start at the customer; find their open tickets, connect them to the engineer on call, and then to the commit that probably caused it. Instead, you get only a bag of similar-sounding snippets, and you need to connect the dots yourself.

So neither one allows an agent to start with one fact and follow it to another related fact…

Even if you ignore these issues, everything falls apart the moment a second agent writes to the same place…

A shared folder has no way to coordinate concurrent writes. If two agents save the same document, the last write wins and the other disappears.

A vector database does NOT distinguish between a proposed change and an accepted one. Once an agent writes new information, every other agent can retrieve it immediately.

i.e., one agent’s mistake becomes every agent’s input (shared context). And agents often get things wrong, so letting one write straight to the shared memory is extremely dangerous.

A “shared” memory for AI agents must do three jobs:

• Read: Assemble context across connected facts. Start from one fact, follow its relationships, and build the context an agent needs.

• Write: Persist new knowledge so it survives after the task ends.

• Coordinate: Allow many agents to read & write simultaneously without overwriting each other or exposing incomplete changes.

Reads and writes aren’t new…but coordination is…

Traditional memory systems assume a few deterministic writers making short, careful, and predictable updates.

But AI agents behave differently:

They reason for minutes, revise earlier conclusions, make ‘mistakes’, and run alongside many other agents touching the same knowledge.

Shared folders & vector databases are NOT designed for this type of workload…

Omnigraph’s core idea is simple:

AI agents don’t need a better folder or a vector database. Instead, a memory system designed for the way they read, write, and coordinate.

I’m happy to partner with Omnigraph on this newsletter.

When dozens of agents share one graph, things get chaotic fast: updates collide, bad writes land immediately, and context drifts.

Omnigraph gives every agent an isolated branch of the same typed graph, so agents can enrich data in parallel while humans deliberately review, approve, and merge changes. Built on Lance, Arrow, DataFusion.

Graph state stored in object storage. MIT.

Check out the repo

Graph based memory

So what should this memory look like?

Imagine facts as dots and relationships as lines connecting them.

• A customer is a dot.

• Each of their tickets is a dot.

• Lines connect the customer to each ticket.

• The engineer who owns a ticket is a dot with a line to it.

• The commit that probably caused the bug is another dot, with a line to the ticket it broke.

This is a graph: dots are nodes & lines are edges.

To answer a question, the system doesn’t search thousands of documents for clues. Instead, it follows the existing edges:

Customer → Ticket → Engineer → Commit 

i.e., four direct hops through connected data.

The graph stores relationships alongside the facts themselves.

This is exactly what shared agent memory has been missing…

Graph schema

A graph alone is NOT enough.

Once many AI agents start writing to the same graph, a new problem appears…

Imagine three agents recording the same production event:

• One writes “deploy”,

• Another writes “deployment”,

• And the third writes “release”.

To you, those are different words with the same meaning…but to the graph, they’re three different nodes.

(Nothing tells the graph they represent the same concept…)

Now when an agent asks, “Show me every deploy this week and who shipped it.”

It follows the deploy nodes and returns only part of the answer. Every deployment and release sits elsewhere on the graph, so the query “silently” never reaches them.

The result looks complete… but it’s WRONG!

The graph didn’t fail,,, but the schema did.

Without a “shared” schema, each agent creates its own vocabulary. The graph stores exactly what each agent writes, even when different terms describe the same concept.

The solution is to define the graph before anyone writes to it.

A schema specifies:

• Which node types are allowed to exist,

• Which relationships connect them,

• How each node is identified.

Every agent writes against the same schema, so they all describe the world in the same way.

So the graph stays consistent, even when hundreds of agents write to it.

Omnigraph schema

In Omnigraph, a schema is a plain text file:

You do NOT write it from scratch.

Instead, the agent generates it from your graph and updates it as your knowledge evolves. And the file describes the graph itself.

Each node defines a type of entity, such as Person, Service, or Deploy. The @key specifies how each node is uniquely identified; for example, a person’s name, a deploy ID, and so on.

Each edge defines a relationship between two node types and the direction an agent can traverse it.

Deploy -> Service literally means “from the deploy to the service it touched.”

Every agent reads the same schema before it writes. This keeps the graph consistent even when many agents update it simultaneously.

So the memory isn’t just a graph…it’s a “typed” graph with a shared schema.

The next question is where this graph lives…

Where memory lives

The graph lives in object storage.

Think of it as a data bucket in the cloud. Some popular examples are Amazon S35, Google Cloud Storage (GCS), and Cloudflare R2.

Modern web apps use object storage for backups, logs, images, videos, and data lakes.

Why Omnigraph chose S3

Omnigraph uses AWS S3 for storage.

Here’s why:

1. Low cost

Amazon S3 storage costs only a few cents per gigabyte each month.

AI agents accumulate knowledge over time. And object storage lets memory grow without the cost of running a large database cluster.

2. High durability

Once data is stored, Amazon S3 replicates it across many availability zones (AZ) within a region to provide very high durability.

This is crucial because the graph becomes your agents’ shared memory and system of record.

3. Shared access

An object storage bucket is NOT tied to one machine.

Every agent with the right credentials can read from and write back to the same bucket. Your coding agent, browser chatbot, and background agents all share “one” memory instead of maintaining separate copies.

i.e., there’s no need to copy state between separate agent data stores.

Yet an object storage has one key limitation:

It stores one object safely at a time, but it can’t commit several objects as a single all-or-nothing operation. A database calls this an atomic write.

Write object A, then B, then C, and the bucket treats them as three independent writes. There’s NO built-in way to say, “Commit all three or none.”

And a graph update almost always touches “many” objects…

So how do you get atomic writes from storage that doesn’t support them?

How atomic writes work

To understand this, you first need to see how Omnigraph stores the graph in the S3 bucket…

How Omnigraph stores the graph

The graph is NOT stored as a single file.

Instead, Omnigraph splits it into many separate stores. This design is what makes atomic writes (all-or-nothing writes) possible.

Take the schema from before:

Each node type & edge type has its own store.

And each store contains every record of that type in versioned files. Under the hood, Omnigraph uses an open columnar file format6 called Lance, yet you never interact with it directly.

So schema above becomes five stores, plus a smaller store called the manifest.

Also, a store is NOT a single file.

Think of nodes/Deploy/ as a folder that keeps growing as new deploys arrive…

(This detail becomes important in a second…)

One update touches many stores

An update to a graph almost always writes to several stores.

This is where object storage turns into a limitation…

Record a single deploy: Alice pushes deploy d-913 to the checkout service.

Omnigraph has to write:

• a new row in nodes/Deploy/ for deploy d-913

• a new row in nodes/Person/ for Alice, if she doesn’t already exist

• a new row in edges/TriggeredBy/ linking d-913 to Alice

i.e., write to three separate stores for one logical update.

And to the bucket, those are just three independent writes…

If the process crashes after writing the Deploy and Person stores but before writing the TriggeredBy edge, the graph becomes inconsistent. Then deploy exists, but nothing says who triggered it…

Besides, object storage can’t roll those writes back because it never knew they belonged to the same logical update.

Manifest tracks live versions

The manifest is the smaller store…It does NOT contain graph data.

Instead, it’s a small table whose only job is to record, for every other store, which version of it is live right now.

It follows these two simple rules:

Rule #1 Stores are immutable

A write “never” modifies a store in place.

A write to nodes/Deploy/ creates a new version alongside the old one. Both versions exist in object storage simultaneously. And the old version remains live (agents keep seeing it) until the manifest says otherwise.

Rule #2 Every store has exactly one live version

The manifest stores a pointer to the current version of each store.

It looks like this:

Now check the deploy from earlier…

The update touches three stores,,, so the manifest has to move three pointers to their new versions.

Publishing a graph update

Here’s the complete (all-or-nothing) write sequence:

1. Omnigraph writes the new store versions without touching the manifest.

2. It creates Deploy v8, Person v12, and TriggeredBy v4 in the bucket.

3. Those versions already exist in object storage, but they’re invisible to agents. Every agent still reads Deploy v7, Person v11, and TriggeredBy v3 because the manifest hasn’t been updated.

4. The final step is to update the “manifest” file.

A single manifest write moves all three pointers at once:

• Deploy → v8

• Person → v12

• TriggeredBy → v4

Thus from an agent’s perspective:

• Before commit: entire old graph is visible (consistent)

• After commit: entire new graph is visible (consistent)

• During commit: there is no partial state (consistent)

The only atomic operation is the update to the manifest file.

If the process crashes before the manifest changes,,, nothing becomes visible. The new store versions remain orphaned files in object storage, while every agent continues reading the previous graph version.

Preventing conflicting updates

What happens if another agent updates the graph before your commit becomes visible?

Omnigraph solves this problem using compare-and-swap7 (CAS).

Before updating the manifest, it checks whether every store is still on the version the writer originally read.

Suppose a writer starts from Deploy v7 and creates Deploy v8.

Before publishing the update, Omnigraph asks one question:

Is Deploy still on version v7?

• Yes. Nothing changed, so the manifest update succeeds.

• No. Another writer already published a newer version. The commit then gets rejected, nothing becomes visible, and the writer retries from the latest state.

Compare-and-Swap simply compares the expected version with the current version before swapping the manifest pointer.

The manifest file is append-only.

i.e., it never modifies older entries. Instead, each successful commit simply adds a new set of live pointers.

CAS prevents one writer from accidentally overwriting changes it never saw.

Proven approach

Omnigraph isn’t the first system to solve this problem…

Apache Iceberg and Delta Lake8 (open table formats) use the same pattern to achieve atomic commits on object storage.

They write new data files first, keep them invisible, then publish them with a single metadata update. It’s a proven design to power production data platforms.

Atomic commits solve one problem:

They guarantee every graph update is complete or NOT visible at all.

But another challenge remains…

What happens when many AI agents want to change the graph at the same time? And how do you stop one bad write from becoming shared memory?

How agents write safely

Compare-and-Swap (CAS) prevents “conflicting” writes:

If two AI agents update the same store at once, one commit succeeds, and the other retries. So the graph remains consistent.

But CAS doesn’t decide whether the write is correct…

AI agents can hallucinate, misread context, or make wrong inferences. If a bad write wins the CAS check, it immediately becomes part of the shared graph.

And then every other agent reads it as fact.

Omnigraph solves this problem using a workflow similar to the Git version control system:

You don’t edit the shared repository. Instead, you create a branch, make your changes, review those, and then merge them. (Omnigraph applies the same idea to graph memory.)

i.e., AI agents don’t write directly to the shared graph, but to branches first.

Let’s dive in!

Every agent gets its own branch

Agents “never” write directly9 to main branch (shared truth).

Instead, each agent creates its own branch from the latest graph and writes there. This branch is private. And the agent can add, modify, or delete data without affecting anyone else’s view. So main branch remains unchanged.

Imagine two agents working on the same graph; each creates their own branch:

They work in PARALLEL without blocking each other, while the main branch remains unchanged…

Branching does NOT copy the entire graph:

A store gets copied only when an agent writes to it for the first time. Until then, the branch shares the same underlying storage as main branch.

Branching is therefore “inexpensive”.

Every merge needs approval

A branch is ONLY a proposal.

It doesn’t become part of main until it’s merged10.

But every merge must be “approved”. You can approve it yourself, or define policy rules that automatically approve routine, low-risk changes from a trusted AI agent (into a scratch branch). Yet larger/riskier changes go through human review.

The key idea is simple:

An AI agent never approves its own changes.

(Let’s return to policy rules later.)

Now, let’s understand what occurs on merge conflicts…

Merge conflicts

Most branches merge automatically.

If two agents change “different” parts of the graph, Omnigraph keeps both changes.

Imagine Agent A updates one deploy, while Agent B adds a new person. Those changes don’t overlap, so the merge succeeds…

Conflicts occur only when two branches modify the “same” field differently:

Suppose Agent A changes deploy d-913 to rolled-back, while Agent B changes the same field to succeeded.

Omnigraph won’t guess which value is correct11. Instead, it reports the exact conflict: node, field, and both competing values. Then a human or a policy rule decides which one wins.

Also graphs introduce another kind of conflict:

One branch can delete a node while another adds an edge pointing to it. And merging both would leave a dangling edge.

Omnigraph detects this conflict before the merge completes, preventing the graph from becoming inconsistent.

i.e., every disagreement becomes an explicit conflict and NOT a silent overwrite of data.

Proven pattern

Omnigraph is NOT the first system to use branches & merges for data.

Dolt brings Git-style branching and merging to SQL databases. lakeFS applies the same idea to data lakes.

Omnigraph extends this pattern to typed graphs, giving AI agents a safe way to propose, review, and merge changes into shared memory.

When many AI agents share the same graph, updates can conflict, incorrect writes can spread instantly, and shared context can drift over time.

Omnigraph gives each agent an isolated branch of a typed graph, allowing agents to work in parallel while humans review, approve, and merge changes before they become shared truth.

Built on Apache Arrow, Lance, and DataFusion. Stores graph state in object storage. Open source under the MIT License.

Check out the repo

Managing graph updates

A safe write still leaves three questions open:

• Who is allowed to make this change?

• Who actually made the change?

• How do you undo the change later?

Omnigraph answers all three with one object: commit.

Every accepted change becomes a commit on main. This commit records who made the change, governs how it entered the graph, and preserves the history to reverse it later if needed.

Authorization layer

Omnigraph itself is the engine—the core that stores the graph, merges branches, executes queries, and manages commits.

You can reach it in three ways:

• over HTTP through the server,

• through omnigraph command-line tool,

• through an SDK embedded in your own app.

AI agents usually connect through a Model Context Protocol12 (MCP) server, which talks to the HTTP server.

It might seem natural to enforce permissions at the HTTP layer. But it protects only one entry point. Plus, the command-line tool and SDK bypass the HTTP server completely.

So Omnigraph enforces permissions inside the engine.

Every write passes through the same authorization check, regardless of whether it came from HTTP, the command line, or an SDK.

There’s NO path to the graph that bypasses the permission system.

What the policy controls

Every authorization check runs against a policy…

Imagine policy as a set of rules that defines who can perform which actions. These are the same policy rules that decide whether a branch can be merged.

The actions are specific:

• read the graph,

• make a change,

• create a branch,

• modify a branch,

• merge into main,

• change the schema.

A policy could allow one actor to merge into main while allowing another to read the graph but never modify it.

Omnigraph writes these rules in Cedar, an open policy language from AWS.

How Omnigraph verifies identity

Every policy rule applies to an actor.

So the first step is identifying who the actor really is. Each actor receives a token when it’s created (similar to a service API key). And every request from the actor includes that token.

The engine then identifies the actor from the token, NOT from information in the request itself.

A request can claim, “I’m an admin.” Yet the token decides whether that’s true…

If the token belongs to a regular actor, the request gets treated as a regular actor every time. Trusting the request rather than the token would allow anyone to claim any identity.

What permissions can control

Omnigraph enforces permissions at the branch level…

A policy can answer questions such as:

• Can this actor13 create a branch?

• Can this actor write to this branch?

• Can this actor merge into main?

Yet it can’t answer more granular questions, such as whether an actor can update one field of a node while leaving every other field untouched. For example, “can this actor edit a deploy’s status without touching its owner?”

(Field-level permissions are on Omnigraph’s roadmap…)

So today the permission system controls access to branches, NOT individual fields.

Commit history

Authorization answers one question:

“Who is allowed to make a change?”

So the next logical question is:

“Who actually made it?”14

Every accepted change becomes a commit on main.

The good news is that commit records:

• who (actor15) made the change,

• when it was committed,

• its parent commit.

Chain those commits together, and you get the history of the entire graph…

Every change has an author, a timestamp, and a place in the commit history.

This chain is the “audit” trail.

Omnigraph does NOT need a separate audit log because the commit history already records every accepted change in order.

Reverting graph updates

Omnigraph never updates data in place.

Every change creates a new version alongside the old one. The manifest simply points to the new version, while the old version remains in object storage.

This is what makes every change REVERSIBLE…

Every historical version of the graph still exists.

You can read the graph exactly as it looked before the merge, inspect what the system knew at that point in time, or compare it with the current state.

Nothing needs to be “reconstructed” because nothing got overwritten…

Rolling back works the same way.

A branch can be discarded, and a merged change can be reverted by creating a new commit that restores an earlier state. The previous versions are still there, ready to be read.

Crash recovery

A write can FAIL halfway through…

New store versions could already exist, while the manifest still points to the previous graph.

The next time Omnigraph opens the graph for writing, it automatically recovers the interrupted operation. It either completes the pending commit or rolls it back.

This recovery gets recorded as its own commit… Yet this commit uses a reserved system identity instead of a human or AI agent, making automatic recovery easy to distinguish in the commit history.

You can immediately tell whether a change came from a human, an AI agent, or the system itself from the audit trail.

How one query searches structure, keyword, and meaning

Every write in Omnigraph gets saved atomically, gets checked before it is shared, and gets protected against crashes & conflicting updates…

Yet an AI agent is only as good as the context it receives.

And the context almost “never” comes from a single lookup:

To answer a question, an agent usually has to assemble related information. Some of it comes from relationships, some from exact keywords, and some from semantic meaning.

The engine has to search all three…

1. Relationships

Start at the LoginService node and follow its edges. Which services depend on it? Which incident involved it? Who owns it?

That’s graph traversal…The engine walks relationships which already exist.

2. Exact keywords

Sometimes you know exactly what you’re looking for:

• the error string ERR_AUTH_443,

• a service name,

• a ticket ID.

That’s full-text search,,, similar to searching a log file with grep.

A graph traversal can’t answer those queries efficiently.

3. Semantic meaning

The agent searches for “can’t log in,” but the relevant incident is titled “authentication failure.”

The words are different… but the meaning is the same…

And that’s what vector search is designed for.

A straightforward solution is to build three separate systems:

• Graph database for relationships,

• Search index for keywords,

• And a vector database for semantic search.

But this approach creates a synchronization problem.

Each system keeps its own copy of the data…

If one index falls behind, they stop agreeing. The graph knows about a node that hasn’t been indexed yet, or the vector database still returns an outdated version of it. Your application now has to query all three systems and combine the results itself.

So Omnigraph takes a different approach…

It stores ONE copy of the graph in object storage and builds three indexes on top of it:

• graph relationships,

• full-text index,

• vector index.

Every index gets built from the same underlying data, so they stay in sync.

Here’s what happens when an AI agent sends a query:

• Omnigraph searches graph, full-text index, and vector index.

• Each search returns its own ranked list of results.

• The engine then combines those lists into one final ranking.

It doesn’t average the scores because each search method uses a different scoring system.

Instead, Omnigraph ranks the results from each search independently, then merges those rankings using Reciprocal Rank Fusion (RRF).

i.e., results that appear near top across many searches rise to the top of final list.

Under the hood, Omnigraph combines several retrieval techniques:

• BM2516 for keyword search,

• Vector k-nearest neighbors17 (k-NN) with cosine similarity for semantic search,

• Fuzzy matching for approximate matches,

• Reciprocal Rank Fusion18 (RRF) for combining results.

You don’t have to know any of these techniques yourself or how to even use them, because the agent using Omnigraph is going to handle it for you…

When to use graph memory

Graph memory solves a specific problem.

Multiple AI agents need to read shared context, write new knowledge, and coordinate those changes without corrupting the shared graph.

Here are some use cases:

• Company knowledge graph where agents read and update decisions, ownership, and relationships.

• Long-term agent memory that survives after each task ends.

• Codebase graph that coding agents continuously read and update.

• Personal knowledge graph managed by one or more AI agents.

The common pattern isn’t the domain…

But multiple agents working on the same knowledge over time.

When NOT to use graph memory

Graph memory is NOT the right tool for every workload…

The biggest limitation is “write” latency.

Omnigraph stores data in object storage. Small writes typically take tens to hundreds of milliseconds, while an in-memory database can respond in well under a millisecond.

It’s a fair tradeoff for AI agents that already spend seconds reasoning…

But it’s NOT a good fit for workloads that require thousands of low-latency writes per second, such as payment processing, click counters, or real-time session state.

Those belong in a transactional database…

You also don’t need graph memory if you only need one retrieval method…

• Need semantic search? Use a vector database to keep it simple.

• Need graph traversal for a single application with one writer? A traditional graph database such as Neo4j is a better fit.

• Need high-throughput transactions? Use a transactional database.

Final thoughts

AI agents turned out to be a new kind of memory user.

They need a memory system built for how they actually read, write & coordinate.

Omnigraph is a typed graph on shared object storage, where every change is proposed, reviewed, and merged before it reaches shared memory.

The agents are already here…And the memory is finally catching up.

Thanks to Omnigraph for partnering on this newsletter.

When dozens of agents share one graph, things get chaotic fast: updates collide, bad writes land immediately, and context drifts.

Omnigraph gives every agent an isolated branch of the same typed graph, so agents can enrich data in parallel while humans deliberately review, approve, and merge changes. Built on Lance, Arrow, DataFusion.

Graph state stored in object storage. MIT.

Check out the repo

Louis & I launched the GENERATIVE AI MASTERCLASS (newsletter series exclusive to PAID subscribers).

When you upgrade, you’ll get:

• Simple breakdown of real-world architectures

• Frameworks you can plug into your work or business

• Proven systems behind ChatGPT, Perplexity, and Copilot

👉 CLICK HERE TO JOIN THE GENERATIVE AI MASTERCLASS

If you find this newsletter valuable, share it with a friend and subscribe if you haven’t already. There are group discounts, gift options, and referral rewards available.

Subscribe now

Want to reach 210K+ tech professionals at scale? 📰

If your company wants to reach 210K+ tech professionals, advertise with me.

Thank you for supporting this newsletter.

You are now 210,001+ readers strong, very close to 210k. Let’s try to get 211k readers by 10 July. Consider sharing this post with your friends and get rewards.

Y’all are the best.

Share

References

• ModernRelay, Omnigraph

• ModernRelay, Omnigraph Cookbooks

• Apache Iceberg, Table Spec

• Delta Lake, Protocol

• DoltHub, Dolt: Git for Data

• lakeFS, Data Version Control for Object Storage

• Cedar, Cedar Policy Language

• Lance, Columnar Data Format

• Share this post & I’ll send you some rewards for the referrals.

How does Netflix stream videos worldwide in milliseconds,,, while some sites take “seconds” to load a basic HTML page?

The answer is proximity…

i.e., closeness between the servers and end-users.

Imagine a single server to handle global traffic:

• International users wait 300ms+ just for the first byte from the “distant” server

• Origin server1 handles all requests from every continent

• One traffic spike could take down the entire site

Onward.

Here’s what you’ll find inside this newsletter:

• Why websites slow down. The hidden costs of distance, overloaded servers, and bandwidth limits.

• Why CDNs became essential. How they evolved from a niche scaling solution into core internet infrastructure.

• What happens when you visit a site. The journey your request takes before content reaches your browser.

• The building blocks of a CDN. Edge servers, origin servers, DNS, Anycast, GSLB, and the components behind global content delivery.

• Where CDNs make the massive impact. The industries and applications that depend on them every day.

• Why CDNs aren’t a silver bullet. The trade-offs, limitations, and operational challenges every engineer should understand.

Read more

• Share this post & I’ll send you some rewards for the referrals.

AI agents can use tools1 today.

Model Context Protocol (MCP) provides an agent with a standard way to communicate with databases, APIs, files, and external services.

But there’s a problem MCP doesn’t solve:

What happens when you need many agents to work together?

Onward.

Build stronger data foundations for “agentic AI” at scale

Building an AI demo is EASY.

Building production-ready agentic AI systems that scale across an enterprise is extremely “difficult”.

That’s exactly what senior enterprise leaders from AWS, Prudential Insurance, Siemens, GAF, and HF Sinclair discuss in this virtual panel: how to build resilient, scalable data foundations and production-ready AI agents.

Here’s what you’ll learn:

• Scale AI beyond the proof of concept. Build industrial-scale systems with classical and generative AI.

• Fix the data problem. Power AI with governed, high-quality data.

• Make better architecture decisions. Choose the right databases for AI applications.

• Streamline deployment. Discover, procure, and deploy AI solutions with AWS Marketplace.

Watch the AWS panel for free right now and see how enterprise leaders are building AI systems with speed, governance, and control.

Watch AWS Virtual Panel for FREE

(Thanks to AWS for partnering on this post.)

I want to introduce Eric Roby as a guest author.

He’s a senior backend and AI engineer focused on building real-world systems and teaching developers how to do the same. He runs the YouTube channel codingwithroby, where he focuses on scalable backend architecture and integrating AI into production applications.

Through his content and courses, he helps engineers go beyond tutorials, think in systems, and develop the skills that actually matter for senior-level roles.

Find him on:

• LinkedIn

• Substack

Here’s what’s inside this newsletter:

• Why MCP wasn’t enough. The missing coordination layer that prevents AI agents built by different vendors and frameworks from working together.

• How AI agents actually collaborate. Agent Cards, task lifecycles, delegation, streaming, discovery, and the protocol that standardizes agent-to-agent communication.

• The emerging AI interoperability stack. How A2A, MCP, RAG, AGNTCY, ACP, and AP2 fit together, and why the industry is converging around open standards.

• Where A2A breaks down in production. Latency, trust, authorization, debugging, observability, and the engineering tradeoffs you’ll face when building multi-agent systems.

• A practical guide to building with A2A. Security, Agent Card design, interoperability, production architecture, and the patterns that make agent ecosystems reliable at scale.

Get Instant Full Access Now

(Golden members get all letters like these!…)

💡 Quick refresher

AI agent is a system that uses a large language model2 (LLM) to reason, plan, and take actions on your behalf. It’s not just answering questions. But also calling tools, checking results, and deciding what to do next. Think of it as an LLM with hands.

A single agent can be brilliant at one job…

They can write code, analyze data, and schedule meetings. But real work crosses domains. And you need one agent to code, one to test, and one to ship. If you stack all of this into one mega-agent, it becomes slow, expensive, and brittle.

i.e., specialization wins, just as it does on a real team…

But then, what is Agent-to-Agent (A2A) protocol?

Can’t agents already communicate through normal API calls?

The problem is every agent-to-agent integration becomes custom: one agent needs to know what another agent can do, what inputs it expects, how to authenticate, how to hand off a task, how to track status, how to pass context, and how to receive the final result.

A2A is an emerging open standard for the coordination layer…

It gives agents a shared way to advertise their capabilities, accept tasks, exchange messages, report progress, and return outputs.

i.e., A2A standardizes the workflow in which one agent delegates work to another.

Imagine A2A as the next connection layer:

• TCP/IP3 lets machines talk to each other

• HTTP lets documents link to each other

• APIs let services compose into systems

• MCP lets agents connect to tools

• A2A does the same thing for agents

Google launched A2A in April 2025.

(They partnered with 50+ tech companies and service providers. 150+ organizations joined them later.)

To understand why A2A matters, you need to understand the pain it solves…

Share

AI Agent Collaboration Problem

AI agents are exponentially growing…

Every team, every vendor, every platform is shipping them…but each one is a SILO.

Your customer relationship management (CRM) agent can’t coordinate with your internal data agent. Your coding agent can’t delegate to a testing agent built by a different team. They might all live in the same company, in the same building, on the same network…

Yet they still can’t work together.

Framework4 lock-in is part of the problem.

An agent built with LangGraph can’t directly communicate with one built with CrewAI, Google’s ADK, or AutoGen. Each framework has its own internal orchestration model. While cross-framework collaboration requires custom glue code, written and maintained by you.

That's where A2A comes in.

Now let’s look at what it is…

What A2A Actually Is

Here’s a simple analogy for you:

Think of A2A like hiring a contractor. You don’t need to know what tools are in their truck or how they manage their schedule. You hand them a job; they confirm what they can do, and then hand back the finished work. A2A is the standard contract format. This lets any contractor work with any client. They don’t need to renegotiate from scratch each time.

The Agent2Agent (A2A) protocol is an open standard.

It helps AI agents find each other’s skills5. They can send messages, delegate tasks, and work together safely. This all happens NO matter their framework, vendor, or platform.

Here are five design principles on which the A2A v1.0 specification is built:

• Simple. Reuse existing, well-understood standards. HTTP, JSON-RPC 2.0, Server-Sent Events. No new transport layer to learn.

• Enterprise Ready. Address authentication, authorization, security, privacy, tracing, and monitoring by aligning with established enterprise practices.

• Async First. Designed for long-running tasks and human-in-the-loop interactions. Quick queries to deep research that takes hours or days.

• Modality Agnostic. Support text, audio, and video via file references, structured data, and forms, and embedded UI components.

• Opaque Execution. Agents work together by sharing their skills and information. They don’t need to reveal their internal thoughts, plans, or how their tools work.

The last one is the most important.

A2A treats agents as autonomous, opaque entities. You don’t need to share your agent’s internals to participate in the ecosystem.

Now let’s open the hood…

Share

How A2A Works Under the Hood

Here are the key points you should understand:

• How agents connect,

• What they share,

• And how they communicate.

Let’s dive in!

Client-Server Model and Agent Discovery

Every A2A interaction has two roles:

• Client agent is the requester. It discovers remote agents, sends tasks, and coordinates the workflow.

• Remote agent is the executor. It receives and processes tasks, does the work, and returns results.

Any agent can play both roles, depending on the workflow.

A coding agent is a client when it delegates testing to a QA agent. The same coding agent becomes a remote agent when a project manager agent asks it to write code.

Agents discover each other through an Agent Card.

This JSON metadata document is published by every A2A server6. It describes the agent’s identity, capabilities, skills, endpoint URL, supported data formats, and authentication needs.

Agent Cards live at a standard, well-known URI: /.well-known/agent-card.json.

The path follows RFC 86157. This is the same way browsers find SSL verification files. A client agent checks another agent’s card. It does this to see what the agent can do before sending requests.

This is like MCP’s capability negotiation8, but for agents instead of tools.

Here’s what an Agent Card includes:

• Name, description, version

• Service endpoint URL

• Supported input and output modalities, as MIME types

• Skills, with IDs, descriptions, tags, and examples

• Authentication requirements, aligned with OpenAPI security schemes

• Supported A2A features like streaming, push notifications, and extended cards

Agents can also serve an Extended Agent Card behind authentication.

This lets you hide sensitive skills or internal capabilities from unauthenticated discovery. i.e., public-facing summary first, full details after you prove who you are.

Tasks, Messages, and Artifacts

Communication in A2A is task-oriented.

When a client agent sends a message, the remote agent creates a Task. This is the fundamental unit of work in A2A. Plus, every task has a unique ID and progresses through a defined lifecycle.

Here are 9 possible lifecycle states:

• submitted: task received by the remote agent

• working: agent is actively processing

• input-required: agent needs more information from the client

• auth-required: agent needs credentials or authorization to proceed

• completed: task finished successfully (terminal)

• failed: task could not be completed (terminal)

• canceled: task was canceled by the client (terminal)

• rejected: agent decided not to perform the task (terminal)

• unknown: state recovery scenario

This is fundamentally different from a simple function call.

Tasks are stateful and can involve many message exchanges. They can also pause to ask the client for more information…Or pause to request authorization before doing anything destructive.

This is the kind of back-and-forth a real collaboration requires…

• Messages are the units of communication between agents. Each message has a role (user or agent) and contains one or more Parts.

• Parts are the smallest content units. Text parts hold plain text. While File parts hold files or images via URI or inline bytes. And Data parts hold structured JSON. This supports rich, multimodal communication without forcing everything into text.

• Artifacts are the outputs of a task. Documents, images, structured data, or any other deliverable remote agent produces. Artifacts, like messages, consist of Parts. They can be streamed as they are created.

• Contexts group related tasks together via a contextId. This provides continuity across a series of interactions, similar to a conversation thread.

Communication Patterns and Protocol Bindings

Before we get into transport details, there are two orchestration patterns to understand:

• Centralized orchestrator. One lead agent owns the plan. It discovers specialist agents, delegates tasks, and stitches the results back together. This makes it easier to reason about and debug. The default pattern for most enterprise workflows today. This is the “tech lead delegates to ICs” model.

• Decentralized swarm. No single orchestrator. Agents find each other directly and share tasks. Each agent chooses whom to contact next based on its own judgment. More flexible. More resilient. Harder to trace. Closer to a marketplace or a team of peers.

A2A supports both.

It’s a protocol, not an orchestration model. The protocol just defines how any two agents talk. The topology is your choice. Most real systems begin centralized. They switch to swarm patterns9 when they can see well and have trust systems to handle complexity.

Plus A2A supports three interaction patterns at the network level:

• Synchronous request-response. Client sends a message and waits for a complete response. Good for quick tasks.

• Streaming via Server-Sent Events. Server streams incremental updates in real-time. Good for tasks where you want to show progress.

• Push notifications. Server sends async updates to a client webhook10. Good for long-running tasks where the client doesn’t want to hold a connection open.

And for the actual wire protocol, A2A v1.0 supports three bindings:

• JSON-RPC 2.011: original and most commonly used binding. Methods include message/send, message/stream, tasks/get, tasks/cancel, and others.

• gRPC: added for high-performance scenarios. Uses Protocol Buffers for message serialization.

• HTTP REST: uses standard HTTP verbs and URL patterns like /v1/message:send and /v1/tasks/{id}. Lowers the barrier for teams that prefer REST over JSON-RPC.

All three bindings share the same semantic model and abstract operations.

The choice of binding is a deployment decision, not a design one. As of v1.0, the protocol’s main data model is in Protocol Buffers12 (a2a.proto). This file is the single source of truth. All bindings derive from this definition.

This familiarity with existing web standards is intentional.

A2A works with your current enterprise setup. It connects easily to load balancers, API gateways, and observability tools. No new tools are needed!

Now that you understand how A2A works, the obvious question: how does it relate to MCP?

A2A vs MCP: Complementary, Not Competing

Read more

• Share this post & I’ll send you some rewards for the referrals.

I’ve been using Claude Code for more than half a year now…daily, for actual work.

For most of the time, I ignored a hidden folder that controlled everything Claude did.

Every session, I copied the same instructions: setup, preferences, and what to avoid1. And Claude kept asking permission for things I’d already approved. So when I opened a new project, it knew nothing.

None of it was necessary…

Claude Code2 is Anthropic’s AI in your terminal.

It reads code, writes files, and runs commands. When you install it, a hidden folder appears: .claude. This folder controls everything: Claude’s instructions, permissions, reusable commands, and session memory3. Configure it right, and Claude walks into every project already knowing the rules.

With the launch of Cowork4, there’s another folder on your machine doing similar work. Same idea, but a different surface. Claude keeps your routines, outputs, and memory5 outside your conversations.

This newsletter gives you a complete map: what’s inside both systems, what each file does, and how it all fits together…

Onward.

Review the actual change, not the file list (Partner)

AI writes more code than ever. Reviewing it shouldn’t mean scrolling forty files in alphabetical order.

CodeRabbit Review reorganizes any pull request from a flat file list into a structured, layer-by-layer walkthrough - the logical reading order of the change, not the order your platform happens to sort it. Every range gets its own plain-language summary, with sequence diagrams, state machines, and ERDs generated inline wherever a visual earns its place.

Cohorts group related files, and chunks so you review one idea at a time. Layers order them so foundational changes - data shapes, contracts - come before the code that depends on them. Code Peek lets you click any variable, function, class, or type to see its definition and usages without leaving the tab, while Semantic Diff view cuts past formatting noise to show what actually changed.

Open it straight from the Review Change Stack button in the PR Walkthrough. Navigate cohorts and layers from the keyboard, comment against exact line ranges, and submit native reviews, comments, and approvals post back to GitHub or GitLab right where your team expects them.

In the early access, available for free to everyone.

From the team that pioneered AI code reviews. 2M reviews every week. 6M repos. 15K customers. The review interface built for the way modern PRs are actually written.

>>> Review your next PR with CodeRabbit Review Today

(Thanks to CodeRabbit for partnering on this post.)

I want to introduce Ilia Karelin as the guest author.

He comes from a data background, data science, analytics, and BI, and brings that analytical mindset to Prosper In AI, his newsletter, where he constantly experiments with AI tools and workflows to help professionals adopt AI in ways that actually improve their work and lives.

Follow him on LinkedIn as well.

Here’s what you’ll find inside this newsletter:

• The hidden operating system behind Claude Code. What every file in .claude/ actually does, why CLAUDE.md is only one piece of the puzzle, and how Claude decides what to load before you type a word.

• Configuration layer nobody explains. Instructions, permissions, commands, hooks, memory, and MCP servers--what each one controls and where they fit in the stack.

• How Claude really works under the hood. The startup sequence, memory system, permission checks, tool pipeline, and why understanding this flow makes debugging dramatically easier.

• Claude Code vs Claude Desktop vs Cowork. Where configuration lives, what carries across sessions, what stays local, and why the same workflow behaves differently on each platform.

• The mistakes that quietly break Claude. Bloated CLAUDE.md files, stale memory, overpowered permissions, weak commands, broken hooks, and the patterns that make Claude ignore your rules.

Get Instant Access

What lives inside

Every file inside the folder falls into one of three categories: instructions, permissions, or reusable commands.

Open a Claude Code session, and this folder is what it reads first. Only then does it act…

💡 Quick note for context:

• Claude Desktop is the native app: conversational, no file access.

• Cowork6 is the visual workspace: live artifacts, local folders, and autonomous runs.

Yet neither uses the .claude/ folder the way Claude Code does.

They follow the same philosophy: persistent context, reusable actions, and behavior controls. But files live in different places and are managed differently.

(I’ll cover this later.)

Now we know what’s inside, let’s look at exactly where everything lives…

The folder map: what lives where

Here’s the first thing that trips people up…there are two .claude folders:

• ~/.claude/ lives in your home directory (/Users/yourname/.claude). This is your “global” Claude configuration. It applies to every Claude Code session you run, across all projects.

• .claude/ lives at the root of a specific repository or working directory (inside a project). Everything here applies only when you’re working inside the project.

Think of these 2 folders as layers…

Claude loads the global config first, then the project config on top of it. i.e., project settings “override” global ones where they conflict.

Here’s what’s in the global folder:

And here’s what you’ll find inside the project folder:

NOTE: CLAUDE.md does NOT live inside .claude/ folder. Instead, it sits at the root of your project.

Think of it this way:

• .claude/ stores Claude’s settings and behavior rules7

• CLAUDE.md stores instructions and project context

They work together, but they are separate files…

Plus, .mcp.json lives at the project root.

Model context protocol (MCP)8 servers allow Claude to connect to external tools such as databases, APIs, and services.

• If you want MCP tools available everywhere, configure them globally in ~/.claude/settings.json.

• But if you want MCP tools for just one project, use .mcp.json inside the project instead.

This matters because different MCP setups can make Claude behave differently across projects.

You can commit the project .claude/ folder to git. i.e., everyone on your team gets the same Claude setup when they clone your repository.

You should commit files like:

• settings.json

• commands/

But do NOT commit settings.local.json. It stores machine-specific paths and personal settings that usually break on other machines.

Claude used to support CLAUDE.local.md for personal instruction overrides… But it’s now deprecated. Today, the recommended approach is to use @ imports inside your main CLAUDE.md.

Also, rules/ helps you split a large CLAUDE.md into smaller files by topic.

For example:

• architecture.md

• mistakes.md

• conventions.md

Claude automatically reads every .md file inside rules/ when a session starts. No imports or extra setup needed. You just add the file, and Claude uses it.

You can also make rules apply only to certain folders.

For example, rules/api-conventions.md can load only when Claude works inside src/api/.

This helps when different parts of the project follow different rules…

TAKEAWAY

Commit these files:

• CLAUDE.md

• .claude/settings.json

• .claude/commands/

• .claude/rules/

So when a new engineer clones the repo, Claude already knows the project rules, restricted files, and available commands.

But do NOT commit .claude/settings.local.json. Instead, add it to .gitignore because it stores personal settings. It works only on one machine.

With the folder structure mapped out, let’s dig into the most important file: CLAUDE.md.

Share

CLAUDE.md: instructions layer

CLAUDE.md is not documentation for humans, but instructions for Claude.

Regular docs explain the project to developers: what the app does, how to install it, and how the API works. While CLAUDE.md tells Claude how to behave on a project: what rules to follow, what conventions matter, and what files to avoid.

When you start a session, Claude loads every CLAUDE.md file it can see, in this exact same order:

1. ~/.claude/CLAUDE.md - your global instructions

2. ./CLAUDE.md - instructions for the current project

3. Any CLAUDE.md files inside subfolders you are working in

They stack together. i.e., each file adds more instructions without replacing the earlier ones.

You can also split large instructions into separate files using @ imports. For example, @docs/architecture.md. Claude will automatically load the file at the start of the session.

This keeps your main CLAUDE.md short while still giving Claude access to deeper project details.

IMPORTANT: imported files still count toward Claude’s context limit. Splitting files improves organization, but it does not reduce the amount of context Claude carries into the session.

Here’s what belongs in CLAUDE.md:

1. Architecture rules Claude cannot infer

Some rules are invisible in the code itself.

For example:

• “Never import directly from utils.ts.”

• “Always use the shared API client.”

• “Do not edit generated files.”

Claude will NOT reliably discover these patterns on its own... So write them down.

2. Environment and workflow context

Include the information a new engineer would spend days piecing together:

• which test runner you use

• how deployments work

• where database configs live

• what setup steps usually fail

• which scripts matter

3. Hard boundaries

Tell Claude what it should never touch without permission.

Use strong markers like:

• IMPORTANT

• YOU MUST

Anthropic has confirmed emphasis markers improve adherence to instructions.

What does not belong

Do not fill CLAUDE.md with:

• long essays

• duplicated documentation

• theory

• things already obvious from the code

Here’s a simple rule:

If removing a line would not change Claude’s behavior, delete it.

CLAUDE.md has no hard length limit. Yet longer files weaken instruction reliability. As the file grows, instructions compete for attention9. And rules near the bottom become easier to ignore than rules near the top…

A 200-line CLAUDE.md is usually not thorough and noisy.

Here’s a tiny but effective CLAUDE.md:

# Project Context

This is a personal finance tracker.
Users log expenses; the app categorizes them.
Run locally with: `npm start`. Tests: `npm test`.

## Rules

- IMPORTANT: never store real financial data in test files
- All database queries go in /db - not scattered across other files
- Don’t touch the /legacy folder - it’s kept for reference only

## Common tasks

- Adding a new feature: check /docs/architecture.md first
- Running into a bug: look in /logs before changing any code

Claude already knows what the project does, what files to avoid, and where to look before changing anything.

Here’s the fastest way to start: run /init inside a Claude Code session. Claude scans your project, detects things like the build system and test framework, then generates a starter CLAUDE.md for you.

That is usually much better than starting from a blank file.

One distinction matters here: prompts and configuration are not the same thing.

Use prompts for one-time requests:

• “Explain this simply.”

• “Summarize this as bullet points.”

You can use CLAUDE.md for rules that should apply every session:

• coding conventions

• hard boundaries

• project context

• workflow rules

Here’s a simple test to help you: If you repeat an instruction in every new session, then put it in CLAUDE.md.

Instructions are only half the picture. Now let’s look at the skills and commands that make Claude reusable…

Skills and commands: reusable workflows

Read more

• Share this post & I’ll send you some rewards for the referrals.

The following is the third of a premium 3-part1 newsletter series… where everything comes together for mobile system design: architecture, how the platform behaves, and what happens after you ship.

On with part 3 of the newsletter:

===

In this newsletter, we’ll learn 14 concepts that shape how production apps are built and operated:

40. Modular Architecture/Feature-Based Modules,

41. Backend for Frontend (BFF) for Mobile,

42. App Versioning & Backward Compatibility,

43. Feature Flags & Remote Configuration,

44. Deep Linking for Navigation & Growth,

45. Navigation & Screen Flow Architecture,

46. Permissions Model,

47. Accessibility (a11y),

48. Localisation & Internationalisation (i18n),

49. Graceful Degradation,

50. Geo Location Usage,

51. Crash Reporting & Performance Monitoring,

52. Crash-Free Rate as an SLI,

53. Staged Rollouts & Safe Releases.

For each concept, you’ll learn:

• What it is and how it works

• Real-world example

• Tradeoffs

• Why it matters for mobile

Onward.

Do you want to become an AI Engineer in 2026?

AI Engineer is LinkedIn’s #1 fastest-growing job, with many roles paying $180K–$350K.

The challenge isn’t learning AI tools.

It’s learning how to design, build, and deploy AI systems that work in production.

That’s why Saras AI Institute created the Advanced AI Engineer Certificate.

Co-designed with AI leaders from Microsoft (Eduardo Kassner, Chief Data & AI Officer) and Google (Tomas Pfister, Head of AI Research @ Google Cloud).

In 5 months, you’ll build a portfolio of 4 real-world AI projects, including agentic workflows, RAG systems, LLM applications, and production-ready AI solutions.

• Turn your AI product idea into a production-ready solution with hands-on mentorship from industry experts.

• Weekly live masterclasses with AI practitioners

• 100% online, designed for working engineers

Engineers from Microsoft, Apple, AWS, and other leading technology companies have already joined.

By graduation, you’ll have a GitHub portfolio that demonstrates your ability to design, build, and lead AI initiatives—not just talk about them.

Cohort starts August 1.

Reserve your seat today for just $99

(Thanks to Saras AI Institute for partnering on this post.)

I want to reintroduce Shefali Jangid as a guest author.

She’s a web developer, technical writer, and content creator with a love for frontend architecture and building things that scale.

Check out her work and socials:

• Shefali.dev

• GitHub

• Twitter

You’ll often find her writing about web development, sharing UI tips, and building tools that make developers’ lives easier.

Architecture

This section explains different architectural patterns and design approaches that help build apps that are modular, easier to maintain, and can grow without becoming difficult to update.

Let’s dive in!

40. Modular Architecture/Feature-Based Modules

As mobile apps grow, the codebase can become very large and difficult to manage.

If everything is written in one BIG project, it becomes harder to understand the code, make changes, and test new features.

To solve this problem, you can use modular architecture.

In modular architecture, app is divided into separate modules, where each module represents a specific feature of the app.

An app could have modules like:

• Auth module: Handles login and signup.

• Feed module: Shows posts or content.

• Checkout module: Handles payments and orders.

Each module contains its own code and exposes only the parts other modules need.

This keeps features separate and prevents different parts of the app from interfering with each other.

This structure makes the app easier to understand and maintain.

Why It Matters

In a very large app, building the entire project can take a long time…

If a single codebase contains hundreds of thousands of lines of code, compiling the app could take several minutes.

With modular architecture, only the module that changed needs to be rebuilt. This makes builds much faster and helps you work more efficiently.

Real-World Example

Airbnb, Uber, and Spotify use modular architecture in their mobile apps.

They adopted this approach to accelerate development, improve code organization, and allow different teams to work on separate features simultaneously.

Trade-offs

Modular architecture also introduces some challenges.

You need to carefully design the boundaries between modules and define clear interfaces for how modules communicate with each other. Some shared functionality, such as analytics, navigation, or design systems, may need to live in a common core module that all features can use.

It can also be difficult to convert an existing large app into a modular structure after it has already grown.

Practical Tip

It’s useful to visualize how modules depend on each other.

Tools like dependency graph visualizers can help teams see how modules are connected and identify problems such as circular dependencies, where two modules depend on each other. This usually means the architecture needs improvement.

While modular architecture improves how the app code is organized, mobile apps also depend heavily on backend systems designed specifically for them.

Share

41. Backend for Frontend (BFF) for Mobile

Mobile apps often need data from “many” different backend services.

A single screen requires data from a user service, a recommendation service, and a content service.

If the mobile app directly calls all these services, it would need to make several network requests and combine the results on the device. This can SLOW down the app and increase battery usage.

To solve this, use a technique called Backend for Frontend (BFF).

A BFF2 is a special backend layer designed specifically for a specific type of client, such as a mobile app.

Instead of the mobile app calling many services, it sends one request to the BFF. The BFF then communicates with the required services, gathers the data, combines it, and sends back a single optimized response.

This makes the mobile app simpler and reduces the number of network requests.

Why It Matters

Every network request adds delay & consumes battery.

If a mobile screen requires data from five services, the app will make five separate API calls. This increases latency and slows down the user experience.

With a BFF, the mobile app only makes one request, and the server handles the rest. This improves performance and keeps the app more efficient.

Real-World Example

Netflix uses the BFF approach to serve different types of clients.

They maintain separate backend layers for iOS apps, Android apps, TVs, and web browsers.

Each BFF returns data formatted specifically for that device.

A mobile BFF sends smaller responses optimized for mobile screens, while the TV version may include higher-resolution metadata.

Trade-offs

Using a BFF also introduces an additional service that needs to be maintained.

It is important BFF does not become overly complex. Its main responsibility should be to aggregate and transform data for the client. Core business logic should remain in the backend services that actually own the data.

As backend APIs evolve, another challenge appears: users often run older versions of the app...

Share

42. App Versioning & Backward Compatibility

When mobile apps are released, users do NOT always update them immediately.

Some people update right away, but others may continue using an older version of the app for weeks or even months.

So the backend system must support many versions of the app simultaneously.

i.e., server should continue to work with older app versions while also supporting the latest version.

To manage this, use versioning:

Versioning assigns each API, or app release, a version number so the server knows how to communicate with different app versions.

Check these API endpoints:

/v1/users
/v2/users

Each version can return data in a format that matches what that specific app version expects.

Why It Matters

If the backend changes something in the API without considering older app versions, older apps will suddenly stop working…

If the server removes a field that an older app depends on, the app could crash or show broken screens.

Unlike websites, mobile apps cannot be updated instantly for every user. This is why systems must remain backward compatible.

Real-World Example

Stripe uses API versioning to maintain compatibility.

When a client makes an API request, it specifies which API version it expects. Even if Stripe changes the API later, older clients still receive responses in the format they were built for.

This allows the platform to evolve without breaking existing apps.

Trade-offs

Supporting many versions of an API requires extra maintenance.

You need to keep older fields or response formats in the system for some time.

A common strategy used to manage this safely is called the Expand–Contract pattern:

1. First, add new fields or features without removing the old ones.

2. Gradually update apps to use the new fields.

3. Once all users have updated, remove the old fields.

This approach prevents breaking older versions while still allowing the system to evolve.

Even when APIs remain backward compatible, you still need safe ways to release new features…

Share

43. Feature Flags & Remote Configuration

When you add a new feature to a mobile app, you must release a new version of the app and wait for users to update it.

This process can take time because apps must go through the app store review, and not all users update their apps immediately.

Feature flags solve this problem:

A feature flag is a switch that lets you remotely enable or disable a feature without releasing a new version of the app.

The feature’s code is already included in the app, but it stays disabled by default. You can later enable it from a dashboard for specific users or for everyone.

This allows you to control when features are released.

Remote configuration extends this idea even further. It allows you to change certain app behaviors remotely, such as:

• UI text or labels

• Feature limits

• Layouts or design elements

• Experiment settings

Tools such as Firebase Remote Config are commonly used for this.

Why It Matters

Submitting an app update to the app store can take up to 3 days for approval.

If a new feature has a bug and has already been released to millions of users, fixing it through another app update can take time.

With feature flags, you can disable the feature instantly from a dashboard without waiting for another release.

This makes apps safer and easier to manage in production.

Real-World Example

Many companies use feature flags to run A/B testing3 experiments.

Airbnb tests two different versions of a feature:

• One group of users sees the original version.

• Another group sees a new version.

Feature flags control which users see which version, allowing the company to measure which design performs better.

Trade-offs

While feature flags are powerful, they can also create complexity.

If too many flags are added and never removed, the codebase can become cluttered with outdated flags. To avoid this, you should define a lifecycle policy where every feature flag has a purpose and an expiry date. Once the feature becomes permanent, the flag should be removed from the code.

Feature flags help control when features are released, but navigation systems determine how users actually reach those features inside the app.

44. Deep Linking for Navigation & Growth

Read more

• Share this post & I’ll send you some rewards for the referrals.

When people talk about Artificial Intelligence (AI), they often refer to models such as Generative Pre-trained Transformers (GPTs), Large Language Models (LLMs), and neural networks.

It sounds like magic, software that thinks…

But here’s the uncomfortable truth:

The biggest breakthroughs in AI are not coming from better models; they’re coming from better infrastructure.

If AI were only about the models, there would be no need for huge, warehouse-sized data centers packed with specialized hardware and powered by massive amounts of electricity.

So what’s actually going on inside those buildings?

To make sense of everything that follows, we need a simple mental model.

Let’s find out one layer at a time…

Onward.

Review the actual change, not the file list (Partner)

AI writes more code than ever. Reviewing it shouldn’t mean scrolling forty files in alphabetical order.

CodeRabbit Review reorganizes any pull request from a flat file list into a structured, layer-by-layer walkthrough - the logical reading order of the change, not the order your platform happens to sort it. Every range gets its own plain-language summary, with sequence diagrams, state machines, and ERDs generated inline wherever a visual earns its place.

Cohorts group related files and chunks so you review one idea at a time. Layers order them so foundational changes - data shapes, contracts - come before the code that depends on them. Code Peek lets you click any variable, function, class, or type to see its definition and usages without leaving the tab, while Semantic Diff view cuts past formatting noise to show what actually changed.

Open it straight from the Review Change Stack button in the PR Walkthrough. Navigate cohorts and layers from the keyboard, comment against exact line ranges, and submit native reviews, comments, and approvals post back to GitHub or GitLab right where your team expects them.

In the early access, available for free to everyone.

From the team that pioneered AI code reviews. 2M reviews every week. 6M repos. 15K customers.The review interface built for the way modern PRs are actually written.

>>> Review your next PR with CodeRabbit Review Today

(Thanks to CodeRabbit for partnering on this post.)

I’d like to introduce Ashish Prajapati.

He is an AI infrastructure specialist and technology educator who helps engineers understand the technologies powering modern AI.

With expertise spanning data centers, cloud computing, GPU infrastructure, Generative AI, and Agentic AI, he focuses on turning complex concepts into practical, real-world knowledge.

His training programs have reached over 39,000 learners worldwide, and through his free mentoring initiative, Become a Solutions Architect (BeSA), he has helped over 23,000 professionals accelerate their careers.

As AI adoption continues to grow, understanding the infrastructure behind AI systems is becoming a valuable skill for architects, engineers, and IT professionals.

For a limited time, this newsletter's readers can use code SYSTEM-DESIGN to get 25% off Ashish’s live NCA-AIIO bootcamp.

This intensive one-day workshop covers the fundamentals of NVIDIA AI infrastructure, GPU networking, accelerated computing, and AI operations, and prepares you for the NVIDIA-Certified Associate: AI Infrastructure and Operations (NCA-AIIO) certification.

Learn, Prepare, and Pass NCA-AIIO in a Day.

Here’s what you’ll find inside this newsletter:

• The hidden architecture behind AI. Why models are only the visible tip of a much larger infrastructure stack.

• The physics that limit AI growth. How power, cooling, and space shape every AI data center.

• Why GPUs became the backbone of AI. The architectural advantages that made modern AI training possible.

• The real bottlenecks in modern AI. Why memory, networking, and coordination determine performance at scale.

• How thousands of GPUs work together as one system. The software, networking, and orchestration layers that enable large-scale AI.

Think of an AI Data Center as a Factory

Before we go deep, build this mental model; it will explain almost everything that follows:

• AI Data Center = Factory

• Data = Raw material

• Graphics Processing Units (GPUs) = Machines on the factory floor

• Network = Internal transport system

• Storage = Warehouse holding raw materials

• Software = Assembly line logic

• Output = Trained intelligence