• Share this letter & I’ll send you some rewards for the referrals.

Every Claude session starts with context:

• conversation,

• your prompts,

• your CLAUDE.md,

• instructions added by the Anthropic team,

• and everything Claude knows about the task.

But Claude has a “limit”…

Every file you paste, every tool you connect, and every long conversation you keep alive uses part of that limit.

So if your context is draining quickly, the simplest & best option is to start a new conversation instead of waiting for Claude to compact everything.

In this newsletter, we’ll cover habits that help you use Claude’s context more efficiently & wisely.

To do that, we won’t just follow practical tips.

(We’ll also test different approaches & compare the results side by side.)

Let’s get started!

§

§

I want to introduce Gencay as the guest author.

He has been building with AI since ChatGPT first shipped in 2022.

For years, he built for clients on Upwork: 7,300+ hours of it.

Now he builds in the open and shares every piece of it on LearnAIWithMe, every Monday, Wednesday & Friday.

Not just theory for its own sake. One real build at a time, from idea to running system.

§

Here’s what you’ll find inside this newsletter:

• The budget most engineers blow in an hour. What quietly drains your context window, and how fast your daily and weekly limits go with it.

• The signal that a conversation is past saving. When to close it, and the one prompt to run first so nothing important gets lost.

• A project memory that lives outside the chat. Nested CLAUDE.md files, Auto Memory, and path-scoped rules, so the conversation itself stays disposable.

• A setup that gets smarter every week. The correction, rule, skill loop, and what happens when the setup meant to save tokens becomes the thing spending them.

• A real audit of a real project. An actual token bill, a 5-step cleanup, a token-saving skill tested head to head & five skills worth installing.

§

Context window is a budget

The context window is Claude’s working memory.

It holds your instructions, files, & entire conversation… and every model has a “fixed” size…

• Opus 5 carries a 1M context on the Max plan,

• Fable 5 draws down your usage faster than the older models.

So the window & your limits are connected.

Claude has daily & weekly limits, and some models spend them quicker than others.

This is why management matters…!

If you fill the window with noise, your limit drops early, and you sit there waiting for the reset instead of working.

§

Token saving habits that compound

If you repeat a job in every conversation, put it in a CLAUDE.md

Here is why:

Even a 1M window fills up, and when it does, Claude compacts the conversation on its own. Compacting is a summary, and summaries drop details. So the instructions you typed at the start may not survive into the rest of the conversation.

But a good CLAUDE.md never gets compacted away, because Claude reads it fresh from disk every session.

Plus, do not stretch one conversation forever. The bigger it grows, the more each compact costs, and those compacts eat into the same limit you wanted to save.

A new conversation is the cheapest fix…and here is the signal for when to switch:

When Claude starts compacting every few messages, or the answers begin forgetting rules you gave an hour ago, the conversation is past saving. Each compact now eats your limit while dropping details, so closing it costs less than keeping it alive.

But you do not want to lose what the old one knows, so ask this before you close:

Summarize this conversation in 5 sentences.
But first, ask me 5 questions to make sure the things I care about end up in the summary.

Then paste the summary into the new conversation & you continue where you left off at a fraction of the cost.

These three commands help you manage the window without leaving it:

• /clear resets the context to zero.

• /compact condenses the conversation, and you steer it with an instruction like /compact focus on the code changes.

• /context shows what is sitting in your window right now, so you see the problem before it costs you.

§

Grow your project in MD files, not in the chat

A big project inside one chat gets heavy.

Some users hold onto the conversation because valuable information lives there, and they are afraid to lose it.

The better way is a folder & an MD file.

Add a folder to the conversation, and then use this:

Create a CLAUDE.md for this project.

Make sure to include the decisions we made in this conversation
and the rules I gave you.

Now the knowledge lives in the file instead of the chat.

Every time you open this project, the saved context comes with it, and the conversation itself stays disposable.

You could even switch models/tools, and the work survives because the files are the project.

§

Memory at scale

Let’s say your project is big, with tens of folders.

A nested CLAUDE.md system is a good road here.

Your main CLAUDE.md stays short & it says something like this:

This project has 10 folders.
Each folder holds its own CLAUDE.md explaining it.
Read the nested file only when you work inside that folder.

And each folder carries its own CLAUDE.md with the details.

The gain is double…

In a normal conversation, the nested files stay unread, so your context stays light & the session runs longer.

And when you need the details, one prompt like

Check this folder and read its CLAUDE.md

…loads only the part you asked for.

You save time & tokens with the same move.

Two more layers work alongside your files:

Claude keeps its own notes with Auto Memory, saving what it learned from your corrections per project, and /memory opens them for editing. So the memory system runs both ways: you write instructions, and Claude records what it learns.

And for larger projects, “rules” split the load further…

A rule is a small MD file under .claude/rules/, one topic per file, and it could carry a “paths” field limiting it to matching files. So your API conventions load only when Claude touches the API folder, and they cost nothing on the frontend days.

§

Compound engineering

Every correction you make is worth money, but only if you write it down.

Claude makes a mistake, you fix it once, and then you turn the fix into a rule in your CLAUDE.md or a skill.

The next session starts smarter, and the mistake never comes back.

I run my entire setup this way. When a correction repeats twice, it becomes a rule.

When a workflow repeats twice, it becomes a skill.

So the system compounds, and the setup I have today is the sum of every mistake Claude made this year.

§

§

Skills that reduce tokens

Read more

• Share this letter & I’ll send you some rewards for the referrals.

Once upon a time, there lived a junior software engineer named Michiko.

She worked for a tech company named Hooli.

Although she was bright, she never got promoted.

So she was sad & frustrated…

Until one day, she had the idea to apply for a new job.

And the interviewer asked her 1 simple question:

“How to effectively scale a system?”

But she couldn’t answer it & failed the interview in 5 minutes.

So she went home & studied it to fill the knowledge gap.

Onward.

Mermaid diagrams are the floor

Your coding agent can already paste a Mermaid chart into a PR description.

You still read every file to learn what it does to your system.

PR Lens draws each pull request as a live architecture canvas, right in your pull request. Press play and it walks you through the change, one step at a time.

On every PR, you see:

• What changed, marked on the parts of the system it touches

• What else it reaches, including services the diff never names

• How a request moves through the system now

• What could break, flagged before you merge

2,200+ GitHub stars and 45,000+ pull requests drawn.

Add it to your repo

(Thanks to Coldtea for partnering on this newsletter.)

Scalability Patterns in System Design

Michiko failed the interview, so you don’t have to.

And here are 11 must-know scalability techniques for you to ace job interviews:

1. Vertical scaling

Vertical scaling gives one machine more power by adding CPU, memory, and/or faster disks.

The app stays the same, but the machine it runs on gets more powerful... Or you move the app to a larger instance.

i.e., you do NOT need to rewrite any code. And vertical scaling is probably the quickest way to handle extra load.

It works well for monoliths & older systems built to run on a single machine.

Yet larger machines cost more & all machines have limits.

Plus, if your app relies on a single server, a failure could take the entire app down.

So scale up when you need more capacity quickly. Then consider horizontal scaling before one machine becomes your system’s biggest bottleneck.

2. Horizontal scaling

Horizontal scaling (scaling out) adds more machines to share the load.

You run several copies of the app & distribute traffic across them using a load balancer.

i.e., you add machines as traffic grows & remove them when demand drops.

It works well for applications with high/unpredictable traffic.

Yet adding machines creates new challenges:

You need load balancing, coordination between instances & keep shared data consistent.

Horizontal scaling improves “availability” with redundant instances. But if your app stores user sessions locally, requests routed to different instances could lose access to session data.

So before you scale out, you need to decide where your application stores its state.

And that’s why you’d need stateless services...

3. Stateless services

A stateless service does NOT store client session state between requests.

Every request provides information the server needs - auth token or session ID.

Plus, you don’t store session data on one server… instead, you keep it in a shared database, a cache like Redis, or use a signed token like a JSON Web Token (JWT).

i.e., any server could handle a request without relying on the previous one.

This makes it easier to add, remove, or replace servers as traffic patterns change.

Yet the state still needs to live somewhere.

And an external database/cache adds infrastructure, network latency & an extra dependency to manage. Plus, relying on sticky sessions to keep users tied to one server reduces flexibility when servers fail or traffic changes.

So keep session state outside individual app instances when possible. Also, design operations to be “idempotent”.

With stateless services, you could scale individual instances more easily.

But what if those services depend heavily on each other…

That’s where you’d need loose coupling...

4. Loose coupling

Loose coupling = each component depends on others as little as possible.

A change in one component could force you to update several others in a “tightly coupled” system.

Plus, if one service becomes slow/fails, the problem could spread to other services that depend on it.

Loose coupling reduces these dependencies by letting components communicate through well-defined interfaces - APIs/messages. So each component knows what to expect from others without understanding their internal implementation.

This makes it easier to update, deploy, and/or scale components independently.

Yet loose coupling introduces extra complexity:

Requests could travel across many services,,, thus making it HARD to trace & debug failures. Also,splitting a system into many tiny services increases communication overhead & coordination costs.

So use loose coupling when components need to evolve independently - microservices & big apps.

And to organize these components into clear, manageable parts, you need modular architecture…

5. Modular architecture

Modular architecture divides a system into separate modules, with each responsible for a specific function.

You divide the app into modules for authentication, orders, payments, notifications, and so on.

i.e., each module has a clear responsibility & communicates with others through defined interfaces.

The goal is to group functionality that belongs together & keep dependencies between modules minimal. This follows the principle of separation of concerns: each module focuses on one part of the system.

So you update/test payment module without changing the authentication module.

Yet choosing the “wrong module boundaries” creates unnecessary dependencies.

Plus, splitting an app into many modules makes the codebase hard to manage. Also, modules within the same app still share resources & failure risks.

So modularity alone does NOT guarantee independent scaling or failure isolation.

It works well for large applications, modular monoliths & microservice architectures!

But if these modules need to communicate without waiting for each other…you’d need event-driven architecture…

6. Event-driven architecture

Event-driven architecture allows services to communicate by publishing & reacting to events.

A service publishes an event to a message broker instead of calling other services directly.

Imagine a user places an order on an e-commerce website:

The order service publishes an OrderPlaced event… Then other services consume it independently:

• Inventory service reserves ordered items,

• Payment service initiates payment processing,

• Notification service sends an order confirmation when required steps succeed.

Order service does NOT wait for each consumer to finish processing the event.

i.e., a slow consumer does NOT block others.

This approach works well for workflows when many services need to react to the same event.

Yet event-driven systems create new problems:

Events could arrive late, out of order, or more than once. Plus, different services could temporarily hold different versions of the same data. This is called eventual consistency.

So you need idempotent consumers, reliable event delivery & a way to handle failed operations.

Event-driven architecture helps services scale independently & reduces synchronous dependencies.

But when you need to organize these services without creating a central bottleneck, you’ve got to “avoid” centralized resources…

7. Shared-nothing architecture

Shared-nothing architecture gives each node its own compute, memory & storage without relying on shared resources for processing.

There are several ways to achieve this:

Read more

• Share this letter & I’ll send you some rewards for the referrals.

The CI/CD pipeline changed how we deploy software.

It made deployments faster, repeatable, and automated. But once a deployment finishes, the pipeline has no idea what happens inside the cluster afterward.

Someone could edit a Deployment with kubectl, apply a hotfix during an incident, or update a resource manually. Those changes bypass Git and the pipeline.

As manual changes increase, the cluster slowly drifts from what gets stored in Git.

At some point, production no longer matches the configuration everyone believes is running. When an issue appears, it becomes difficult to see what changed... Also it becomes difficult to know who changed it or whether the running state still matches the last reviewed configuration.

GitOps solves this by making Git the source of truth for the cluster…not just another step in the deployment process…

An agent continuously compares the live cluster with the desired state stored in Git and reconciles any differences. If someone changes a resource outside Git, ArgoCD detects the drift and brings the cluster back to the expected state.

The CI pipeline still builds, tests, and publishes the application1.

But instead of deploying directly to Kubernetes, it updates Git. ArgoCD then handles the deployment and keeps the cluster aligned with the repository.

§

[Webinar] How to stop rewriting your agent’s code

Right now, you’re your agent’s context engine. You write the rules files, maintain the skills, and paste in the Slack thread it missed.

But hand-curated context goes stale. And when there are conflicting answers, your agent can’t tell which is right. You pay for it in higher token bills and longer reviews.

Join us live on Oct 28 (FREE) to see:

• The 8-levels of AI maturity and where teams get stuck

• How a context engine curates context and resolves conflicts before your agent writes a line of code

• Live demo: token savings in the same coding task, and use cases beyond code gen

Bonus: 2 open source tools you can use today.

Register now

(Thanks to Unblocked for partnering on this newsletter.)

§

I want to introduce Ayaan as a guest author.

He’s a Certified Kubernetes Administrator (CKA) who writes about Kubernetes, cloud infrastructure, and cloud-native systems. He builds in public and shares practical tips, projects, and lessons learned along the way.

Check out his work & socials:

• Twitter

• Github

§

Here’s what’s inside:

• Your deployment succeeds. Then production starts drifting. How manual Kubernetes changes create configuration drift, why traditional CI/CD pipelines miss them, and how GitOps keeps your cluster aligned with Git.

• Your CI pipeline no longer needs the keys to Kubernetes. How ArgoCD’s pull-based model changes deployments, where credentials live, and what happens when a system gets compromised.

• ArgoCD runs a reconciliation loop behind the scenes. How its core components detect changes, compare desired and live states, and use sync policies, self-healing, and hooks to control deployments.

• One template, hundreds of applications. How ApplicationSets manage deployments across clusters, why repository structure matters, and how Git commits promote releases from development to production.

• GitOps gets harder when you reach production. How to manage secrets, access control, monitoring, progressive delivery, and disaster recovery, plus the tradeoffs you need to understand before adopting it.

§

What Is GitOps

GitOps2 is an operating model where a Git repository is the single source of truth for everything running in your infrastructure.

Every change to your application or cluster configuration happens through a Git commit. It gets reviewed, approved, and merged through the same process as application code.

i.e., changes are not applied directly to the cluster outside Git.

§

GitOps Principles

The OpenGitOps3 working group at CNCF4 defined four principles to distinguish GitOps from a traditional deployment pipeline.

Let’s dive in!

1. Declarative

In a declarative system, you describe what you want, not how to get there.

Instead of writing a script with steps like “create this deployment, then scale it to three replicas, then expose it on port 80,” you write a YAML file describing the end state. The system figures out how to make it happen.

Kubernetes is declarative by nature.

Helm charts5, Kustomize6 overlays, and raw YAML manifests all follow the same model.

This matters for GitOps because a declarative system supports storage, versioning, and comparison. Unlike a sequence of commands, two YAML files allow you to see exactly what changed.

2. Versioned and Immutable

Every change to the desired state is a Git commit.

And Git commits do not change…

This gives you a complete, tamper-evident history of everything your cluster has ever been asked to run. Who requested the change, who approved it, when it got merged, and exactly what it modified. The audit trail becomes part of the deployment workflow, not a separate system.

Rollback is also a natural consequence of this principle. Reverting to a previous state means reverting a commit.

3. Pulled Automatically

This principle changes who is responsible for applying changes.

Instead of an external CI/CD pipeline pushing updates into Kubernetes, an agent inside the cluster pulls the desired state from Git… It then applies the changes.

The security implications of this shift are significant, and they get their own section shortly.

4. Continuously Reconciled

The previous three principles describe how state gets defined and delivered.

This one describes what happens after:

Continuous reconciliation means repeatedly comparing the live cluster state with the desired state in Git. These comparisons detect changes introduced by a manual edit, an autoscaler, or a partial update. The configured sync policy determines whether ArgoCD reports the difference or corrects it automatically.

To understand why GitOps works differently, we first need to compare it with the deployment model most people already use…

Share

§

CD Without GitOps vs CD With GitOps

With the four principles in place, the difference between traditional CD & GitOps is easier to see.

It is not just a philosophical shift. The two models produce different security properties and failure modes.

They also provide different levels of visibility into what runs in production…

How CD Worked Before GitOps

In a traditional push-based setup7, the CI/CD pipeline carries a release from source code to the Kubernetes cluster.

It runs tests, builds a container image, and pushes the image to a registry. The deployment stage then updates Kubernetes using a command such as kubectl apply or helm upgrade.

For example, releasing version 1.5 means the pipeline updates the Deployment to use the new image. Kubernetes then replaces the existing pods according to the configured rollout strategy.

The pipeline directly initiates the deployment. The deployment stage delivers the new configuration to the cluster.

How CD Works With ArgoCD

With ArgoCD8, the pipeline still tests the application, builds the image, and publishes it to a registry. It then updates the image reference in the configuration repository.

For the same version 1.5 release, the pipeline changes the image tag in a Kubernetes manifest, Helm values file, or Kustomize configuration. This change becomes a Git commit rather than a deployment command executed against the cluster.

ArgoCD detects the updated configuration and compares it with the live resources. With automated sync enabled, it applies the update. With manual sync, it waits for an operator to trigger synchronization. Kubernetes then performs the rollout.

i.e., Git becomes the handoff between the two systems.

CI produces the image and records the intended version. ArgoCD handles applying the configuration to Kubernetes.

This separation also changes where deployment credentials are needed, which brings us to the security boundary between push and pull.

§

Push vs Pull: Security Boundary

The shift from push to pull is not just an architectural preference.

It changes where credentials live, how far a compromised system can reach, and the damage a supply chain attack can cause.

Now let’s compare the two approaches:

Push-Based: Pipeline Holds the Keys

In a push-based model, the CI pipeline needs direct access to the cluster to deploy anything.

i.e., cluster credentials live inside the CI system.

Anyone with sufficient access to the CI system or pipeline configuration could also gain access to those credentials. Every third-party action, plugin, or integration running inside the pipeline runs adjacent to them. And because pipelines are triggered by code changes, a compromised dependency runs in the pipeline. It runs in the same environment as your cluster credentials.

A compromised pipeline in this model is effectively a compromised cluster.

The attacker does not need to find a vulnerability in Kubernetes… They just need to get malicious code into the build process & let the pipeline do the rest.

The blast radius is as wide as the credentials allow.

For most engineers, those credentials have broad permissions. Restricting them precisely is operationally painful. A compromised push-based pipeline often means cluster-admin access is in the wrong hands.

Pull-Based: Cluster Holds the Keys

In a pull-based model, the cluster reaches out to Git.

The CI system never touches the cluster.

ArgoCD runs inside the cluster with scoped RBAC9 (Role-Based Access Control) permissions. It pulls the desired state from Git and applies it. The CI pipeline updates the Git repository when a build succeeds, and its job ends there. No cluster credentials ever leave the cluster environment.

A compromised CI system in this model could modify Git.

But modifying Git only changes the desired state. It does not directly execute anything in the cluster. ArgoCD still has to pull the change and reconcile it with the cluster. Because every change goes through Git first, you get a reviewable, auditable record before it reaches production.

And the blast radius10 shrinks dramatically. A compromised pipeline becomes a much less dangerous event.

§

§

ArgoCD Architecture

Let’s look at how ArgoCD is built:

ArgoCD is not an external service you call into.

It runs as Kubernetes workloads inside the cluster it manages, in the argocd namespace. This is intentional. The agent needs direct access to the Kubernetes API to reconcile state. Running inside the cluster provides the access & keeps cluster credentials inside the cluster.

Here are 5 core components that work together to implement the GitOps reconciliation loop:

1. API Server

API Server is the entry point for everything human-facing in ArgoCD.

It exposes both gRPC11 & REST APIs consumed by the ArgoCD UI, argocd CLI, and any external tools integrating with ArgoCD. Every user request, whether it comes from the UI, CLI, or an external integration, passes through the API Server.

Also it handles all authentication & authorization.

Before any request reaches the rest of the system, the API Server verifies who is making it and whether they have permission.

2. Repo Server

The Repo Server does one thing: render Kubernetes manifests from a Git repository.

It clones the repository, reads the source, and generates the manifests ArgoCD will apply to the cluster. It supports plain YAML, Helm charts, Kustomize overlays, and Jsonnet12. Whatever format your configuration lives in, the Repo Server handles the rendering.

It’s deliberately isolated from the Application Controller.

Rendering manifests requires running Helm, Kustomize & Jsonnet. Those processes are often CPU-intensive and occasionally unstable. Keeping the Repo Server separate isolates render failures from the controller’s reconciliation loop.

3. Application Controller

The Application Controller is the heart of ArgoCD…It runs the reconciliation loop.

On every cycle, it gets the desired manifests from the Repo Server and the live cluster state from the Kubernetes API. Then it computes the diff between the two. Based on the application’s sync policy, it either alerts you to the difference or corrects it automatically.

The Application Controller runs as a StatefulSet.

(Scaling and sharding are covered later.)

4. Redis

Redis serves as a cache layer between Application Controller and Kubernetes API.

Without it, every UI load and every reconciliation cycle would need to query the Kubernetes API directly. At scale, this would generate unnecessary load on the Kubernetes API. Redis stores application state and cluster data so repeated reads hit the cache instead of the API.

It is worth being clear about what Redis is and is not in this context.

It is a cache, not a primary data store. ArgoCD does not depend on Redis for correctness. If Redis goes down, ArgoCD degrades gracefully. Performance suffers, but reconciliation continues.

5. Notifications Controller

Notifications Controller watches ArgoCD events & route alerts to external systems.

When an application goes OutOfSync, a sync fails, or a health check degrades, the Notifications Controller sends an alert. It uses the destination you configured, such as Slack, email, PagerDuty, or a webhook. You define triggers and templates, and the controller handles the delivery.

In most production deployments, it gets installed alongside the rest of ArgoCD. It connects ArgoCD’s internal events with the people and systems responsible for responding to them.

Together, these components form ArgoCD's control plane...

Each component has a specific responsibility:

• API Server handles user requests.

• Repo Server renders manifests.

• Application Controller reconciles applications.

• Redis caches data to reduce repeated API calls.

• Notifications Controller reports important events.

Together, these components keep the cluster aligned with the desired state in Git.

With the architecture covered, let’s see how these components work together during reconciliation…

§

§

How ArgoCD Keeps Your Cluster in Sync

Read more

• Share this letter & I’ll send you some rewards for the referrals.

Computer networking is an important part of ALL systems you’ll work on in your engineering career.

While junior developers may get by with writing code, senior engineers are valued for their ability to understand how systems communicate & troubleshoot network problems.

What happens when a network connection fails?

How does data travel between devices & servers?

How do we diagnose slow connections, packet loss & routing problems?

These questions form the foundation of every reliable, high-performing system.

Onward.

[Webinar] How to stop rewriting your agent’s code

Right now, you’re your agent’s context engine. You write the rules files, maintain the skills, and paste in the Slack thread it missed.

But hand-curated context goes stale. And when there are conflicting answers, your agent can’t tell which is right. You pay for it in higher token bills and longer reviews.

Join us live on Oct 28 (FREE) to see:

• The 8-levels of AI maturity and where teams get stuck

• How a context engine curates context and resolves conflicts before your agent writes a line of code

• Live demo: token savings in the same coding task, and use cases beyond code gen

Bonus: 2 open source tools you can use today.

Register now

(Thanks to Unblocked for partnering on this newsletter.)

Computer Networking Fundamentals

Here are all the key concepts that you need to understand computer networking & become a better engineer:

1. Ethernet

Ethernet is a networking technology that connects devices within a Local Area Network (LAN1).

How it works2:

1. Framing: The sender wraps an IP packet inside an Ethernet frame3.

2. Addressing: Frame contains source & destination Media Access Control (MAC) addresses.

3. Transmission: The network interface sends the frame over a physical link.

4. Switching: Network switches forward frames toward their destinations using MAC addresses.

5. Error detection: The receiver checks the frame’s Frame Check Sequence (FCS) to detect transmission errors.

Typical Ethernet speeds include 1 Gbps for desktops, 10 or 25 Gbps for servers, and 100+ Gbps for data center networks.

2. MAC address

A media access control (MAC) address4 identifies a network card. They help devices communicate within a “local” network.

Every card ships with a 48-bit address, written as six pairs of hexadecimal digits. The first half identifies the manufacturer, while the second half identifies the card.

IP addresses move packets between networks, and MAC addresses move frames within one network. A device learns its neighbor’s MAC address by broadcasting a question through the address resolution protocol5 (ARP), and then caches that answer.

3. Network switch

A switch6 forwards frames between devices on the “same” network, using their MAC addresses.

A switch learns devices that are reachable through each port by reading the source MAC address of incoming frames. Then it stores these MAC address-to-port mappings in a forwarding table.7

How it works:

• Known destination: The switch forwards the frame through the port associated with the destination MAC address.

• Unknown destination: The switch floods the frame through all other ports in the same VLAN until it learns where the destination is located.

A switch forwards Ethernet frames within a local network using MAC addresses, while a router forwards IP packets between networks using IP addresses.

4. Wi-Fi

Wi-Fi8 is a wireless networking technology that uses radio waves to transmit data between devices.

How it works:

• Wireless transmission: Devices send & receive data using radio signals.

• Channel sharing: Devices use mechanisms such as Carrier Sense Multiple Access with Collision Avoidance9 (CSMA/CA) to reduce transmission collisions.

• Frequency bands: Wi-Fi operates on 2.4 GHz, 5 GHz, and 6 GHz bands.

• Signal strength: Devices adjust transmission rates based on signal quality & interference.

• Retransmissions: Lost/corrupted frames require retransmission; thus reducing throughput.

The 2.4 GHz band typically provides longer range but has less available capacity & more interference.

The 5 GHz & 6 GHz bands offer wider channels and potentially higher speeds, but their signals have shorter range & weaker penetration through obstacles.

Also devices with poor signal quality transmit at lower data rates and consume more airtime. This reduces the capacity available to other devices sharing the channel.

5. Static IP address

A static IP address is a “fixed” IP address assigned to a device or network interface.

You configure it manually or reserve an address through Dynamic Host Configuration Protocol (DHCP).

A static IP10 typically remains the same across device reboots as long as its assignment remains valid. Servers, printers, routers & other devices that need predictable network addresses use static IPs.

DNS records, firewall rules & network configurations reference these stable addresses.

6. DHCP

Dynamic Host Configuration Protocol (DHCP) automatically assigns IP addresses & other network settings to devices when they join a network.

How it works:

1. Discover: The device broadcasts a message to find available DHCP servers.

2. Offer: A DHCP server offers an IP address & network configuration.

3. Request: The device requests the offered address.

4. Acknowledge: Server confirms the assignment & grants an IP address lease.

(This process is called DORA.)

Plus, DHCP11 server provides a subnet mask, default gateway, and Domain Name System (DNS) servers.

Yet each IP address lease has a lifetime.

Devices typically renew leases before they expire. If a lease expires without renewal, the server reassigns the address to another device.

7. Default gateway

A default gateway is a router that forwards traffic to destinations outside a device’s local network when “no” specific route exists.

How it works:

• Local destination: The device sends traffic directly to another device on the same subnet.

• Remote destination: The device forwards traffic to its default gateway.

• Packet forwarding: Gateway uses its routing table to forward the packet toward its destination.

A laptop with IP address 192.168.1.10/24 might use 192.168.1.1 as its default gateway.

The gateway12 typically has an IP address on the same subnet,,, thus allowing the laptop to reach it directly.

An incorrect default gateway prevents a device from reaching “external” networks, even if local communication works.

8. Static route

A static route is a (manually) configured entry in a routing table that tells a router where to forward packets.

You specify a destination IP range & an outgoing network interface.

How it works:

• Configure a route: Define a destination network, such as 10.0.2.0/24.

• Specify the next hop: Point the route to another router, such as 192.168.1.1.

• Forward packets: The router uses the route to forward matching traffic toward its destination.

Yet static routes13 do not automatically learn new network paths.

If a network path fails, the static route still remains configured. Some implementations detect failures & deactivate/replace the route.

Static routes work well for cloud networking configurations.

9. OSPF

Open Shortest Path First (OSPF) is a dynamic routing protocol that lets routers automatically discover & update routes within an Autonomous System (AS).

How it works:

1. Discover neighbors: Routers exchange Hello packets to discover & maintain neighboring routers.

2. Share topology: Routers flood Link-State Advertisements14 (LSAs) to share network information within an OSPF area.

3. Build a network map: Routers in the same area maintain a synchronized link-state database.

4. Calculate routes: Each router uses Dijkstra’s Shortest Path First15 algorithm to calculate the lowest-cost paths.

5. Adapt to failures: When a link fails, routers propagate updated information and recalculate affected routes.

OSPF16 selects paths based on configurable interface costs.

Bandwidth influences these costs, but administrators could adjust them manually.

Plus, OSPF reroutes traffic after network failures. Convergence time depends on failure detection, topology size & router configuration.

10. ICMP

Internet Control Message Protocol (ICMP17) carries network error messages & diagnostic information. It helps devices identify problems such as unreachable destinations, expired packet lifetimes, and packets too large for a network link.

How it works:

• Destination unreachable: A router or destination host sends an ICMP message when it cannot deliver a packet.

• Time exceeded: A router sends an ICMP message when a packet’s Time to Live (TTL) or Hop Limit reaches zero.

• Packet too big: ICMP reports when a packet exceeds the supported Maximum Transmission Unit (MTU).

• Echo request and reply: Tools such as ping use ICMP to test whether a destination responds.

Yet ICMP messages are NOT guaranteed to arrive…

Firewalls, rate limiting & network conditions could block/drop them.

ICMP helps diagnose network problems but does not directly retransmit lost application data or repair failed connections.

11. Ping

Ping18 is a network diagnostic tool that tests whether a destination host responds & measures round-trip time (RTT). It uses ICMP Echo Request & Echo Reply messages.

How it works:

1. Send request: Ping sends an ICMP Echo Request to the destination.

2. Receive reply: If the destination responds, it sends an ICMP Echo Reply.

3. Measure latency: Ping calculates time between sending request & receiving reply.

4. Track responses: Ping reports round-trip times & percentage of requests without replies.

Repeated tests help identify latency spikes, intermittent connectivity & potential packet loss.

Final Words

Now you understand the basic networking concepts that help devices communicate, it’s time to explore how they work in real systems.

Look at the apps & networks you use every day and think about what happens behind the scenes.

What happens when a network link fails?

How does your laptop connect to the internet?

How do routers find the right path for packets?

How do engineers diagnose slow or failed connections?

After understanding these concepts, start applying them in your own work.

Use tools like ping, mtr, and tcpdump to explore network behavior. Experiment with routing configurations and troubleshoot connectivity problems.

Remember, every experienced engineer started with the fundamentals. The more you practice, the easier it becomes to understand and troubleshoot complex networks.

If you find this newsletter valuable, share it with a friend, and subscribe if you haven’t already. There are group discounts, gift options, and referral rewards available.

Subscribe now

Want to reach 250K+ tech professionals at scale? 📰

If your company wants to reach 250K+ tech professionals, advertise with me.

Thank you for supporting this newsletter.

You are now 250,001+ readers strong, very close to 251k. Let’s try to get 251k readers by 27 October. Consider sharing this letter with your friends and get rewards.

Y’all are the best.

Share

References

• Google, IPv6 Statistics: https://www.google.com/intl/en/ipv6/statistics.html

• Meta Engineering, More details about the October 4 outage: https://engineering.fb.com/2021/10/05/networking-traffic/outage-details/

• RFC 9293, Transmission Control Protocol: https://www.rfc-editor.org/rfc/rfc9293.html

• RFC 9000, QUIC: A UDP-Based Multiplexed and Secure Transport: https://www.rfc-editor.org/rfc/rfc9000.html

• RFC 9110, HTTP Semantics: https://www.rfc-editor.org/rfc/rfc9110.html

• RFC 9111, HTTP Caching: https://www.rfc-editor.org/rfc/rfc9111.html

• RFC 8446, The Transport Layer Security Protocol Version 1.3: https://www.rfc-editor.org/rfc/rfc8446.html

• RFC 1035, Domain Names: Implementation and Specification: https://www.rfc-editor.org/rfc/rfc1035.html

• RFC 4632, Classless Inter-domain Routing: https://www.rfc-editor.org/rfc/rfc4632.html

• RFC 4271, A Border Gateway Protocol 4: https://www.rfc-editor.org/rfc/rfc4271.html

• HTTP Archive, Web Almanac 2025: https://almanac.httparchive.org/en/2025/

Build for the long run

Long-running agents need endurance.

They must endure waiting for users and weather tool failures. They must take each new jump in model intelligence in stride.

Inngest makes long-running agents durable, observable and improvable over time.

Build for the long run with Inngest »

(Thanks to Inngest for partnering on this newsletter.)

§

The 80/20 rule states that 80% of results come from 20% of actions.

It is the ONE rule you must obey & this is the most important rule when it comes to learning something new.

Let me explain...

Imagine you want to fill a bottle with pebbles, sand & water.

If you pour in the sand & water first, you’ll struggle to fit the big pebbles.

But if you start with the pebbles, the sand automagically fills the gaps between them, and the water fills the remaining space.

i.e., do the 20% that makes the real difference (80%) first.

And this is why I’m writing these letters1 on AI engineering concepts for you.

I want you to win in the AI era by understanding the “fundamental” AI concepts.

Inside this newsletter, you’ll find:

35. Mixture-of-Experts,

36. Large Language Model Serving & Inference Engines,

37. Synthetic Data & Self-Instruction,

38. LLM-as-a-Judge,

39. Direct Preference Optimization,

40. Knowledge Distillation,

41. Temperature & Sampling,

42. Gradient Descent & Optimization,

43. Backpropagation,

44. Loss Functions,

45. Generalization,

46. Scaling Laws,

47. AI Alignment,

48. AI Safety & Guardrails.

For each, we’ll cover:

• What it means & how it works,

• A real-world analogy where useful,

• A common mistake to avoid,

• Why it matters.

Let’s go!

§

• Share this letter & I’ll send you some rewards for the referrals.

§

35. Mixture-of-Experts

Mixture-of-Experts (MoE2) is a model design that uses a router to direct information to neural network components3 called experts. Common sparse designs activate only some experts for each token.

Let’s look inside a model as it processes a question:

Imagine one layer contains eight experts. A router selects two experts to process a token’s numerical representation. Then the layer combines their results & passes the information onward.

The next token could use a different pair of experts.

The router learns how to assign work during training. Each expert performs part of the model’s calculations, while other components also contribute to the response.

(These experts work within the same model.)

Analogy

Think of a workshop with several workers & a coordinator.

The coordinator assigns each task to a few workers. Those workers contribute their results, and the work continues through the workshop.

Not every worker needs to handle every piece.

Common mistake

Assuming each expert has a fixed subject - mathematics, coding, or history.

The word “expert” describes a neural network component. Training shapes its behavior & router’s choices, but those choices don’t necessarily follow recognizable subject categories.

Why it matters

Sparse MoE allows a model to have more parameters without using all its experts for every token. Only selected experts perform calculations, thus reducing computation while giving the model more capacity to learn patterns.

Yet all expert weights still need to be stored in memory/storage.

So using fewer experts per token reduces computation, but doesn’t reduce the model’s memory requirements.

36. LLM Serving and Inference Engines

LLM serving makes a trained model available to receive requests & return responses. An inference engine is the software that runs the model’s calculations.

You submit “Explain this error” through a chatbot:

1. The serving system accepts your request & passes it to the software that runs the model.

2. Inference engine uses the model’s weights & your input to generate tokens.

3. The serving system sends those tokens back to your screen.

Your answer arrives through several coordinated steps.

Now imagine hundreds of users submitting questions at once...

The system schedules work within the available memory & computing capacity. Some requests need longer responses than others. Requests that exceed available capacity must wait or get rejected.

The model needs software around it to serve real users.

Analogy

Think of a restaurant’s kitchen & service team4:

The kitchen prepares the meals… The wider restaurant operation accepts orders, coordinates workload, and delivers food to customers.

Preparing a meal is one part of serving a restaurant full of people.

Common mistake

Testing one request & assuming the system is ready for traffic.

A setup that responds quickly to one user could slow down when many requests arrive at once5. Test realistic request volumes & response lengths, then check waiting times, failures & memory use.

Why it matters

LLM serving helps applications handle requests from many users.

While inference engines run the model’s calculations to generate responses. Together, they affect response time, hardware efficiency, and how reliably an application serves users.

Choosing a suitable engine & testing the complete serving system helps turn a trained model into a dependable service.

37. Synthetic Data and Self-Instruction

Synthetic data is “artificially created” data used for purposes such as training/testing. Self-instruction6 uses a language model to generate tasks & corresponding responses for further training.

Imagine you need more training examples for a model that classifies “customer support” messages.

You ask a language model to generate fictional messages about billing & technical problems.

With self-instruction, the model also creates instructions & expected answers for those messages.

Each example contains a task & its correct label:

• Instruction: Classify this customer message.

• Message: “My card got charged twice.”

• Expected answer: Billing.

The system then checks generated examples & removes incorrect, repetitive, and/or unsuitable ones.

The remaining examples could be used to fine-tune the model…

Also generating examples alone doesn’t change the model’s weights… A separate training step uses those examples to update the weights, thus helping the model learn to classify similar messages.

Analogy

Think of preparing practice questions with an answer key:

You create exercises to cover different situations. Before anyone studies from them, you check whether the questions make sense & if the answers are correct.

Faulty practice material teaches faulty lessons.

Common mistake

Treating a large generated dataset as a high-quality dataset.

Generated examples could contain wrong answers, repeated patterns, or unrealistic situations. Check their accuracy & variety, then evaluate the trained model on separate examples that reflect real use.

Why it matters

• Synthetic data helps expand training material when suitable examples are scarce.

• Self-instruction provides one way to create examples for instruction-based tasks.

The value comes from useful, checked examples rather than volume alone.

§

§

38. LLM-as-a-Judge

LLM-as-a-Judge uses a language model to evaluate another model’s responses against predefined criteria, such as factual accuracy, relevance, or completeness, and assigns scores/rankings based on those criteria.

Ask an assistant to shorten a customer’s complaint.

The customer wrote, “My order arrived three days late, and the box was damaged.” But the assistant produces “The order arrived late.” You give a judge model both versions & ask it to check whether the summary includes both problems.

The judge could flag the missing detail: damaged box.

Your instructions define what the judge checks. It could score one response, explain a problem, or compare two answers & select the better one.

Clear criteria give the review a specific purpose.

Analogy

Think of a teacher checking a student’s work:

The teacher compares the answer to the assignment requirements. A judge model performs a similar review using the instructions and reference material you supply.

Common mistake

Treating the judge’s score as proof of correctness.

A judge could overlook an error or favor a longer answer without good reason. Compare some of its reviews with careful human assessments. When comparing 2 answers, swap their order to check whether its preference changes.

Why it matters

Reading every response by hand takes time.

LLM judges help teams review large collections of answers & identify responses that need closer inspection.

Their usefulness depends on how reliably their assessments match qualities you need.

39. Direct Preference Optimization

Direct Preference Optimization (DPO) “trains” a language model to favor preferred responses over less preferred alternatives using pairs of answers to the same request.

Imagine you want an assistant to reply politely to customers.

For “My order is late,” one response says, “Just wait.” Another says, “I’m sorry about the delay. Please send your order number.” A reviewer marks the second response as the better choice.

The pair then becomes a training example.

DPO uses many such pairs to adjust the model’s weights toward preferred responses during training. It uses recorded choices directly, without first training a separate reward model to score answers.

Choosing a response records feedback; the training step changes the model.

Analogy

Think of helping someone write customer replies:

You show two drafts & point out which one better follows the company’s standards. Repeated practice guides how the person writes future replies.

DPO guides a model through similar comparisons.

Common mistake

Choosing preferred answers based only on how pleasant they sound.

A polite reply could promise a refund the customer doesn’t qualify for. Review accuracy & company policy alongside tone, so the training examples reward useful behavior.

Why it matters

DPO provides a simpler way to train from preferences than approaches which require a separate reward model and reinforcement learning.

The quality of those preferences shapes the behavior the model learns.

40. Knowledge Distillation

Knowledge distillation trains a student model7 to learn from a teacher model’s outputs. A common goal is to preserve useful abilities in a smaller model.

Imagine a large model recognizes cats & dogs well, but your application needs something smaller.

You show photos to the teacher model & collect its predictions. The student processes the same photos, and the training algorithm compares its predictions to the teacher’s. Then it adjusts student’s weights to bring their predictions closer together.

The teacher’s outputs guide the student’s training.

Some methods also use the teacher’s confidence across possible answers. This gives the student more information than a single label such as “cat.” The student learns through its own weight adjustments.

After training, the student handles new photos independently.

Analogy

Think of a teacher helping a student practice.

The teacher demonstrates answers, and the student learns from those examples. The student attempts new questions later without the teacher beside them.

Practice doesn’t guarantee the student matches the teacher on every question.

Common mistake

Assuming the student inherits every ability the teacher has.

A student trained on cat & dog photos hasn’t necessarily learned to recognize other animals. It could also repeat the teacher’s errors. Test it on separate examples which reflect the job you need it to perform.

Why it matters

Distillation helps teams develop models which need less memory and computation for a chosen task. A smaller student could serve requests faster or run on more limited hardware.

Check whether those savings preserve the quality your application needs.

41. Temperature and Sampling

Sampling is how an AI model chooses the next token during inference. Tokens the model considers likely have a higher chance of being chosen.

Temperature controls how predictable that choice is by adjusting the probabilities to make likely options more dominant or give less likely options more chance.

Ask an assistant to suggest a name for your bakery:

The model builds each suggestion one token at a time. At each step, it assigns probabilities to possible next tokens. Sampling chooses among those options according to their chances.

Different choices lead to different names.

Lower temperature favors the leading options more strongly, usually producing less variation across attempts. Higher temperature gives less likely options more chance, usually producing more varied suggestions.

Yet some results could be unusual/awkward… More variation doesn’t guarantee a better name.

These settings affect token selection without changing the model’s weights.

Analogy

Think of drawing a slip from a bowl.

Several slips carry the same word, while others carry different words. A word with more slips has a higher chance of being selected.

Temperature changes the balance of slips; sampling makes the draw.

Common mistake

Treating temperature as a quality control.

A higher setting doesn’t guarantee originality, and a lower setting doesn’t guarantee accuracy. Review results for readability & relevance, and check factual answers against reliable sources.

Why it matters

Different tasks need different amounts of variation…

Name suggestions benefit from a range of options, while extracting details from a receipt benefits from consistent results.

Test the settings against the task rather than using one value for everything.

§

§

42. Gradient Descent and Optimization

Read more

“Hi Neo, do I really need to understand all these AI concepts? RLHF, vector search, quantization, diffusion models, MCP, hybrid retrieval... It feels like every time I learn one concept, five more appear. Lol”

All I will say is this:

You don’t need to master every AI concept to start building AI apps.

At least, not all at once.

But understanding how these concepts work makes a BIG difference.

Onward.

§

Same model, 2x the fixes: the coding harness is part of the system

Most teams choose a coding agent by its model. AgentField tested the other half: ten coding harnesses on one open model, run on the same 113 real GitHub issues from the DeepSWE benchmark. Results ran from 3 solved issues to 62. The system around the model decides as much as the model does.

CodeAF, a new open-source coding harness by AgentField AI, finished #1 of the ten (method and data are public on GitHub):

• 2x OpenCode’s solves and nearly 4x Claude Code’s, on the same model

• 4.8x less per solved issue than OpenCode, 3.4x less than Claude Code

• Built for open models like DeepSeek, Qwen, GLM and Kimi, and it works with closed models too

• One small Go binary, open source

Try CodeAF on GitHub

(Thanks to AgentField for partnering on this newsletter.)

§

• Share this letter & I’ll send you some rewards for the referrals.

§

This is part 2 of a four-part1 newsletter series:

18. Reinforcement Learning from Human Feedback,

19. Pretraining,

20. Vector Search and Approximate Nearest Neighbors,

21. In-Context Learning,

22. Hallucination, Factuality and Grounding,

23. Quantization,

24. Context Windows and Long-Context Modeling,

25. Structured Outputs and Constrained Decoding,

26. Model Context Protocol,

27. Parameter-Efficient Fine-Tuning,

28. Diffusion Models,

29. Autoregressive Generation and Next-Token Prediction,

30. Self-Supervised Learning,

31. Inference,

32. Retrieval Reranking,

33. Hybrid Retrieval,

34. Chunking and Document Segmentation.

( …and much more in Parts 3 & 4!)

For each, I’ll share:

• What it means & how it works,

• A real-world analogy where useful,

• A common mistake to avoid,

• Why it matters.

Let’s dive in!

§

18. Reinforcement Learning from Human Feedback

Reinforcement Learning from Human Feedback (RLHF) uses human feedback to guide further model training toward preferred responses.

Give reviewers two answers to the same question.

The reviewers compare qualities such as usefulness & how well each answer follows instructions. Those comparisons train a reward model2 to score responses. The language model then receives further training to produce responses with higher scores.

Human preferences help shape what the training process rewards.

This improves behavior according to the chosen criteria, but still needs careful evaluation.

Analogy

Think of a coach reviewing practice performances:

Feedback guides the next attempt. The improvement depends partly on whether the coach rewards the qualities the task actually requires.

Common mistake

Confusing a preferred answer with a “verified” answer.

Reviewers could favor a polished response containing a subtle factual error. Check accuracy separately from tone & helpfulness.

Why it matters

RLHF helps shape a broadly trained model into a more useful assistant. The feedback criteria influence which behaviors the model learns to favor.

19. Pretraining

Pretraining is an initial training stage in which a model learns broad patterns from a large dataset before more specialized training.

Before a model follows your request, it needs a foundation.

For a typical generative language model, pretraining repeatedly tests predictions of the next token3. Prediction errors guide adjustments to the model’s weights. Across many examples, these adjustments help the model learn patterns in the text.

The text itself supplies the next-token answers.

Other models use different tasks, such as predicting missing pieces of an input.

Analogy

Think of a broad education before a specialized course:

The foundation covers many subjects & patterns. Later instruction directs those abilities toward a particular role.

Common mistake

Assuming pretraining produces a ready-to-use assistant.

A model trained to predict text doesn’t automatically respond in the style users expect. Further training & application design help shape how it handles requests.

Why it matters

Pretraining provides capabilities reused across many tasks.

The data influences what the model learns well and where its knowledge has gaps.

20. Vector Search and Approximate Nearest Neighbors

Vector search finds items with similar numerical representations. Approximate Nearest Neighbors (ANN) methods speed up search while accepting the possibility of missing some closest matches.

Imagine searching millions of support documents:

The system converts your question into an embedding4 & compares it with document embeddings with vector search. An exact search finds the closest matches under a chosen comparison rule. An approximate search uses an index, an organized search structure, to reduce the work.

The shortcut helps results arrive faster.

How much accuracy it sacrifices depends on the method & its settings…

Analogy

Think of a librarian directing you to likely shelves…

You search a smaller area instead of checking every book. The shortcut saves time, but you could miss a useful book elsewhere.

Common mistake

Choosing the fastest setup without checking its results.

A quick search is unhelpful if it repeatedly misses the relevant document. Compare retrieval quality, response time & memory use.

Why it matters

• Vector Search: Best for smaller datasets requiring the true nearest matches.

• Approximate Nearest Neighbors: Best for large datasets where search speed & scalability matter more than finding the exact nearest match.

§

§

21. In-Context Learning

In-context learning is a model’s ability to use instructions/examples in its current context without updating its weights.

Show the model a few examples before asking it to label a new message:

The examples demonstrate the categories & response format. The model then uses the pattern to label the next message. No separate training run occurs!

The examples guide the response through the context.

(Remove them from a later request, and you shouldn’t assume the model still has access to them.)

Analogy

Think of filling out a form after examining a completed sample:

The sample shows how the fields work. You apply the pattern immediately without taking a new course.

Common mistake

Providing examples that cover only obvious cases.

If every billing example contains “invoice,” the model could rely too heavily on a specific word. Include varied wording & difficult cases, then test on separate examples.

Why it matters

In-context learning lets you demonstrate a task quickly. It helps you test formats/categories before investing in fine-tuning.

22. Hallucination, Factuality, and Grounded Answers

A hallucination is generated content presented as “true” despite being false/unsupported. Factuality concerns correctness; grounding connects an answer to evidence.

An assistant gives you a confident answer & 3 citations.

One source doesn’t exist. Another exists without supporting the claim. The answer looks well researched, but its appearance tells you little about its accuracy.

The evidence needs inspection too…

A grounded answer uses supplied or retrieved material. It could still repeat an error from an outdated source, so the material's quality matters.

Analogy

Think of a report with footnotes.

The footnotes help you trace the claims. You still need to check whether the sources exist & support what the report says.

Common mistake

Treating a citation as a completed fact-check.

Check the source & compare its content with the answer. Give the system a way to report insufficient evidence instead of filling gaps with guesses.

Why it matters

These distinctions help you find the source of an error. The application could need better documents, better retrieval, and/or stricter checks on the final answer.

23. Quantization

Quantization stores model values using fewer bits, thus reducing memory needs while preserving less numerical details.

A model is too large for your machine's memory.

Its weights are numbers, and storing those numbers takes space. Quantization converts selected values into smaller numerical representations. The model needs less memory, though the conversion introduces approximations.

You trade some precision for a smaller memory footprint.

The effect on output quality depends on the model & the quantization method.

Analogy

Think of recording measurements to the nearest centimeter instead of millimeters.

You preserve less detail. Whether the difference matters depends on what you need the measurements for.

Common mistake

Assuming a smaller model representation always runs faster with identical results.

Speed depends on the hardware & software used to run it. Test both quality and response time before choosing a quantized version.

Why it matters

Quantization helps larger models fit within smaller memory budgets. It creates more deployment options when the quality tradeoff is acceptable.

24. Context Windows and Long-Context Models

A context window is the amount of information a model handles within a single context (measured in tokens). Long-context models support larger amounts of material.

Upload a long manual & ask about a detail halfway through it.

Your instructions, conversation history, document text, and tool results all take up space. In typical text-generation systems, the response also needs room within the overall limit. The application decides what to include & how much space to reserve for the answer.

A larger window holds more material.

Yet it doesn’t guarantee the model uses every detail correctly.

Analogy

Think of a desk covered with reference pages.

A larger desk holds more pages. It doesn’t guarantee you notice every detail or connect the right passages.

Common mistake

Assuming a document that fits is a document the model uses reliably.

Test questions whose answers appear in different parts of the document. Remove irrelevant material or retrieve targeted passages when full-document answers become unreliable.

Why it matters

Context limits affect document analysis, code review & long conversations. Plus the amount of material processed influences cost & response time.

25. Structured Outputs and Constrained Decoding

Structured outputs follow a specified format. Constrained decoding5 restricts what model generates, so the output follows supported format rules.

Your application needs 3 fields from a support ticket.

A paragraph is inconvenient for code to process. A schema, or set of format rules, defines the field names, value types, and permitted choices. Constrained decoding limits the model’s next-token options to keep the response within those rules.

This makes the result easier for software to process.

For example, a structured output could follow a JSON schema like this

{
 “category”: “billing”,
 “priority”: “high”,
 “order_id”: null
}

Analogy

Think of a form with fixed fields & dropdown menus.

The form controls where information goes & which options are allowed. It doesn’t establish whether the selected option is correct.

Common mistake

Confusing correct format with correct information.

A response could follow every format rule and still misclassify a ticket. Check the values and handle refusals or incomplete responses separately.

Why it matters

Structured outputs reduce work needed to turn model responses into usable application data. They solve a formatting problem, while factual checks remain necessary.

26. Model Context Protocol

Model Context Protocol (MCP) is an open standard for connecting AI apps to external tools & information sources.

Your assistant needs access to a document store & a ticket system.

An MCP server exposes capabilities, such as a document search or ticket lookup. The AI application connects through an MCP client, which discovers those capabilities and exchanges requests & results. The shared protocol gives both sides a common way to communicate.

The connection follows a standard format.

(The application & connected services still need to enforce access permissions.)

Analogy

Think of a standard connector between devices:

A shared connector reduces custom wiring. Each device still offers different capabilities & plugging it in doesn’t grant permission to use everything.

Common mistake

Treating an MCP server as automatically trusted.

A server could expose sensitive data or powerful actions. Review its capabilities, limit permissions & decide which actions need approval.

Why it matters

MCP reduces custom integration work across compatible AI applications.

A server’s capabilities become reusable without designing a different communication interface for every application.

27. Parameter-Efficient Fine-Tuning and LoRA

Parameter-Efficient Fine-Tuning (PEFT) adapts a model by training only a small portion of its parameters or added parameters. Low-Rank Adaptation (LoRA) is one method that learns compact additions to selected model weights.

You want a model to follow your support team’s response style.

Full fine-tuning updates all of the model’s weights. LoRA keeps the original weights fixed and trains smaller groups of numbers that represent changes to selected layers. These learned additions influence the model’s output.

The additions are then saved as an adapter6.

The adapter works alongside the base model, or its changes are merged into compatible weights.

Analogy

Think of a company handbook with a short supplement for each department:

The main handbook stays the same. Each supplement adapts to a specific role. A LoRA adapter serves a similar purpose through learned numerical changes.

Common mistake

Assuming a small adapter works with any model.

Its changes depend on the base model & layers used during training. Keep the matching model version and configuration, then test the combined result on separate examples.

Why it matters

PEFT reduces the resources needed to train and store specialized adaptations.

The base model still needs memory and computation, but maintaining several task-specific versions becomes more practical.

§

§

28. Diffusion Models

Read more

He was very good at building apps & deploying them.

Until one day, when the DevOps engineer from his team fell ill.

And he got the task of updating the VPC for their production app.

Yet he had zero knowledge of networking fundamentals.

So his tiny config change took down the entire app…

Onward.

§

The Missing Layer in Your Security Stack (Partner)

Most security tools focus on preventing or detecting an attack.

Password managers protect credentials. Monitoring tools flag suspicious activity. But when a scam succeeds and money is lost, detection alone doesn’t help you recover it.

Coveron adds a recovery layer to your security stack.

As someone who spends most of the day online, I already use tools to reduce security risks, but I like the idea of having a fallback if those protections fail.

Coveron monitors for suspicious activity and exposed personal data, then adds financial protection and hands-on recovery support when something goes wrong.

What you get:

• Up to $100K for eligible scam or online fraud losses

• Up to $100K for cyber extortion or threats

• Up to $2M for eligible identity theft recovery costs

• Real people who guide you step-by-step through recovery

Tools reduce risk. Insurance reduces loss. Coveron is built for what happens when prevention isn’t enough.

One scam can cost you everything — protect yourself now, first 100 users get 20% off with code neocoveron

Get Coveron →

(Thanks to Coveron for partnering on this newsletter.)

§

I spent a few hours putting together this list of foundational networking concepts, so you don’t have to.

Curious to know how many are new to you:

1. Network stack,

2. Encapsulation,

3. IP addressing,

4. Subnets & CIDR,

5. VPC,

6. Routing & forwarding,

7. NAT,

8. BGP,

9. Ports and sockets,

10. VPN.

For each, I’ll share:

• What it is & how it works in simple words,

• A real-world analogy,

• Why it matters.

Let’s go!

§

• Share this letter & I’ll send you some rewards for the referrals.

§

1. Network stack

The network stack divides communication into layers. Each layer handles a specific task & passes data to the next layer.

Two models explain how these layers work:

• OSI Model: The Open Systems Interconnection (OSI1) model has 7 layers. It helps you understand and troubleshoot network communication.

• TCP/IP Model: The Transmission Control Protocol/Internet Protocol (TCP/IP) model has four layers: Application, Transport, Internet, and Link. It reflects how modern internet protocols work.

Think about opening a website:

Your browser sends an HTTP request. The transport layer manages communication between applications, the internet layer routes packets toward their destination, and the link layer delivers data across the local network.

Each layer handles its own responsibility.

So your app does not have to manage packet routing or physical transmission.

Analogy

A letter passing through the postal system:

You write the words. An envelope carries the address; a sorting office picks the route; and a truck moves the physical mail. Each stage trusts the next to do its job.

Why it matters

Imagine your application suddenly loses its database connection…

The problem could occur in several places:

• Application layer: DNS fails to resolve the database hostname.

• Transport layer: A firewall blocks the database’s TCP port.

• Transport/Application boundary: A TLS handshake fails.

• Internet layer: A routing issue prevents packets from reaching the database.

Understanding the network stack helps you identify where communication fails & narrow down the root cause.

Reach for tcpdump when you suspect the transport, and curl when you suspect the application. Pair it with encapsulation, which shows what each layer adds to your data.

2. Encapsulation

Encapsulation is the process of wrapping data with protocol headers as it moves down the network stack.

When you send an HTTP request, each layer adds information needed to deliver data:

1. Application Layer: Creates the HTTP request containing the payload.

2. Transport Layer: Adds a TCP header with source & destination ports, sequence numbers, and control information.

3. Network Layer: Adds an IP header with source & destination IP addresses.

4. Data Link Layer: Wraps the IP packet in an Ethernet frame with a header & trailer.

The receiving machine reverses this process; it’s called decapsulation.

Each layer processes its protocol information & passes remaining data to layer above.

Analogy

A gift in a box, inside a padded parcel, inside a shipping crate.

Each wrapper carries the label its own handler needs, and nobody opens more than one layer.

Why it matters

Every header adds overhead to network communication.

Sending a 10-byte payload over TCP/IPv4 and Ethernet requires:

• TCP header: 20 bytes

• IPv4 header: 20 bytes

• Ethernet header and trailer: 18 bytes

• Application payload: 10 bytes

That’s 58 bytes of protocol overhead for just 10 bytes of payload, excluding Ethernet padding and other link-level overhead.

Small messages consume significantly more bandwidth than their payload size suggests.

Read frames with Wireshark and set path sizes in your virtual private network config. Read it alongside ports and sockets, which explain the transport header fields.

3. IP addressing

An Internet Protocol (IP) address identifies a network interface & helps routers deliver packets to the correct destination.

There are two main versions:

• IPv4: Uses 32-bit addresses, such as 192.168.1.1. It supports 4.3 billion unique addresses.

• IPv6: Uses 128-bit addresses, such as 2001:db8::1. It provides a vastly larger address space.

IPv4 addresses became scarce as the Internet grew…

To conserve public addresses, networks adopted Network Address Translation (NAT). This allows many devices to share one public IPv4 address.

IPv6 addresses this limitation by providing enough addresses for an enormous number of devices & networks.

Plus, many networks support both IPv4 & IPv6, an approach called dual stack.

• Your laptop could have both an IPv4 and an IPv6 address.

• A website could publish both IPv4 (A) and IPv6 (AAAA) DNS records.

• Your device selects a reachable address & connects using corresponding protocol.

IPv6 adoption varies across networks…

Some mobile carriers and cloud providers support it extensively, while many corporate networks still rely on IPv4.

Analogy

Think of IPv4 as a city running out of house addresses.

IPv6 expands the address space so dramatically that there is room for an enormous number of new buildings.

Why it matters

When designing a cloud network, you need to choose which IP versions to support.

• AWS VPC: Configure IPv4-only, dual-stack, or IPv6-only subnets according to your requirements.

• Kubernetes: Configure IPv4, IPv6, or dual-stack networking based on your cluster and Container Network Interface (CNI) support.

Also you need to understand subnetting.

It divides an IP address range into smaller networks for routing & address management.

4. Subnets and CIDR

Read more

Security at the Architecture Level (Partner)

Most online attacks don’t happen in one step.

A phishing link leads to a fake site, the site pushes a malicious download, and the download delivers the payload.

NordVPN approaches this like a defense-in-depth problem.

As someone who works online every day, I already think about security in layers, so having phishing, scam, and malware protection alongside the VPN is useful to me.

NordVPN’s built-in next-gen antivirus aims to intercept threats earlier in the attack pipeline instead of waiting until a malicious file has already reached your device. NordVPN uses a proprietary machine learning model to block phishing sites before they cause harm, while its malware protection scans downloading files and removes them if they’re dangerous.

What you get:

• Anti-phishing protection powered by NordVPN’s proprietary machine-learning model

• Protection against scam websites and fake online stores

• Malware scanning for downloading files

The VPN remains a core part of NordVPN, alongside these additional security features.

Depending on your plan and operating system, one subscription gives you access to multiple security tools.

Get NordVPN 2Y plan with 4 extra months→

Plus, it comes with a 30-day money-back guarantee.

(Thanks to NordVPN for partnering on this newsletter.)

§

HTTP/2 was supposed to make the web faster.

It introduced multiplexing, header compression & better connection reuse.

But one lost TCP packet could still delay every stream on the connection.

HTTP/3 was built to solve this problem.

In this newsletter, we’ll break down 12 essential concepts behind HTTP/2 & HTTP/3:

12. Multiplexing,

13. HPACK header compression,

14. Flow control,

15. Stream prioritization,

16. Server push & early hints,

17. TCP head-of-line blocking,

18. QUIC over UDP,

19. QUIC handshake,

20. Independent streams & loss recovery,

21. Connection IDs & connection migration,

22. QPACK,

23. Alt-Svc & HTTPS DNS records.

For each, I’ll share:

• What it is & how it works in simple words,

• A real-world analogy,

• Why it matters.

Let’s go!

§

• Share this letter & I’ll send you some rewards for the referrals.

§

12. Multiplexing

Multiplexing sends many requests & responses at the same time over ONE connection by interleaving their frames.

The queue disappears…

Request one waits on a slow database while other requests return & render. The server replies in whatever order the data becomes ready.

The result shows up plainly in connection counts:

A desktop page opens fewer connections on HTTP/2 than on HTTP/1.1. The gap roughly doubles on lighter pages, where one connection does the work of two.

Analogy

One conversation channel where each sentence carries the speaker’s name.

Three people talk at once, and nobody loses the thread.

Why it matters

Multiplexing changes how you bundle because many tiny files beat a large one for cache reuse.

Confirm which protocol your assets use with the devtools protocol column and measure the bundling change in WebPageTest. Yet TCP head-of-line blocking1 marks the limit multiplexing could never remove.

13. HPACK Header Compression

HPACK is the header compression format for HTTP/2.

It’s built from a static table, a dynamic table & Huffman coding2.

Headers repeat almost perfectly. Your user agent string, accept headers, and cookies go out identically on every request, and on an API-heavy page they outweigh the payload.

HPACK indexes them:

A fixed table of common header names covers the obvious cases, so a method line becomes a single byte. Headers you have already sent go into a dynamic table shared by both ends.

The second request references those entries by number rather than repeating them, and the saving compounds across every request on the connection.

Analogy

Shorthand between two colleagues who talk daily.

Once you both agree a short name means one specific document, you stop saying the full title.

Why it matters

Header weight decides real latency on pages carrying large authentication tokens.

Inspect encoded header sizes in Wireshark and compare them against the devtools network panel. QPACK rebuilds this idea for out-of-order delivery.

14. Flow Control

Flow control limits how much data a sender may have outstanding, applied per stream & across the whole connection.

The receiver sets a flow-control window. This limits how much DATA the sender can transmit before receiving additional flow-control credit. A WINDOW_UPDATE frame grants more room as the receiver consumes data, and the default initial window is 65,535 bytes.

This mechanism protects slow consumers3:

A browser decoding a large image should slow the pipe rather than buffer forever, and a proxy feeding a slow client behaves the same way.

Also it stops one greedy stream from monopolizing a connection everything else depends on.

Analogy

A coworker saying, “Send me five files at a time, and I will ask for more.”

They control the pace, so nothing piles up on their desk.

Why it matters

Large uploads, gRPC streams, and video segments all hit this ceiling before they hit bandwidth.

Watch window updates with nghttp and confirm socket-level throughput with ss4. Independent streams show how QUIC applies the same idea per stream.

§

§

15. Stream Prioritization

Stream prioritization lets a client tell the server which responses matter most, so the server can send that data first.

A browser needs CSS to render a webpage. Giving CSS higher priority than a large image helps the page become usable quicker, even if the total amount of data stays the same.

HTTP/2 used a complex system of stream dependencies & weights to decide which data to send first. But browsers & servers struggled to implement it consistently!

Then a simpler priority scheme got introduced... It uses 2 signals:

• Urgency (0–7): Indicates how important a response is. Zero is the highest priority & seven is the lowest.

• Incremental flag: Indicates whether a response can be delivered progressively, thus allowing the server to interleave its data with other responses.

Also browsers provide the Fetch Priority API.

This lets developers influence the priority of resources using attributes such as fetchpriority="high".

The browser uses these hints when scheduling requests and assigning HTTP priorities, although the Fetch Priority API does not directly expose HTTP’s urgency values.

Adoption of the Fetch Priority API grew rapidly after Chrome introduced support.

Analogy

A triage nurse.

Everyone in the waiting room gets seen & order reflects who needs help soonest.

Why it matters

One attribute on your largest visible image moves a headline performance metric.

Set fetchpriority in your markup and verify the assigned value in the Chrome devtools priority column. Early hints reach the same goal from the server side.

16. Server Push and Early Hints

Server Push lets a server send resources to a browser before browser requests them.

It was one of HTTP/2’s most promising features.

But it had its problems:

Servers pushed files already stored in the browser’s cache, thus wasting bandwidth. Also performance improvements were inconsistent, so adoption remained low.

(Chrome eventually disabled Server Push by default, and Firefox removed support.)

Early Hints emerged as a simpler alternative:

Instead of sending the resource itself, the server sends an HTTP 103 Early Hints response containing Link headers while preparing the final response.

The browser then uses these hints to:

• Preconnect: Establish connections to other servers before they’re needed.

• Preload: Start downloading crucial resources, such as CSS files, before the final HTML arrives.

This helps the browser start useful work earlier & potentially render the page faster.

Yet adoption remains limited… Only 2.9% of desktop pages used Early Hints, with Shopify accounting for roughly 90% of those deployments.

Analogy

Push is a waiter setting down a dish you never ordered & may already have.

Early hints is a waiter handing you the menu while the kitchen warms up.

Why it matters

Dead push configuration survives in production long after browsers stopped honoring it.

Enable early hints at Cloudflare/Fastly and prove the gain with real user monitoring rather than a lab score. Chunked transfer encoding fills the same dead time once the response finally starts.

17. TCP Head-of-Line Blocking

TCP delivers data to applications in a strict order.

If a TCP packet gets lost, later data must wait until the missing packet gets retransmitted. Yet this creates a problem for HTTP/2…

HTTP/2 multiplexes multiple streams over one TCP connection. And TCP sees only a single ordered sequence of bytes, with no knowledge of individual HTTP/2 streams. So one lost TCP packet delays data delivery across multiple streams, even when their data has already arrived.

This is called TCP head-of-line (HOL) blocking.

TCP uses acknowledgments & retransmissions to provide reliable, ordered delivery.

But this ordering can delay otherwise available data… On lossy networks, HTTP/2 using one TCP connection sometimes performs worse than HTTP/1.1 using six parallel connections.

Each TCP connection handles packet loss independently, so a lost packet on one connection does not directly block the others.

Analogy

One conveyor belt at a sorting depot.

Drop a box & everything behind it stops moving, even if those parcels have nothing to do with each other.

Why it matters

This single property is the entire reason HTTP/3 exists.

Confirm retransmits with ss -i and capture the stall itself with tcpdump5. QUIC over UDP replaces the transport rather than working around it.

§

§

18. QUIC over UDP

Read more

Live demo: give AI agents the access they need without credentials they can leak

Infisical’s Agent Vault gives AI agents access to services without ever handing them secrets. A proxy attaches credentials on the way out, with no code changes. You control what agents can reach and when access expires.

See it in a 30-minute live demo on Wednesday, October 21 at 1 PM ET.

Save your spot

(Thanks to Infisical for partnering on this newsletter.)

§

Why does a site take several seconds to load, even if server responds in milliseconds?

Before a single page appears, the browser must establish connections, send requests, and receive responses…

Understanding underlying mechanisms changed how I think about HTTP & web performance.

In this newsletter, I’ll break down must-know HTTP concepts. Some of these are foundational & some are quite advanced.

All of them are useful to engineers building production-ready systems:

1. Request-response cycle,

2. TCP handshake,

3. Persistent connections,

4. Pipelining,

5. HTTP head-of-line blocking,

6. Chunked transfer encoding,

7. Caching & conditional requests,

8. Parallel connections & domain sharding,

9. TCP slow start,

10. The binary framing layer,

11. Streams, messages & frames.

(…and much more in Part 2!)

For each, I’ll share:

• What it is & how it works in simple words,

• A real-world analogy,

• Why it matters.

Let’s go!

§

• Share this letter & I’ll send you some rewards for the referrals.

§

1. Request-Response Cycle

Every HTTP exchange is one request and one response.

The client sends a method, a path, and a set of headers. The server replies with a status code, its own headers, and usually a body. In HTTP/1.x, the message format is text-based, so you could type a request by hand into a terminal and read the reply with your own eyes1.

The earliest version of the protocol supported a single method and returned a single file type. The next version added headers, status codes, POST, and HEAD.

Also, it opened a fresh connection for every exchange and threw it away afterward. (This decision shaped the next 20+ years of workarounds.)

Analogy

Ordering by postcard:

You write one item, mail it, and wait for one parcel.

To order a second item, you write a second postcard.

Why it matters

Debugging an unexpected status code starts with reading the raw exchange, not your application logs.

Send requests by hand with curl and inspect real traffic in the Chrome DevTools Network tab. The binary framing layer picks up from here, keeping these semantics and changing the bytes.

2. TCP Three-Way Handshake

Transmission Control Protocol (TCP) opens a connection with 3 messages before either side sends data.

1. Client sends a synchronize packet.

2. Server replies with an acknowledgment plus its own sequence number.

3. Client acknowledges the number & connection opens.

Three messages cost one full round trip…

i.e., on a link with 100ms of latency, you burn 100ms before the first byte of your request leaves the machine.

Encryption adds more on top…

A Transport Layer Security (TLS) 1.2 handshake costs two extra round trips, and TLS 1.3 cut that to one.

Analogy

A phone call:

You dial, they say hello, and you say hello back. And only on the fourth breath does anyone say something useful.

Why it matters

Connection setup shows up in your p99 long before your handler code does.

Measure it with curl -w and watch it across a full page load in WebPageTest. Persistent connections spread that cost across many requests.

3. Persistent Connections

A persistent connection stays open after a response, so the next request reuses it.

HTTP/1.1 made this the default behavior. The connection lives until either side sends a close header or a timeout fires, so one handshake now covers dozens of requests.

The saving compounds on real pages.

A page with 40 assets used to pay 40 handshakes; on a single persistent connection, it pays one.

Analogy

Keeping the phone line open between questions instead of hanging up and redialing every time you think of something new.

Why it matters

Connection churn on a busy proxy exhausts temporary ports well before it exhausts bandwidth2.

Tune keepalive_timeout in nginx and server.keepAliveTimeout in Node.js, then check both against your load balancer. TCP slow start explains why a reused connection also runs faster.

§

§

4. Pipelining

Pipelining lets a client send several requests back-to-back on one connection without waiting for each response.

The browser asks for two images at once rather than asking, waiting, then asking again. On paper, it removes a full round trip per request while using nothing more than the connection you already have.

Yet one rule ruined it:

Responses must return in the exact order the requests went out, so a slow first response stalls everything behind it. Buggy proxies mangled pipelined traffic on top of that.

(NOTE: Because of these problems, major browsers disabled HTTP/1.1 pipelining, and it never gained widespread adoption.)

Analogy

Ordering all three courses at once:

The kitchen still serves them in the order you named them, so a slow starter holds up your main.

Why it matters

You will find pipelining disabled in curl and every browser, and now you know why.

Confirm the behavior with curl and watch real request ordering in Wireshark3. Multiplexing solves the same problem without the ordering rule.

5. HTTP Head-of-Line Blocking

Head-of-line blocking occurs when first item in a queue delays every item behind it4.

One HTTP/1.1 connection processes one exchange at a time:

A slow database query on request one could keep four other requests waiting, and the network sits idle while the queue holds.

This is the ceiling HTTP/1.1 could never lift… Persistent connections removed the handshake cost and left the queue completely intact.

Every workaround built on top of it, from parallel connections to asset bundling, exists to hide this single property.

Analogy

A single supermarket checkout:

The shopper at the front needs a price check, and the four people behind them stand still.

Why it matters

Stair-stepped requests with visible gaps in a waterfall chart point straight at this.

Read the waterfall in WebPageTest and count concurrent requests in the Chrome DevTools Network panel. Parallel connections show the workaround the browsers actually picked.

6. Chunked Transfer Encoding

Chunked transfer encoding lets a server send a response in pieces without knowing the total size in advance.

The server marks the response as chunked and writes each piece with its size in hexadecimal on the line above. A “zero-length” chunk marks the end.

The browser could start rendering before the full response arrives.

In HTTP/1.1, chunked transfer encoding supports streaming generated HTML, server-sent events, and token-by-token model responses.

HTTP/2 and HTTP/3 use DATA frames instead.

Analogy

A publisher shipping a book one chapter at a time.

You start reading chapter one while the author is still writing chapter four.

Why it matters

Streaming responses depend on the HTTP version’s framing mechanism, and accidental buffering often stops streaming.

Check proxy_buffering in nginx and res.flushHeaders in Express before you blame your application code. Early hints apply the same idea to the wait before a response starts.

7. Caching and Conditional Requests

Caching stores a copy of a response so a later request skips the network entirely.

Cache-Control sets the rules:

A max-age directive says how long a copy stays fresh; no-cache forces a check on every use, and immutable promises the file never changes under its URL.

Conditional requests handle the check cheaply:

The client sends its stored entity tag in an If-None-Match header, and a server holding the same version answers 304 Not Modified without a body.

A cache hit costs zero bytes on the wire and zero milliseconds of server time, thus making it the cheapest performance work available to you.

Analogy

Asking a librarian whether the report changed since your last visit.

If nothing changed, you read the copy already on your desk.

Why it matters

Static assets & API responses need opposite caching policies, and mixing them up costs freshness or bandwidth.

Set the rules at the edge in Cloudflare/Varnish and read the real cache hit ratio there rather than trusting your config. Conditional requests lose their value once you version filenames properly.

§

§

8. Parallel Connections and Domain Sharding

Browsers open several TCP connections (typically 6) to the same origin at once.

Six lanes beat one lane, so head-of-line blocking hurts less. Sites wanting more than six invented extra hostnames, serving images from one subdomain & scripts from another. That bought another six per name.

This technique is called domain sharding… It shaped front-end architecture.

But it became obsolete the moment multiplexing arrived… yet survives in production long after the sites that used it moved on…

Analogy

Opening more checkout lanes.

When six lanes fill up, you rent the shop next door and open six more.

Why it matters

Removing dead shards is one of the easier wins available on an older codebase.

Measure the before and after in WebPageTest and let Lighthouse5 flag the redundant origins. TCP slow start explains why those extra connections start slow.

9. TCP Slow Start

Slow start is a congestion control algorithm where a new connection begins with a small send window & then grows it each round trip.

TCP has no idea how much bandwidth sits between you and your user. Guessing high floods the path and drops packets, so it guesses low and probes upward instead.

The window typically starts at 10 packets, roughly 14 kilobytes, and doubles on every successful round trip. Packet loss halves it and switches the sender to a slower climb6.

Without this algorithm, every new connection would flood the path and collapse throughput for everyone sharing it.

Analogy

Merging onto a motorway:

You accelerate in stages instead of flooring it blind into traffic you cannot see yet.

Why it matters

The first 14 kilobytes of your HTML carry more weight than the next 500.

Put critical CSS & your hero image reference inside that window and let Lighthouse confirm the render path. Persistent connections keep the window you already earned.

10. Binary Framing Layer

Binary framing layer encodes HTTP messages as binary frames instead of plain text lines.

Each frame carries a length, type, set of flags, and a stream identifier in a fixed nine-byte header. The methods, headers, and status codes stay exactly as they were.

You still send GET… You still read 404. The bytes on the wire look completely different, and frames parse faster and pack tighter than text ever did.

The stream identifier is the important part, and the next concept explains why.

Analogy

Shipping containers replacing loose cargo.

The goods are identical, and the uniform box is what lets a crane stack, sort, and interleave them at speed.

Why it matters

Reading HTTP/2 traffic needs different tools than reading HTTP/1.1 traffic.

Dump frames with nghttp and decode full sessions in Wireshark with a TLS key log. Streams & frames explain what the identifier inside each frame does.

11. Streams, Messages, and Frames

A connection carries many streams; each stream carries one request-response pair, and each message splits into frames tagged with a stream identifier.

i.e., three levels, one job.

The identifier lets frames from different requests share the wire and still reassemble correctly at the far end.

Numbering follows a simple rule:

Client-initiated streams take odd numbers, server-initiated streams take even ones, and identifiers always increase.

A stream costs only a few bytes per frame to label; this is what makes concurrency on a single connection practical.

Analogy

One highway with many lanes.

Every car carries a destination label, so they travel together and still arrive at different lanes.

Why it matters

Default stream limits rarely match the concurrency your service actually sees.

Set http2_max_concurrent_streams in nginx and the matching limit in Envoy against your real p99. Flow control decides how much data those streams may send.

§

Final Words

HTTP/1.1 made the web faster, but it left one major problem unsolved: responses still had to arrive in order on each connection.

HTTP/2 addressed this with multiplexing… But TCP’s head-of-line blocking remained.

In Part 2, we’ll explore how HTTP/2 & HTTP/3 tackled these bottlenecks, and why the web needed another major upgrade.

If you find this newsletter valuable, share it with a friend, and subscribe if you haven’t already. There are group discounts, gift options, and referral rewards available.

Subscribe now

Want to reach 250K+ tech professionals at scale? 📰

If your company wants to reach 250K+ tech professionals, advertise with me.

Thank you for supporting this newsletter.

You are now 250,001+ readers strong, very close to 251k. Let’s try to get 251k readers by 17 October. Consider sharing this letter with your friends and get rewards.

Y’all are the best.

Share

References

• RFC 9110, HTTP Semantics: https://www.rfc-editor.org/rfc/rfc9110.html

• RFC 9111, HTTP Caching: https://www.rfc-editor.org/rfc/rfc9111.html

• RFC 9112, HTTP/1.1: https://www.rfc-editor.org/rfc/rfc9112.html

• RFC 9113, HTTP/2: https://www.rfc-editor.org/rfc/rfc9113.html

• RFC 7541, HPACK: Header Compression for HTTP/2: https://www.rfc-editor.org/rfc/rfc7541.html

• RFC 9218, Extensible Prioritization Scheme for HTTP: https://www.rfc-editor.org/rfc/rfc9218.html

• RFC 8297, An HTTP Status Code for Indicating Hints: https://www.rfc-editor.org/rfc/rfc8297.html

• RFC 9000, QUIC: A UDP-Based Multiplexed and Secure Transport: https://www.rfc-editor.org/rfc/rfc9000.html

• RFC 9114, HTTP/3: https://www.rfc-editor.org/rfc/rfc9114.html

• RFC 9204, QPACK: Field Compression for HTTP/3: https://www.rfc-editor.org/rfc/rfc9204.html

• HTTP Archive, Web Almanac, HTTP chapter: https://almanac.httparchive.org/en/2024/http

• Chrome for Developers, Removing HTTP/2 Server Push from Chrome: https://developer.chrome.com/blog/removing-push

But I used to forget important formulas & theorems quickly… and because of this, it became difficult to apply them when solving new problems.

Until one day, I started making notes on key ideas & revisiting them regularly. This helped me absorb what I learned and become better than my past self.

Now I’ve created a playbook to help you learn AI engineering.

It gives you:

• Core concepts behind modern AI systems.

• Condensed notes to understand the AI engineering stack.

• Must-know techniques to build, deploy, debug & scale AI apps.

(Wait a few minutes for the welcome email & check all folders.)

PS - This newsletter exists because of your support. Thank you.

• Share this letter & I’ll send you some rewards for the referrals.

Traditional software gets cheaper per user as it scales.

Yet AI software on an API does not!…

Every request runs a model, every model call has a per-token price, and the price stays “flat” whether ten people use it or a million do.

Self-hosting changes this, as GPU costs amortize across users.

But it introduces a new problem: capacity planning, hardware management, and a minimum spend that has to be justified before the first user arrives.

So scaling an AI product forces cost decisions most software never encounters, and most engineering teams face them only after users have already arrived.

Building an AI system that produces correct, grounded, and safe responses is one challenge…Serving those responses reliably and affordably to a million users is another.

This newsletter shows you how to redesign a chat assistant for large-scale production, reduce inference costs & build the infrastructure needed to handle millions of users.

§

Make your agents realtime (Partner)

Agents are killing your apps with compounding latency across workflows.

Your customers want their agent workflows to be efficient, so you’re losing deals if you don’t make them event-driven. Webhooks solve this issue, but implementing them is harder than it looks.

If you homebrewed your webhooks and are getting pinged at 4 in the morning for failed deliveries, this is for you.

Svix handles everything: retries, idempotency, security, and compliance. Qualified startups get $12,000 in free credits.

Get started for free

(Thanks to Svix for partnering on this newsletter.)

§

I want to introduce Louis-François Bouchard as a guest author.

He’s a best-selling author (Building LLMs for Production), the co-founder of Towards AI, and the creator of the YouTube Channel, What’s AI, where he helps people understand AI and learn how to apply it in the real world. Through his development work with clients, content, teaching, AI training programs, and the new Mentorship program at Towards AI Academy, Louis focuses on making AI practical for builders, engineers, and curious learners alike.

At Towards AI, he and his team train AI engineers through courses and mentorship built for every stage, from beginner to advanced. That educational mission and the real-world experience building for his clients are exactly why I wanted him in this newsletter series.

§

Here’s what’s inside this newsletter:

• Your $193,000 AI bill could drop to $48,000. Three optimizations cut the projected monthly cost by 75% at 100,000 daily conversations. Most teams overlook where the biggest savings actually come from.

• Your smartest AI model could be your biggest money leak. Up to 70% of requests in our example go to a cheaper model, while the expensive one handles the rest. The surprising part is what happens to the total bill.

• At 100K users, renting AI could beat owning GPUs. A $48,000 API bill competes with a $49,560 self-hosted setup. At 1 million daily conversations, the economics flip dramatically.

• One GPU benchmark could wipe out $182,000 in monthly savings. Self-hosting looks 38% cheaper at scale, until lower throughput forces you to rent dozens of additional GPUs.

• Your AI could pass every test and still break in production. The hidden gaps between evaluations, guardrails, and real user behavior, and why production failures need to become tests before the next release.

§

Should You Plan for a Million Users?

For most AI products, planning for a million users too early wastes time & money.

An API key and a credit card are often enough to launch a prototype. You should focus on building a useful product before investing in complex infrastructure.

But as your user base grows, these challenges become HARD to ignore:

• Cost increases with usage. A chat assistant costs approximately $0.064 per conversation using GPT-5.4.

  • At 100 daily users, the monthly model bill reaches roughly $193.

  • At 100,000 daily users, it grows to $193,000.

  • At one million, it reaches $1.93 million.

  Model routing & caching reduce the cost per conversation, but total spending still grows with usage.

• Latency increases under heavy traffic. An AI system that responds in 400 milliseconds with ten concurrent users could take two seconds with a thousand. As incoming requests approach the system’s serving capacity, queues grow & response times increase.

• Small prompt changes become expensive. Adding 200 tokens to your system prompt across one million daily requests produces six billion additional input tokens per month. At GPT-5.4’s input price of $2.50 per million tokens, that adds $15,000 to your monthly bill before prompt caching.

• API rate limits restrict scalability. AI providers enforce limits on requests & tokens per minute. As traffic grows, these limits could prevent your application from serving more users, even when your infrastructure has spare capacity.

• Unit economics determine profitability. If each conversation costs more than the revenue it generates, acquiring more users increases your losses. You need to measure cost per conversation against the revenue it produces.

These challenges determine when your AI application needs more advanced production infrastructure.

So before moving beyond a managed API, evaluate these four constraints:

• Unit economics: Does the cost per successful conversation support your target profit margin?

• Service level objectives (SLOs): Does the system meet your latency, availability, and reliability targets?

• Rate limits: Does your provider support your expected request and token volume?

• Data residency: Does the provider meet your requirements for where user data gets processed & stored?

As long as your API provider meets these requirements, staying with managed infrastructure is often the “simplest” choice.

The rest of this newsletter shows you how to estimate costs, identify scalability bottlenecks, and decide when more advanced infrastructure becomes necessary.

Share

§

Target System Design

Our goal is to scale an AI chat assistant to one million daily users while keeping inference costs under control.

For simplicity, we assume each daily active user starts one conversation. Each conversation contains eight requests to the language model.

Let’s define the token usage for each request:

• Static input: 1,024 tokens for the system prompt and a fixed few-shot examples. This meets OpenAI’s minimum prefix length for prompt caching.

• Dynamic input: 400 tokens for conversation history & current user message.

• Output: 300 billed tokens generated by the model.

Each request therefore processes 1,424 input tokens & generates 300 output tokens.

With eight requests per conversation, the total becomes:

• Input: 11,392 tokens per conversation.

• Output: 2,400 tokens per conversation.

(These are illustrative assumptions. In production, you should replace them with actual token usage collected from your application.)

System Requirements

Our production system must meet seven requirements:

• Cost: Keep model costs below a budget of $0.02 per conversation. Determine your actual budget from revenue per successful conversation and required gross margin.

• Latency: Define service level objectives (SLOs) for each workload, including time to first token (TTFT) for interactive chat & completion deadlines for asynchronous jobs.

• Availability: Maintain 99.9% uptime, including during model & prompt updates.

• Safe deployments: Support model changes with a reliable rollback mechanism.

• Observability: Record token usage, cost, latency & model information for each request.

• Feedback: Collect user feedback & incorporate reviewed examples into evaluation datasets.

• Cost forecasting: Estimate the financial impact of every change before deployment.

Our Approach

We’ll make five major design decisions:

1. Cost engineering: Reduce model costs by 50% to 80% through model routing, semantic caching, and prompt optimization.

2. Inference optimization: Explore quantization, GPU utilization, and serving frameworks to determine when self-hosting makes financial and technical sense.

3. Production infrastructure: Handle concurrent traffic, control overload, and deploy model updates without disrupting users.

4. Data flywheels: Use production feedback to improve evaluation datasets & model quality over time.

5. Capacity and cost estimation: Calculate infrastructure requirements & monthly costs at 10,000, 100,000, and one million daily users.

We’ll start with cost engineering because the budget per request influences every infrastructure decision that follows…

§

Part 1: Cost Engineering

Let’s dive in!

Three Types of Token Costs

Every request to a language model creates costs based on token usage.

For this example, we’ll focus on three types:

• Uncached input tokens: Tokens you send to the model, including the system prompt, conversation history, and user message, without a cached-input discount.

• Cached input tokens: Previously processed input prefixes that the provider reuses at a discounted rate.

• Output tokens: Tokens the model generates, including any billable reasoning tokens.

Output tokens typically cost more than input tokens because generating tokens one at a time (decode) has lower throughput than processing input tokens in parallel (prefill).

Some providers charge separately for extra features:

• Cache writes: Anthropic charges approximately 1.25 to 2 times the standard input rate to create a cache entry, depending on its duration. Cache reads cost approximately 10% of the standard input rate.

• Tool usage: Certain tools or external services introduce extra charges.

• Priority processing: Higher-priority inference tiers could have different pricing.

• Data residency: Regional processing requirements could affect the final bill.

So check your provider’s billing structure before estimating production costs.

For the two OpenAI models below, output tokens cost six times as much as uncached input tokens.

These prices are a historical snapshot… Model availability & pricing change over time, so verify the latest rates before making deployment decisions.

Next, let’s use these token prices to calculate the monthly cost of serving thousands/millions of conversations.

Estimation Funnel

Estimate costs before building the system.

Start with requests per day, estimate input, output, and cached-input tokens per request, then apply their respective rates. Multiply the daily cost by the number of days to get the monthly bill.

The arithmetic is straightforward…The challenge is accounting for every cost.

Retries & failed calls increase usage.

Agent loops multiply model calls per task. Embeddings and reranking add separate costs. Image and video processing introduce media-specific charges.

Semantic caching adds infrastructure costs, while provider-side prompt caching may introduce cache-write charges. Storage, network egress, observability, and idle GPU capacity when self-hosting also contribute to the total.

Even a complete cost estimate does not tell you whether the system produces useful answers. Track cost per successful task alongside total spending to measure efficiency.

Cost per Successful Task

Cost per successful task measures total spending divided by the number of tasks completed successfully.

Track it alongside quality and latency to understand the actual cost of delivering useful results. For example, suppose each conversation costs $0.064 on average. If 10% of conversations fail, the cost per successful conversation increases to approximately $0.071, assuming the average cost already includes failed attempts and retries.

The same principle applies to self-hosted models. Estimate the required token throughput, translate it into GPU capacity & calculate infrastructure costs.

H100 GPU rentals may cost around $2 to $6 per GPU-hour, depending on the provider, region, and rental terms. Newer Blackwell-class accelerators may have higher hourly rental prices.

But GPU rental prices alone do not determine inference cost:

Utilization, batching efficiency, memory requirements & model quality also affect the cost per successful task.

Faster or Cheaper, and When You Get Both

Latency & cost typically involve tradeoffs.

More powerful GPUs and lower batching levels reduce response times but increase cost per request.

Some optimizations improve both.

Smaller models require fewer resources, while shorter prompts reduce input processing. Prompt caching, quantization, and efficient attention kernels also reduce latency and cost, depending on the workload.

Streaming lets users see the response as it’s generated.

This improves perceived responsiveness without necessarily reducing total computation. Measure time to first token (TTFT), total response time & stream reliability separately.

Batching improves GPU utilization by processing multiple requests together.

Yet excessive batching can increase tail latency (p99). So interactive applications need different batching configurations from background workloads.

For tasks that do not require immediate responses, consider a provider’s Batch API. These asynchronous endpoints process requests within a specified window at a lower price.

OpenAI’s Batch API offers a 50% discount on supported workloads with a 24-hour completion window. Other providers have different pricing and processing limits.

Reasoning-Token Line Item

Some reasoning models generate internal reasoning tokens that users never see.

These tokens count toward output usage & get billed at output token rates.

This increases costs when switching models. Different models use different default reasoning-effort settings. If you leave the setting unspecified, a model upgrade could generate more reasoning tokens and increase the bill, even when the visible response stays the same.

Set reasoning effort explicitly where supported.

Track reasoning-token usage and total output costs for each request. Increase reasoning effort only when evaluations show enough improvement in answer quality to justify the extra cost.

The Anchor: 100K Conversations a Day

Picture an AI assistant handling 100,000 conversations per day, with eight model requests per conversation.

Each request uses 1,424 input tokens and 300 billed output tokens.

Over 30 days, the system processes 34.176 billion input tokens and 7.2 billion output tokens.

At GPT-5.4 rates of $2.50 per million input tokens and $15 per million output tokens, the monthly cost is:

• Input: $85,440

• Output: $108,000

• Total: $193,440 or $0.064 per conversation.

Suppose 70% of requests use GPT-5.4-mini, which costs $0.75 per million input tokens and $4.50 per million output tokens.

Assuming identical token usage across both models, routing reduces the monthly bill to approximately $98,635, a 49% reduction.

Plus, adding semantic caching, provider-side prompt caching, and shorter prompts could further reduce costs. Under a scenario with a 30% semantic-cache hit rate and additional prompt optimizations, the monthly bill could approach $48,000.

That represents approximately 75% savings compared with the original system.

Actual savings depend on traffic patterns, cache hit rates, token usage & model quality. Validate these assumptions against production measurements before relying on the estimate.

Model Routing

Model routing sends simpler requests to a cheaper model & complex requests to a more capable model.

A lightweight classifier or a smaller language model could decide which model should handle each request.

At scale, this significantly reduces inference costs. For example, routing 70% of requests to a cheaper model could lower the monthly bill. But the actual split depends on your workload and should come from production traffic & evaluation results.

The biggest risk is routing a complex request to a model that cannot handle it reliably. The smaller model may produce a confident but incorrect answer, reducing quality without an obvious failure.

To prevent this, evaluate the route system against the original system using a representative test dataset. Measure task success, factual accuracy, relevance, and other application-specific metrics.

The router should reduce costs while keeping quality within predefined acceptance thresholds. Track routing accuracy and cost per successful task to verify that the savings are worthwhile.

Semantic Caching

Model routing reduces costs by sending simpler requests to cheaper models.

Semantic caching goes further by reusing previously generated answers for similar questions, thus avoiding repeated model inference.

Semantic caching uses embeddings to find previously answered questions with similar meanings. If a cached answer passes the similarity and validity checks, the system returns it without calling the language model.

GPTCache is one open-source library for implementing this approach.

Cache effectiveness depends on traffic patterns. Frequently repeated questions, such as FAQs, produce higher hit rates, while unique or context-dependent queries benefit less.

Yet two failure modes require attention:

1. False cache hit

Two questions may have similar embeddings but require different answers.

Reusing an incorrect or outdated response silently reduces answer quality. Similarity thresholds, time-to-live (TTL) policies, and extra validation help reduce this risk.

2. Cross-user data leakage

Cache entries must be isolated by tenant & authorization scope.

Cache validity should account for relevant prompt, model, and retrieval-index versions, along with data freshness requirements. Authorization checks must remain enforced even when a cached answer exists.

Personalized responses should only be cached within properly isolated scopes.

Requests that perform actions or modify data must execute the required operations rather than replay a cached response.

Finally, semantic caching has its own costs, including embedding generation, vector search, and cache storage. So measure cache hit rate, false-hit rate, latency, and net cost savings to determine whether it improves the system.

Prompt Optimization

The system prompt gets included in every model request.

So its length directly affects input-token usage and cost. In our example, the static prefix accounts for nearly three-quarters of each request’s input tokens.

Shortening instructions and removing unnecessary few-shot examples reduce token usage. But the actual savings depend on how much of the prompt already benefits from caching. Plus, removing important instructions reduces answer quality, so evaluate every change before deployment.

Prompt caching introduces another consideration:

Providers reuse previously processed input when requests share an identical token prefix. To maximize cache hits, place reusable instructions and examples first, followed by dynamic content such as user questions and retrieved documents.

For OpenAI’s automatic prompt caching, eligible prompts must contain at least 1,024 tokens. Shortening a reusable prefix below this threshold eliminates its caching benefit and potentially increases costs, even when the prompt becomes shorter.

In our cost example, we retain the 1,024-token static prefix and optimize the dynamic portion instead. This preserves the opportunity for prompt-cache savings while reducing unnecessary input tokens.

The goal is to minimize total input cost without sacrificing response quality.

Compare cached-token usage, total spending, and evaluation results before deciding which prompt version performs best.

When Smaller Models Are Good Enough

Smaller models significantly reduce inference costs, but only if they maintain acceptable answer quality.

Evaluate each candidate model against a representative test dataset that includes common requests, complex queries, and edge cases. Measure task success, factual accuracy, relevance, and other application-specific metrics.

If the smaller model meets predefined quality thresholds, it passes the quality gate. Before deployment, it must also satisfy latency, reliability, security & rollback requirements.

Use evaluation results to decide when a smaller model is sufficient rather than relying on model size or benchmark rankings alone.

Model routing reduces average inference costs by selecting cheaper models, while semantic caching avoids repeated model calls.

The next step is optimizing the inference stack itself to reduce the computational cost of generating each token…

§

§

Part 2: Inference Optimization and Self-Hosting

Read more

• Share this letter & I’ll send you some rewards for the referrals.

Every task you give Claude runs while you sit there & watch it.

But some work does not need your presence at all… You can check your website for availability issues daily, watch a repo for changes, or let Claude send you a weekly report via Slack.

Anthropic created two different features for this:

• Claude Code calls them Routines1,

• Cowork2 calls them Scheduled Tasks.

You could create them from scratch or simply by prompting Claude.

Some of these automations run on your laptop. To make sure they work, your laptop needs to stay awake. But you can also run your code in the cloud on Anthropic’s servers. This works even when your laptop is closed.

Let’s build two of them in this newsletter & explore their limitations.

At the end, we’ll also look at guardrails, because we all know that conversations between us and AI can leak, and we should take steps to make them safer.

Let’s get started!

§

[Webinar] How to stop babysitting your agents (Partner)

Agents can generate code. Getting it right for your system, team conventions, and past decisions is the hard part. You end up wasting time and tokens in correction loops.

More MCPs give agents access to information but not understanding. The teams pulling ahead use a context layer to give agents exactly what they need.

Join live on Oct 7 (FREE) to see:

• Where teams get stuck on the AI maturity curve

• How a context layer solves for quality, efficiency, and cost

• Live demo: the same coding task with and without a context layer

Register now

(Thanks to Unblocked for partnering on this newsletter.)

§

I want to introduce Gencay as the guest author.

He has been building with AI since ChatGPT first shipped in 2022.

For years, he built for clients on Upwork: 7,300+ hours of it.

Now he builds in the open and shares every piece of it on LearnAIWithMe, every Monday, Wednesday & Friday.

Not just theory for its own sake. One real build at a time, from idea to running system.

§

Here’s what you’ll find inside this newsletter:

• What a schedule changes about an agent. Routines in Claude Code, Scheduled Tasks in Cowork, and why a job that runs without you is a different kind of worker than one you watch.

• Where your automation actually runs. Local routines that need your computer awake, cloud routines that keep running with the laptop closed, and the triggers (schedule, Git event, API) and delivery options (push, email, Slack) the cloud version adds.

• Headless mode, the engine underneath. claude -p runs Claude with no chat window at all, and every scheduled job is built on it.

• Two routines you can build, and how to reach them from anywhere. A morning site check and a GitHub watcher, each created from a single prompt, then dispatched from your phone with reports landing in Slack or a custom Telegram bot.

• How to trust a job you were asleep for. Guardrails on permissions, keys, and connectors, a validation harness where tests and hooks check the agent’s work, and reports worth reading.

§

What routines actually are

Everything Claude does normally starts with you!

You type, it works, and when you leave the desk, the work stops…

A schedule breaks this link... The agent stops being an extension of your hands and becomes a worker in your absence, since the job now runs whether you are there/not. And that changes what you use it for, because the tasks worth scheduling are the ones you kept forgetting/postponing.

Routines let Claude work on a schedule you define.

They have two different modes:

• Local

• Cloud

(We’ll cover both in the next sections.)

Let’s first create a routine manually…

Manual

To create a routine, open your Claude app, click on “Routines” and “New routine”.

You will see two options:

Local

Click on Local.

You can choose the model, set permissions & schedule when you want the routine to run.

Cloud

Now let’s check the cloud version.

This version lets you customize a lot more.

First, you can select a trigger: Schedule, Git Event, or API.

Next, click Notifications to choose how the report is delivered to you.

The options are push notifications in the app, email, or Slack.

You can also enable “Auto-fix” for pull requests.

This way Claude watches CI and review comments and pushes fixes automatically.

Create Routine by prompting

Let’s create one routine by prompting.

Here is a site I run… I change it often, so pages break sometimes, and I check them by hand.

Follow along with any site you run, since the prompt only needs a URL.

Let’s hand this task off to Claude Code to save time.

Here is the prompt:

Create a routine that runs every morning at 8.
It opens my AI Labs and AI Academy pages in Chrome,
checks for broken links, layout problems, and console errors.

If everything is fine, report one line saying clean.
Otherwise, list only the real issues to Slack.

After pasting this prompt, Claude asks for my approval.

The first approval is to list the scheduled tasks.

Next, it asks a couple of questions…

After my answers, it started working on it…

Then it asked the final question before creating this routine.

I allowed & it created the routine.

When I went back to the routine tab, I saw mine.

Testing The Routine

I clicked on the routine and “Run now”.

And I saw this notification:

To see more details, you need to click on History. (I clicked it twice by mistake.)

And Claude took control of my laptop & started testing…

When I check Claude app, I see Claude took screenshots & analyzed the pages.

A couple of minutes later, I got this Slack message:

“Clean”, simple yet effective.

Here is a more detailed report:

§

§

Routine vs scheduled

Read more

• Share this letter & I’ll send you some rewards for the referrals.

You ask an AI assistant a question…

…it answers in seconds!

But what actually happened between your question & the answer?

Behind that simple interaction sits a stack of models, tokens, context, retrieval, tools, evaluations, and infrastructure.

These 64 concepts1 will help you understand how modern AI systems actually work, from the neural networks underneath them to agents & applications built on top.

Curious to know how many were new to you:

1. Large language models,

2. Generative AI,

3. Machine learning,

4. Deep learning,

5. Neural networks,

6. Retrieval augmented generation,

7. AI agents & agentic workflows,

8. Prompt engineering,

9. Transformers and self-attention,

10. Embeddings,

11. Tool & function calling,

12. Model evaluation and evaluation-driven development,

13. Context engineering,

14. Multimodal AI,

15. Fine tuning and instruction tuning,

16. Tokenization,

17. Reasoning & chain-of-thought.

( …and much more in Parts 2, 3 & 4!)

For each, I’ll share:

• What it means and how it works,

• A real-world analogy where useful,

• A common mistake to avoid,

• Why it matters.

Let’s dive in!

§

Make your agents realtime (Partner)

Agents are killing your apps with compounding latency across workflows.

Your customers want their agent workflows to be efficient, so you’re losing deals if you don’t make them event-driven. Webhooks solve this issue, but implementing them is harder than it looks.

If you homebrewed your webhooks and are getting pinged at 4 in the morning for failed deliveries, this is for you.

Svix handles everything: retries, idempotency, security, and compliance. Qualified startups get $12,000 in free credits.

Get started for free

(Thanks to Svix for partnering on this newsletter.)

§

1. Large Language Models

A large language model (LLM) is an AI model trained on vast amounts of text to learn language patterns.

You type “Why is the sky blue?” into a chatbot & hit Enter.

The model processes your question as small pieces of text called tokens. A token could be a word, part of a word, or punctuation. A typical text-generating LLM selects the next token based on your question and the text it has produced so far.

Then it repeats. Token by token, the answer appears…

During training, the model adjusted internal numerical values called weights. Those weights capture patterns useful for language tasks, from sentence structure to relationships between ideas.

Your prompt gives the model context for its response.

Analogy

Think of autocomplete on your phone:

You type “See you,” and the keyboard suggests “soon.” An LLM takes this idea much further, using the available conversation to produce paragraphs, explanations, or code.

Predicting a plausible continuation still leaves room for a wrong answer.

Common mistake

Trusting an answer simply because it sounds confident.

Suppose you ask for code to resize an image. The model could suggest a function with a sensible name that the library doesn’t provide. Check the documentation & run the code before you rely on it.

Why it matters

You don’t need a separate model for every language task.

Different instructions let the same LLM summarize a document, explain an error, or draft a reply.

That flexibility makes LLMs useful across many software products.

2. Generative AI

Generative AI produces content such as text, images, audio & video from patterns learned during training.

Ask an image generator for “a red bicycle outside a bakery.”

Your description guides the model toward a result. Some image models begin with random noise and refine it step by step into an image. A text model uses a different process, typically selecting tokens in sequence.

Both produce content from learned patterns.

Generative AI is the broader category… Text-generating LLMs are one example; image and audio generators are others.

Analogy

Think of a cook who has learned from hundreds of recipes:

You ask for a spicy vegetable dish. The cook draws on familiar ingredients and techniques to prepare something that fits your request.

You still taste the dish before you serve it.

Common mistake

Assuming the generated content is ready to use.

An image could contain unreadable labels. A summary could leave out a key decision. Review the result against your original request and check factual details against the source.

Why it matters

Generative AI helps you get from a blank page to a first draft.

A few notes become an email, a description becomes an illustration, or a specification becomes starter code.

The next step is to check whether the draft does the job.

3. Machine Learning

Machine Learning (ML) is a branch of AI in which systems learn from data to perform tasks without explicit rules for every situation.

Think about the spam folder in your inbox:

You could write a rule to block every email containing “free money.” But a spammer could change the wording; also, a legitimate email could contain the same phrase. A model learns from examples & considers several clues together.

Those clues help it assess a new message.

For a spam filter, training examples typically include the correct label: spam or legitimate. The training algorithm adjusts the model to reduce classification errors.

This approach is called supervised learning.

Other approaches discover patterns in unlabeled data or learn through rewards after actions.

Analogy

Think about how you learn to choose ripe fruit:

Someone shows you a few good examples & a few bad ones. Over time, you connect color, firmness, and smell with ripeness.

You use those clues on your next trip to the market.

Common mistake

Testing the model with the same examples it learned from.

That’s like giving a student the practice questions again and calling it a final exam. Reserve separate examples to check whether the model handles unfamiliar data.

Why it matters

ML helps when a task involves more patterns than you could reasonably capture in hand-written rules. It powers uses such as spam filters, fraud detection, and product recommendations.

The goal is useful predictions on new data.

§

§

4. Deep Learning

Deep learning (DL2) is a type of ML that uses multiple layers of a neural network to learn complex patterns.

Show a model a photograph of a bicycle.

The model receives numbers representing the image’s pixels. Early layers often detect simple features, such as edges & textures. Later layers combine these features into more complex shapes & object parts.

Together, the layers help the model recognize the bicycle.

The word “deep” refers to these multiple layers. During training, the model adjusts them to improve its predictions.

Analogy

Think of a sketch as someone draws it:

First, you see a few lines and circles. Add a frame, handlebars, and pedals, and the shapes start to make sense together.

A deep network also combines simpler features into more complex patterns.

Common mistake

Assuming every problem needs deep learning.

For a small spreadsheet of house sizes and prices, a simpler model could do the job well. Test a basic approach first, then compare whether a deep network earns its extra cost and complexity.

Why it matters

Images, speech & language contain patterns which are difficult to describe with fixed rules. Deep learning helps models learn useful features from these inputs.

It underpins tools such as speech recognition & image classification.

5. Neural Networks

A neural network3 is a model made of connected mathematical units that transform inputs into an output.

Let’s look inside the layers we just discussed:

Each unit receives numbers, combines them, and passes a result onward. Connections have adjustable values called weights, which control how much each input contributes. Layers also apply transformations, so the network learns more complex relationships.

The final layer produces the model’s output.

During training, an error measure tells the system how far its predictions are from the target. The training algorithm uses this feedback to adjust weights and other values across the network.

Tiny adjustments, repeated across many examples, change its predictions.

Analogy

Think of the controls on a sound mixing desk:

One control increases the voice; another reduces the drums. Changing the controls changes the final sound.

Weights play a similar role in how a network combines its inputs.

Common mistake

Taking the word “neural” too literally.

The name draws inspiration from the brain, but these units perform mathematical operations. A network’s ability to recognize a face or produce a sentence doesn’t establish human-like thought.

Why it matters

Neural networks provide the structure behind deep learning.

Understanding weights also explains what training changes: numerical values a model uses to turn an input into a prediction.

6. Retrieval-Augmented Generation

RAG is an approach that retrieves relevant information & supplies it to a model to help generate an answer.

Ask a company assistant, “How long do I have to return this order?”

The application searches the company’s documents for the relevant return policy. It places matching passages alongside your question. The model uses those passages to write its response.

The answer has information from the company’s documents.

Updating those documents gives the application new material to retrieve. It doesn’t require a new training run for every policy change.

Analogy

Think of an open-book exam:

You find the relevant pages before you answer. Your result depends on both finding the right material & interpreting it correctly.

Common mistake

Assuming retrieval guarantees a correct answer.

The search could return an outdated policy and/or miss an exception. So check the retrieved passages & final response. When the evidence is missing, the assistant should say so.

Why it matters

RAG helps assistants answer questions about private or frequently updated information. Also it gives readers source material to check against the answer.

7. AI Agents and Agentic Workflows

An AI agent uses a model to choose actions toward a goal and respond to the results. Agentic workflows organize model and tool steps into a process.

Ask an assistant to investigate a failed software test.

An agent would inspect the error, open a file, make an edit, and rerun the tests. The result helps it decide what to do next. A fixed workflow follows a prewritten sequence, such as “summarize the ticket, classify it, then route it.”

The difference is how much control the model has over the next step.

The term “agentic workflow” covers a range of designs. Some follow mostly fixed steps; others give the model more freedom to choose.

Analogy

Think of a technician with a checklist:

For a routine inspection, the checklist determines the order. For an unfamiliar fault, the technician chooses the next check based on what the previous one revealed.

Common mistake

Giving an agent tools without clear limits.

A failed action could trigger repeated retries/unwanted changes. So set permissions, spending limits, and stopping conditions. Plus, require approval before consequential actions, such as a payment/deletion.

Why it matters

Agents help with tasks whose steps depend on discoveries along the way.

Fixed workflows work when the sequence is known and predictable execution matters.

§

§

8. Prompt Engineering

Read more

Download the “Agentic AI” e-book for free

AWS isn’t winning the AI race with the smartest model. It’s winning by owning everything that surrounds the model. Six layers. Sixty-plus services.

But how do you actually put all those pieces together?

AWS asked 15+ data and AI leaders from companies including PayPal, Siemens, Morgan Stanley, and Sanofi how they are approaching agentic AI in the enterprise.

The result is The Agentic Enterprise: a 16-chapter e-book covering practical topics such as:

• Building, monitoring, and governing reliable AI agents

• Improving agent performance and scalability

• Implementing guardrails throughout the AI lifecycle

• Moving agentic systems toward production

If you’re working on AI agents beyond the prototype stage, it’s worth a read.

Download the e-book for free here »

(Thanks to AWS for partnering on this newsletter.)

Imagine you’re building Shazam1.

A user hums a 10-second song clip…Your system has 100 million songs.

Your job is simple: find the most similar songs in less than 50 milliseconds.

So how’d you do it?

Here’s the obvious solution: compare the user’s song with each song in the database.

Query song
   ↓
Song 1
Song 2
Song 3
...
Song 100,000,000

Yet this does NOT work at scale.

Even if each comparison is fast, 100 million comparisons are “expensive”.

When you search documents in a Retrieval-Augmented Generation (RAG2) system, find similar images, or generate recommendations, the system is doing the same thing:

Find the most similar items in a massive collection.

This is where Hierarchical Navigable Small World (HNSW3) comes in…

§

• Share this letter & I’ll send you some rewards for the referrals.

§

I want to introduce Avani Chaskar as the guest author.

She writes technical guides on building, evaluating & scaling deterministic AI systems.

Subscribe to her newsletter to join developers receiving her latest engineering breakdowns.

§

The Idea: Search Only What Matters

The idea behind HNSW is simple:

Instead of comparing against every item, build shortcuts that lead you close to the answer.

Think about finding a coffee shop in a city you do not know.

You would not walk down every street…Instead, you’d:

1. Use highways to reach the right area.

2. Take the main road to get closer.

3. Then use local streets to find the exact place.

§

First Attempt: Build a Graph

Let’s go back to our 100 million songs.

Suppose each song is connected to similar songs:

If Song B is similar to Song E, we connect them…

So instead of checking every song, we can jump from one song to another.

Start somewhere:

• Move to a neighbor that is closer to the query,

• Then repeat.

This is much faster!..

But there is still a problem: a graph with 100 million nodes still requires “many” hops.

i.e., we need a faster way to move…

§

But What Does “Similar” Actually Mean?

Before HNSW navigates the graph, it needs a way to measure how close two vectors4 are.

Here are the most common distance or similarity metrics:

• Cosine similarity: Measures the angle between vectors. A smaller angle means higher similarity. It ignores vector length, making it ideal for text and NLP5.

• Dot product/inner product: Multiplies & sums elements. A larger positive value means higher similarity (it grows with both alignment and length). It’s commonly used in recommendation systems.

• Euclidean distance (L2): Measures straight-line distance. A smaller distance means higher similarity. It captures absolute size differences, making it great for images.

So the right similarity metric depends on the embedding model6 & use case.

For HNSW, the important idea is simple:

The closer a vector is to the query according to the chosen metric, the better the match.

§

Exact Search vs Approximate Search

There are two ways to find the nearest vectors:

• Exact search: Compares the query against every vector & returns the mathematically closest results. That gives us the true nearest neighbors, but at 100 million or a billion vectors, checking everything is “expensive”.

• Approximate Nearest Neighbor (ANN): Instead of examining the entire dataset, it explores the most promising clusters of the index and returns results that are very likely to be the true nearest neighbors.

HNSW is an ANN algorithm.

Yet this creates an important tradeoff:

• Explore more → higher recall7, higher latency.

• Explore less → lower latency, potentially lower recall.

To measure search effectiveness, we use Recall@k.

It measures how many of the true top-k nearest neighbors an approximate search actually retrieves.

HNSW gives us a way to search a tiny fraction of the dataset while still achieving high recall.

§

The Real Trick: Add Layers

This is the key idea behind HNSW:

HNSW builds many layers of graphs8.

The bottom layer contains every song.

Higher layers contain fewer songs. The higher you go, the fewer nodes exist. These upper layers create long-distance shortcuts.

This makes HNSW fast!

§

How HNSW Search Works

Here’s how:

• A user hums a song: the system converts it into a vector.

  • A vector is simply a list of numbers: [0.12, 0.87, 0.34, 0.91, ...].

  • Songs with similar vectors sound similar.

  

2. Now the search begins…

   • Step 1 (Top Layer): Performs a high-level traversal on a sparse graph (A → C → D) to locate the general neighborhood rapidly.

   • Step 2 (Middle Layers): Drops down through denser layers (D → F → G), narrowing the distance to refine the search space.

   • Step 3 (Bottom Layer): Executes a targeted, local search among millions of nodes- exploring only a tiny fraction of the dataset to return Approximate Nearest Neighbors.

§

Why the “Small World” Matters

Think about social networks:

You know someone. That person knows someone else. After a few connections, you could reach almost anybody.

Graphs with this property are called small-world graphs9.

HNSW uses this idea:

Most connections are local…some connections jump far.

Together, they create short paths through huge datasets.

§

Why “Hierarchical”?

Because there are layers…

Each higher layer contains exponentially fewer nodes.

Think of it like this:

• Layer 3: Highways

• Layer 2: Major roads

• Layer 1: Streets

• Layer 0: Every location

Search starts “fast”…then becomes “precise”.

§

How Layers Get Built?

When inserting a new vector into an HNSW index:

• The algorithm assigns the vector a maximum height layer based on a decaying probability distribution10.

• Most vectors reside exclusively on Layer 0.

• A small fraction are promoted to middle layers, and only a tiny minority reach the sparse top layer.

This is similar to a data structure called a skip list11.

Skip lists let you jump over many elements.

HNSW applies the same idea to vectors.

Randomness creates shortcuts → Those shortcuts make navigation efficient.

§

How HNSW Builds the Graph

When a new vector gets inserted, HNSW searches the existing graph to find promising neighbors.

It does NOT simply connect the new vector to the closest M vectors it happens to find. Instead, a neighbor-selection heuristic tries to keep connections useful and diverse, avoiding too many redundant links.

The process looks roughly like this:

This is where efConstruction12 matters:

• A higher efConstruction means the algorithm considers a larger candidate set while building the graph. That usually produces a better-connected graph, but indexing takes longer.

• A lower value makes construction faster, but the resulting graph may provide fewer useful paths during search.

Good graph construction matters because graph quality directly affects both search speed & recall.

§

§

Three Knobs Engineers Tune

In production systems, these three parameters matter most:

1. M

M controls how many connections each node has.

More connections mean:

• Better search quality

• More memory usage

Think of it as building more roads:

• More roads improve navigation.

• But roads are expensive.

2. efConstruction

This controls how carefully the graph gets built.

Higher values mean:

• Better graph quality

• Slower indexing

Lower values mean:

• Faster indexing

• Slightly lower search quality

This matters if you insert data frequently.

3. efSearch

This determines how many candidate nodes the system explores during the search.

Higher values mean:

• Better recall

• Higher latency

Lower values mean:

• Faster queries

• Lower recall

For example:

efSearch = 50
>Recall = 92%
>Latency = 8 ms


efSearch = 200
>Recall = 98%
>Latency = 20 ms

The exact values depend on the dataset...this is an engineering tradeoff.

§

Why HNSW Uses So Much Memory

HNSW stores:

• Vectors,

• Graph connections,

• Metadata.

The graph itself could become large…for millions/billions of vectors, memory usage grows quickly…

So many systems combine HNSW with compression techniques such as:

• Scalar Quantization (SQ8): Compresses float32 vectors down to int8, reducing RAM footprint by ~75% with minimal recall impact.

• Product Quantization13 (PQ): Segments vectors into sub-vectors and quantizes them into cluster centroids, unlocking extreme memory savings for billion-scale indexes.

§

When to Use HNSW

HNSW is a good choice when:

• Search speed matters: get results in milliseconds at large scale.

• High recall matters: for the search to capture most of the truly similar results.

• The dataset fits in memory: larger datasets demand more RAM, since HNSW keeps its graph in memory

• Data updates are moderate: HNSW works best when vectors get added occasionally, and not constantly.

Best use cases:

• RAG systems,

• Image search,

• Audio matching,

• Semantic search,

• Recommendations.

But HNSW is NOT ideal when:

• Memory is very limited: Graph connections add significant RAM overhead.

• Data changes constantly: Frequent updates make the graph expensive to maintain.

• Exact nearest neighbors are required: HNSW is approximate, so it occasionally misses the mathematically closest match.

No algorithm is free → HNSW trades memory for speed & search quality.

§

The HNSW Mental Model

HNSW is built around one simple idea:

Don’t make the search faster…Make the search smaller.

Instead of looking at every vector, build a structure that tells you where to look.

The idea is simple…The scale it unlocks is not.

Millions of vectors. Billions of vectors. Milliseconds.

That’s the power of HNSW.

§

I’d like to thank Avani Chaskar for writing this newsletter.

She writes technical guides on building, evaluating & scaling deterministic AI systems.

Subscribe to her newsletter for more such quick reads on AI Engineering topics.

§

If you’re serious about AI engineering, you really can’t miss out on the footnotes:

Get Instant Access Now

Want to reach 250K+ tech professionals at scale? 📰

If your company wants to reach 250K+ tech professionals, advertise with me.

Thank you for supporting this newsletter.

You are now 250,001+ readers strong, very close to 251k. Let’s try to get 251k readers by 17 October. Consider sharing this letter with your friends & get rewards.

Y’all are the best.

Share

Read more

• Share this letter & I’ll send you some rewards for the referrals.

No time for small talk today:

The following is the second of a premium 2-part newsletter... if you’re serious about API design in 2026, then this newsletter is for YOU.

On with part 2 of the newsletter:

===

Some of these API design concepts are foundational, and some are advanced. ALL of them are super useful to software engineers building scalable APIs.

Curious to know how many were new to you:

18. JSON Web Token,

19. Throttling,

20. Endpoint,

21. Request-Response,

22. Headers,

23. OpenAPI,

24. Timeouts,

25. Microservices,

26. WebSockets,

27. CORS,

28. HTTPS and TLS,

29. Filtering & Sorting,

30. Retries with Backoff,

31. Load Balancing,

32. Circuit Breaker,

33. Error Handling.

For each, I’ll share:

• What it is & how it works (in plain English),

• A real-world analogy (if I found one),

• Tradeoffs,

• Why it matters.

Let’s go!

§

[Webinar] How to stop babysitting your agents (Partner)

Agents can generate code. Getting it right for your system, team conventions, and past decisions is the hard part. You end up wasting time and tokens in correction loops.

More MCPs give agents access to information but not understanding. The teams pulling ahead use a context layer to give agents exactly what they need.

Join live on Oct 7 (FREE) to see:

• Where teams get stuck on the AI maturity curve

• How a context layer solves for quality, efficiency, and cost

• Live demo: the same coding task with and without a context layer

Register now

(Thanks to Unblocked for partnering on this newsletter.)

§

18. JSON Web Token

JSON Web Token (JWT) is a signed token that contains claims about the caller.

Three parts, separated by dots.

• Header names the algorithm.

• Payload holds the claims: this is user 42, they expire at noon, and have the admin role.

• Signature proves nobody altered the first two parts. The server verifies signature with a key it already holds, reads the claims, and answers the request without touching a session store.

The token carries its own proof, so verification costs no database call.

Analogy

A festival wristband.

It says which zones you may enter and which day it dies. Security reads it in a glance without radioing the ticket office. If you tamper with it, the seal breaks, so the guard knows.

Tradeoff

You cannot easily revoke a token.

Fire an employee at nine, and their token still opens doors until it expires. The payload is signed, not encrypted, so anyone holding the token can read every claim inside it.

So keep expiry short, keep secrets out of payload, and use a refresh token to renew.

Why it matters

Stateless authentication across microservices, and the access token most OAuth providers hand back.

Its structure is fixed by a public specification, which is why every language has a library for it. Pair it with Authentication, which is the job it exists to do.

19. Throttling

Throttling slows down a caller instead of turning them away.

Rate limiting & throttling get used interchangeably, and they do different things.

A rate limiter counts requests and rejects the ones over the cap with a 429. A throttle accepts the request and delays it. It queues the call, releases it when capacity frees up, or answers with a Retry-After that asks the client to come back shortly. The work still happens, just later and more slowly.

Rejection protects the server…Throttling protects the server & keeps the caller alive.

Analogy

A metered on-ramp to a motorway.

The light drips cars onto the carriageway one at a time. Nobody is sent home. The queue on the ramp is annoying, and it's why the motorway itself keeps moving at seventy.

Tradeoff

A queued request holds a connection, and enough of them exhaust the very resource you were protecting.

Plus, delay hides the problem, so a service that has been struggling for an hour looks merely sluggish. So cap the queue & alert on the delay rather than the errors.

Why it matters

Expensive operations, shared infrastructure, and any burst you would rather absorb than reject.

Envoy, Kong, and most message queues implement it. Pair it with rate limiting, and be explicit in your documentation about which one you are doing.

20. Endpoint

An endpoint is a uniform resource locator (URL) that points to one resource on your server.

It’s the address you aim a request at.

• /users is the collection.

• /users/42 is one user.

• /users/42/orders is that user’s orders.

The path names the resource & the method names the action; together, they define one operation. Get the path wrong, and you receive a 404 rather than the data you meant to ask for.

Read a good endpoint aloud, and it explains itself.

Analogy

A street address.

The number and the road identify one door out of a million. Post a letter to the wrong number, and it arrives somewhere real, which is worse than it not arriving at all.

Tradeoff

Endpoint names drift as the API grows.

One person writes /users, the next writes /getUser, and six months later your API reads like it was designed by a committee that never met.

Use plural nouns for collections and let the verb live in the method.

Why it matters

The first thing anybody learns about your API, and the thing they will curse if you got it wrong.

Every REST framework routes on them. Pair it with OpenAPI, which turns your endpoint list into documentation nobody has to maintain by hand.

Share

21. Request-Response

The request-response cycle is the basic way clients & servers communicate. The client sends a request, and server returns a response.

Every call follows the same simple script.

A request names what you want and where: a method, a path, a few headers, a body when there is one to send. The server reads it, does the work, and answers with a status code and usually a body of its own. The exchange is synchronous, so the client holds the line open and waits for that single reply before it does anything else.

Send, wait, read the reply. That is the entire cycle.

Analogy

Posting a letter and waiting by the mailbox.

You write the address on the envelope, the details inside, and a return address on the back. Then you stand there. Nothing else happens until the reply lands.

Tradeoff

The client blocks and a slow server holds that connection, and work that takes two minutes leaves a user staring at a spinner. Nothing comes back until everything comes back.

Accept the request, return a 202, and finish the work in the background.

Why it matters

The foundation under REST, GraphQL, and almost every API you will build.

Its rules are set by the HTTP specification. Pair it with Webhooks and WebSockets, the two answers for when waiting is the wrong shape.

Want life to feel a little easier?

Ben Meer’s new book, How to Be Good at Life, is packed with practical systems for work, health, money, relationships, home, and more.

Get the book →

22. Headers

Headers are key-value pairs that carry information about a request/response, separate from the body.

They describe the message rather than contain it.

• Content-Type says what format the body is in.

• Authorization carries the credential.

• Cache-Control tells every cache along the path how long to keep a copy.

• Accept states what the client can understand.

• User-Agent names the software that called.

Servers, proxies, and browsers all read them before touching the body.

Metadata belongs on the outside.

Analogy

The outside of an envelope.

Stamps, postcodes, fragile stickers, return address. Every courier along the route reads the outside and acts on it. None of them open the letter.

Tradeoff

Headers are easy to forget, which is why so many bugs end in a Content-Type nobody set.

Sizes are capped, casing rules surprise people, and a header stripped by a proxy fails silently rather than loudly. So log the headers you rely on, and set Content-Type explicitly every time.

Why it matters

Authentication, caching, versioning, compression, and cross-origin rules all ride on them.

Any browser’s network tab shows you every one. Pair it with Caching, which does nothing at all unless the headers say so.

23. OpenAPI

OpenAPI is a standard file that describes every endpoint, parameter, and response of your API in one place.

You write it in YAML/JSON, and machines read it.

From that one file, you generate interactive documentation, client libraries in eight languages, a mock server for the front end to build against, and a test suite that catches the day your response shape quietly changed. The specification becomes the contract, and the contract becomes the source of truth.

Write it once & stop maintaining documentation by hand.

Analogy

An architect’s blueprint.

The plumber, electrician, and joiner all work from the same drawing without ever meeting. Change the drawing, and everyone sees the change. Change the wall without changing the drawing, and somebody drills through a pipe.

Tradeoff

A hand-written spec drifts from the code within weeks, and a spec that lies is worse than no spec at all. The file grows long, and reviewing a two-thousand-line YAML diff is nobody’s favorite afternoon.

Generate the spec from your code, or test your code against the spec.

Why it matters

Public APIs, partner integrations, and any backend whose consumers sit in a different room.

Swagger UI, Redoc, and Stoplight render it. Pair it with API Versioning, since each version deserves its own contract.

24. Timeouts

A timeout is the longest a caller will wait for a response before giving up.

Without one, a caller waits forever.

Set a connect timeout for reaching the server and a read timeout for hearing back. When a downstream service hangs, your request fails in two seconds instead of hanging alongside it. This matters because every waiting request holds a thread, a connection, and a slot in your pool. Enough of them & your healthy service falls over next to the sick one.

A fast failure beats a slow hang.

Analogy

Hanging up after thirty seconds of ringing.

Nobody is home. Listening to the ringtone for an hour won't make them answer, and it stops you from calling anybody else.

Tradeoff

Set one too tight, and you kill healthy requests that were simply slow; then you retry them, and you have doubled the load on a service that was only having a bad minute.

Measure your ninety-ninth percentile, then set the timeout above it.

Why it matters

Every network call you make is the first line of defense against a cascading failure.

Every HTTP client has a default, and the default is usually wrong. Pair it with Circuit Breaker, which you reach for when timeouts stop being occasional.

25. Microservices

Microservices split one application into small, independent services that talk to each other over APIs.

Each service owns one job and its own data.

The orders service owns the orders table. The payments service owns payments. Neither reaches into the other’s database. They call each other’s APIs, deploy on their own schedule, and scale independently, so a search feature under load gets more servers without anybody touching checkout.

Independence is the whole point, and the network is the whole cost.

Analogy

A food court instead of one enormous kitchen.

Each unit runs its own stoves, staff, and menu. The pizza place closes for a refit, and everyone else keeps serving. Ordering a meal from three units means three queues.

Tradeoff

Every function call you used to make in memory is now a network call that can time out, arrive twice, or never arrive.

A bug spans four services and six log files. Keeping data consistent across services is a hard problem with no free answer. So start with a monolith, and split it when the pain is real, not theoretical.

Why it matters

Large systems, large organizations, and any codebase where two changes cannot ship without waiting for each other.

The patterns that make it survivable deserve an article of their own. Pair it with API Gateway and gRPC, which keep the pieces reachable and fast.

§

§

26. WebSockets

Read more

• Share this letter & I’ll send you some rewards for the referrals.

Some of these API design concepts are foundational, and some are advanced. ALL of them are super useful to software engineers building scalable APIs.

Curious to know how many were new to you:

1. REST,

2. GraphQL,

3. Authentication,

4. Authorization,

5. Rate Limiting,

6. Caching,

7. Pagination,

8. Status Codes,

9. Webhooks,

10. Idempotency,

11. API Gateway,

12. API Versioning,

13. JSON,

14. OAuth 2.0,

15. HTTP Methods,

16. gRPC,

17. API Keys.

(…and much more in Part 2!)

For each, I’ll share:

• What it is & how it works (in plain English),

• A real-world analogy (if I found one),

• Tradeoffs,

• Why it matters.

Let’s go!

§

Beyond the Coding Agent: Building at Enterprise Scale (Partner)

Coding agents made individual tasks faster. They didn’t remove the coordination, validation, and handoffs that actually determine how fast enterprise software ships.

Join Sid, CTO and Co-founder of Blitzy, for a live session on what changes when a system can hold context across an entire codebase, not just a single file or function. Sid will walk through a real project in Blitzy, covering everything from reverse engineering and tech spec generation to the Agent Action Plan, code generation, and PR Review. You’ll see what happens at each stage and where human review gates the process.

What you’ll see:

• Reverse engineering and tech spec generation on a real codebase

• Where human review sits in an autonomous build: sign-off at the plan stage, approval at the merge stage

• Enterprise modernization results

We’ll close with an open Q&A, so bring your questions on agent orchestration and governance at scale.

Save Your Seat

(Thanks to Blitzy for partnering on this newsletter.)

§

1. REST

Representational State Transfer (REST) is a way to build APIs that organizes data as resources and uses standard HTTP methods to work with them.

Every resource gets its own address.

Each resource has its own URL. For example, a user is at /users/42 and an order is at /orders/17.

Use:

• GET to read it,

• POST to create it,

• PUT to replace it,

• DELETE to remove it.

The HTTP method defines the action, and the URL identifies the resource. Both the client and server follow the same contract.

REST also stays stateless, so any server can answer any request.

Analogy

A library with a fixed shelf number for every book:

You hand the librarian a shelf number and say borrow, return, or replace. They never ask what you meant.

The number identifies the book; the verb identifies the action; they leave no room for guessing.

Tradeoff

REST wastes requests & bandwidth.

One screen may need a user, their orders, and their payment method, which costs three round trips. Ask for a user, and the server sends every field, including the twenty you ignore. Add a field for one client, and you risk breaking another.

So keep your resources small and reach for GraphQL when clients need to shape their own responses.

Why it matters

Public APIs, mobile and web clients, anything cached at the edge, and any service where a new developer should guess the right endpoint on the first try.

Spring, Express, Django REST Framework, and FastAPI all lean on it. Pair it with OpenAPI so the contract stays readable, and with API Versioning so today’s shape survives tomorrow’s change.

2. GraphQL

GraphQL is a query language for APIs that lets clients request exactly the data they need in a single response.

There is one endpoint instead of dozens.

Client sends a query that names the fields it needs. The server walks the query, fetches each field, and returns a response shaped like the request. Queries read data, mutations change it, and subscriptions stream updates as they happen. Ask for three things at once, and you get one round trip.

The client decides the shape, so the server stops guessing.

Analogy

A sandwich counter where you name every ingredient:

A fixed combo hands you the pickles you scrape off and skips the extra cheese you wanted. Naming each ingredient gets you the sandwich you meant to order, in one pass down the counter.

Tradeoff

GraphQL costs you more.

Caching gets harder because every query is a POST to the same URL. Plus one careless nested query could walk your database for a minute. You carry a schema, a resolver for every field, and a plan for the N+1 problem.

So set a depth limit and a query cost budget on day one.

Why it matters

Mobile clients on slow networks, screens that stitch together several resources, and front ends that change faster than the backend.

Apollo, Relay, Hasura, and graphql-js are common choices. Pair it with Rate Limiting, because a single expensive query can cost more than a thousand REST calls.

3. Authentication

Authentication proves “who” is calling your API.

The caller presents a credential & server checks it.

That credential could be a password, an API key, or a signed token. The server verifies it against something it trusts, then attaches an identity to the request. Everything downstream reads that identity.

A caller who fails the check gets a 401 and never reaches the business logic.

Analogy

Showing your passport at the border desk.

The officer compares the photo to your face & the name to a list. They are not deciding where you may travel inside the country. They only confirm you are the person on the document.

Tradeoff

Each check costs a lookup/signature verification.

Secrets leak through logs, screenshots, and public repositories. Sessions expire at the worst moment for the user.

So rotate credentials on a schedule & keep them out of your URLs.

Why it matters

Every API that touches private data, money, or a user account.

Auth0, Okta, Keycloak, and Firebase Auth handle the hard parts. Pair it with Authorization, because proving who you are says nothing about what you may do.

Share

4. Authorization

Authorization decides what a proven caller is allowed to do.

Authentication runs first, then authorization runs.

The server knows the identity & now checks the rules:

• Does this user own this order?

• Does this token carry the write scope?

• Does this role permit a delete?

A caller who fails the check gets a 403 i.e., the server knows exactly who you are and still says no.

Authorization answers a second question: what may you touch?

Analogy

A hotel keycard.

The card proves nothing about your name. It opens your room and the gym on the third floor. It stays dead against the manager’s office and the room next door, no matter how many times you tap it.

Tradeoff

Rules sprawl across the codebase, and one missing check becomes a breach.

Test the negative cases, because a passing test that proves a user can read their own order says nothing about whether they can read somebody else’s.

So check ownership at the data layer & at the route.

Why it matters

Multi-tenant systems, admin panels, and any endpoint where one user’s record sits next to another’s.

Role-based access control and attribute-based access control are the two common models. Pair it with OAuth 2.0, whose scopes carry authorization decisions across service boundaries.

5. Rate Limiting

Rate limiting caps how many requests one caller can send in a time window.

The server counts & past the cap, refuses them.

A common limit is a hundred requests per minute per API key. Cross it, and the server returns 429 Too Many Requests along with a Retry-After header. Response headers usually report the ceiling and how much of it you have left, so a well-behaved client slows itself down before it gets refused.

One noisy caller stops starving everyone else.

Analogy

A nightclub with a counter at the door.

The room holds four hundred people. The doorman counts everyone in & everyone out. When the room is full, the next person waits outside, and the people already dancing keep enjoying the music.

Tradeoff