<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:itunes="http://www.itunes.com/dtds/podcast-1.0.dtd" xmlns:googleplay="http://www.google.com/schemas/play-podcasts/1.0"><channel><title><![CDATA[The System Design Newsletter]]></title><description><![CDATA[Download my best playbooks on newsletter signup for free]]></description><link>https://newsletter.systemdesign.one</link><image><url>https://substackcdn.com/image/fetch/$s_!W5r-!,w_256,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa1c8067a-95bb-416b-9114-e0b9fb8821d4_256x256.png</url><title>The System Design Newsletter</title><link>https://newsletter.systemdesign.one</link></image><generator>Substack</generator><lastBuildDate>Sun, 22 Nov 2026 12:28:36 GMT</lastBuildDate><atom:link href="https://newsletter.systemdesign.one/feed" rel="self" type="application/rss+xml"/><copyright><![CDATA[Neo Kim]]></copyright><language><![CDATA[en]]></language><webMaster><![CDATA[systemdesignone@substack.com]]></webMaster><itunes:owner><itunes:email><![CDATA[systemdesignone@substack.com]]></itunes:email><itunes:name><![CDATA[Neo Kim]]></itunes:name></itunes:owner><itunes:author><![CDATA[Neo Kim]]></itunes:author><googleplay:owner><![CDATA[systemdesignone@substack.com]]></googleplay:owner><googleplay:email><![CDATA[systemdesignone@substack.com]]></googleplay:email><googleplay:author><![CDATA[Neo Kim]]></googleplay:author><itunes:block><![CDATA[Yes]]></itunes:block><item><title><![CDATA[Top 13 Microservices Interview Questions Answered]]></title><description><![CDATA[#205: Ace Software Engineering Job Interviews]]></description><link>https://newsletter.systemdesign.one/p/microservices-interview-questions</link><guid isPermaLink="false">https://newsletter.systemdesign.one/p/microservices-interview-questions</guid><dc:creator><![CDATA[Neo Kim]]></dc:creator><pubDate>Fri, 20 Nov 2026 12:02:22 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/a8f1499b-8203-4b34-a17d-6625d08b2bae_1280x720.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p></p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption"><span data-color="#3a7bf9" style="color: rgb(58, 123, 249);">Get my system design playbook for FREE on newsletter signup:</span></p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><ul><li><p><em><a href="https://newsletter.systemdesign.one/p/microservices-interview-questions/?action=share">Share this letter</a> &amp; I&#8217;ll send you some rewards for the referrals.</em></p></li></ul><div><hr></div><p>Last year, I met a young software engineer who walked into an interview, confident he knew the fundamentals.</p><p>But he struggled to answer some simple questions about microservices.</p><p>His story is why I wrote this letter.</p><div><hr></div><div class="callout-block" data-callout="true"><h3><a href="https://miro.pxf.io/n4dEzR">If you want to build AI-powered apps (in 2027), try this shortcut:</a></h3><div class="native-video-embed" data-component-name="VideoPlaceholder" data-attrs="{&quot;mediaUploadId&quot;:&quot;fba0d162-96ab-428e-976e-50789be26a60&quot;,&quot;duration&quot;:null}"></div><ul><li><p>Design your system architecture in Miro</p></li><li><p>Document API, database &amp; caching decisions</p></li><li><p>Connect Claude Code through Miro&#8217;s MCP server</p></li><li><p>Retrieve the architecture context from the Miro board</p></li><li><p>Generate an implementation plan based on that context</p></li><li><p>Bring the plan back to Miro for team review</p></li></ul><p>I tried this workflow with a URL shortener architecture &amp; it saved me hours of work.</p><p>If you&#8217;re building AI-powered apps, <strong><a href="https://miro.pxf.io/n4dEzR">Miro&#8217;s MCP Server</a></strong> is worth exploring.</p><p>It lets your coding agent work with existing architecture decisions, so you can spend less time repeating context and more time building.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://miro.pxf.io/n4dEzR&quot;,&quot;text&quot;:&quot;Try Miro's MCP Server Today&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://miro.pxf.io/n4dEzR"><span>Try Miro's MCP Server Today</span></a></p><p><em>(Thanks to <a href="https://miro.pxf.io/n4dEzR">Miro</a> for partnering on this newsletter.)</em></p></div><div><hr></div><p>Here are the top 13 must-know interview questions with answers:</p><h2><strong>1. Microservices</strong></h2><blockquote><p>&#8220;What problems do microservices solve &amp; what new problems do they create?&#8221;</p></blockquote><p>Microservices split one application into small services, each running in its own process and deployed independently.</p><p>An online shop might run separate services for catalog, cart, payments, and shipping. Each has its own code, data, and release schedule. Services talk over the network through application programming interfaces (<strong>APIs</strong>).</p><p>A small group of engineers owns each service from design to production.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!U5pc!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f32ced5-e6e5-41f6-9a2e-2782bbef51c7_1536x1024.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!U5pc!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f32ced5-e6e5-41f6-9a2e-2782bbef51c7_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!U5pc!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f32ced5-e6e5-41f6-9a2e-2782bbef51c7_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!U5pc!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f32ced5-e6e5-41f6-9a2e-2782bbef51c7_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!U5pc!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f32ced5-e6e5-41f6-9a2e-2782bbef51c7_1536x1024.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!U5pc!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f32ced5-e6e5-41f6-9a2e-2782bbef51c7_1536x1024.png" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/4f32ced5-e6e5-41f6-9a2e-2782bbef51c7_1536x1024.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!U5pc!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f32ced5-e6e5-41f6-9a2e-2782bbef51c7_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!U5pc!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f32ced5-e6e5-41f6-9a2e-2782bbef51c7_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!U5pc!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f32ced5-e6e5-41f6-9a2e-2782bbef51c7_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!U5pc!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f32ced5-e6e5-41f6-9a2e-2782bbef51c7_1536x1024.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><strong>Tips for a strong answer</strong></h3><ul><li><p>They solve coordination problems. Each group ships its own service without waiting for a shared release.</p></li><li><p>Also they let you scale each service independently. A busy search service gets more instances while the rest stay small.</p></li><li><p>Yet they create network problems. Every call can be slow, fail, or arrive twice.</p></li><li><p>Plus, they create data problems. No single database transaction spans services, so you need sagas or eventual consistency.</p></li><li><p>And they need tooling before the first service goes live: service discovery, tracing, and automated deployment.</p></li></ul><h3><strong>Likely follow-up</strong></h3><blockquote><p>&#8220;How do you decide where to draw the line between two services?&#8221;</p></blockquote><p>Split along business capabilities, such as payments/shipping, not technical layers.</p><p>Each service should own its data and change for one business reason. If two services always deploy &#8220;together or call each other&#8221; on every request, they belong in one service.</p><h3><strong>Common mistake</strong></h3><p>Listing only the benefits.</p><p>This question tests whether you know the costs. &#8220;Microservices are more scalable&#8221; with no mention of network failures or split data sounds like you&#8217;ve only read about them.</p><p>Strong candidates also say when not to use them, such as a small product with a handful of engineers.</p><h2><strong>2. API gateway</strong></h2><blockquote><p>&#8220;What&#8217;s the difference between an API gateway &amp; a load balancer?&#8221;</p></blockquote><p>An API gateway is a single entry point that receives every client request and routes it to the right service.</p><p>The gateway handles the work every request needs. It checks the token, applies rate limits, and validates the request. Then it finds the target service and forwards the call.</p><p>It also merges several service responses into one reply for the client.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!VvCP!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff0677ec0-6bae-49ee-83a1-dc3e852287bf_1536x1024.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!VvCP!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff0677ec0-6bae-49ee-83a1-dc3e852287bf_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!VvCP!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff0677ec0-6bae-49ee-83a1-dc3e852287bf_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!VvCP!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff0677ec0-6bae-49ee-83a1-dc3e852287bf_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!VvCP!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff0677ec0-6bae-49ee-83a1-dc3e852287bf_1536x1024.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!VvCP!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff0677ec0-6bae-49ee-83a1-dc3e852287bf_1536x1024.png" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/f0677ec0-6bae-49ee-83a1-dc3e852287bf_1536x1024.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!VvCP!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff0677ec0-6bae-49ee-83a1-dc3e852287bf_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!VvCP!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff0677ec0-6bae-49ee-83a1-dc3e852287bf_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!VvCP!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff0677ec0-6bae-49ee-83a1-dc3e852287bf_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!VvCP!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff0677ec0-6bae-49ee-83a1-dc3e852287bf_1536x1024.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><strong>Tips for a strong answer</strong></h3><ul><li><p>A load balancer spreads traffic across copies of one service.</p></li><li><p>While a gateway reads paths, headers, and tokens, then decides which service gets the request.</p></li><li><p>They work together. A load balancer sits in front of several gateway instances.</p></li></ul><h3><strong>Likely follow-up</strong></h3><blockquote><p>&#8220;Your gateway is becoming a bottleneck. What do you do?&#8221;</p></blockquote><p>Keep the gateway stateless and run more instances behind the load balancer.</p><p>Cache token checks. Move heavy work like response merging into &#8220;backend for frontend&#8221;. If it&#8217;s still too much, split the gateway by domain.</p><h3><strong>Common mistake</strong></h3><p>Putting business rules in the gateway, such as discount logic.</p><p>The gateway then changes with every feature and needs every group&#8217;s sign-off, which recreates the enterprise service bus problem.</p><h2><strong>3. Service discovery</strong></h2><blockquote><p>&#8220;Compare client-side and server-side service discovery.&#8221;</p></blockquote><p>Service discovery is how a caller finds a healthy instance of the service it wants to reach.</p><p>In <em>client-side</em> discovery, the caller queries the registry itself. It gets a list of instances and picks one.</p><p>In <em>server-side</em> discovery, the caller sends the request to a router/gateway. The router then queries the registry and forwards the call.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!mA4t!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62f5f528-f6f1-4874-9210-57c06c518f77_1536x1024.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!mA4t!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62f5f528-f6f1-4874-9210-57c06c518f77_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!mA4t!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62f5f528-f6f1-4874-9210-57c06c518f77_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!mA4t!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62f5f528-f6f1-4874-9210-57c06c518f77_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!mA4t!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62f5f528-f6f1-4874-9210-57c06c518f77_1536x1024.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!mA4t!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62f5f528-f6f1-4874-9210-57c06c518f77_1536x1024.png" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/62f5f528-f6f1-4874-9210-57c06c518f77_1536x1024.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!mA4t!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62f5f528-f6f1-4874-9210-57c06c518f77_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!mA4t!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62f5f528-f6f1-4874-9210-57c06c518f77_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!mA4t!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62f5f528-f6f1-4874-9210-57c06c518f77_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!mA4t!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62f5f528-f6f1-4874-9210-57c06c518f77_1536x1024.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><strong>Tips for a strong answer</strong></h3><ul><li><p>Client-side: one less network hop &amp; callers can pick smarter routes.</p></li><li><p>But client-side needs a discovery library in every language you use.</p></li><li><p>Server-side: callers stay simple as the router does the lookup.</p></li><li><p>Yet the router adds an extra hop and must keep its address cache in sync.</p></li></ul><h3><strong>Likely follow-up</strong></h3><blockquote><p>&#8220;Which would you use on Kubernetes?&#8221;</p></blockquote><p>Server-side, since it&#8217;s built in. </p><p>Each Service gets a stable DNS name, and the platform routes to healthy pods. A client-side library adds little there. If you need smarter routing, a service mesh<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-1" href="#footnote-1" target="_self">1</a> adds it without code changes.</p><h3><strong>Common mistake</strong></h3><p>Mixing up the registry and discovery.</p><p>The registry stores where instances live&#8230; Discovery is how a caller uses it.</p><p>Treating them as one thing makes the answer vague.</p><div class="captioned-button-wrap" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/p/microservices-interview-questions?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;}" data-component-name="CaptionedButtonToDOM"><div class="preamble"><p class="cta-caption"><em><span data-color="#3a7bf9" style="color: rgb(58, 123, 249);">Like what you read? Share it with your friends.</span></em></p></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/p/microservices-interview-questions?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://newsletter.systemdesign.one/p/microservices-interview-questions?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p></div><h2><strong>4. Database per service</strong></h2><blockquote><p>&#8220;Order service needs customer details stored in customer database. How does it get them?&#8221;</p></blockquote><p>Database per service means each service owns its data store, and other services reach data only through the owner&#8217;s API.</p><p>Order service owns the orders database. If shipping needs order details, it calls the order API or listens for order events. Each service can also pick the store that suits it best.</p><p>Search might use Elasticsearch while payments stays on PostgreSQL.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!J9p3!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F522a8c07-cef2-44b1-9e07-eaf1abf6dae9_1536x1024.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!J9p3!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F522a8c07-cef2-44b1-9e07-eaf1abf6dae9_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!J9p3!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F522a8c07-cef2-44b1-9e07-eaf1abf6dae9_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!J9p3!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F522a8c07-cef2-44b1-9e07-eaf1abf6dae9_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!J9p3!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F522a8c07-cef2-44b1-9e07-eaf1abf6dae9_1536x1024.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!J9p3!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F522a8c07-cef2-44b1-9e07-eaf1abf6dae9_1536x1024.png" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/522a8c07-cef2-44b1-9e07-eaf1abf6dae9_1536x1024.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!J9p3!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F522a8c07-cef2-44b1-9e07-eaf1abf6dae9_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!J9p3!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F522a8c07-cef2-44b1-9e07-eaf1abf6dae9_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!J9p3!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F522a8c07-cef2-44b1-9e07-eaf1abf6dae9_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!J9p3!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F522a8c07-cef2-44b1-9e07-eaf1abf6dae9_1536x1024.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><strong>Tips for a strong answer</strong></h3><ul><li><p>Never read another service&#8217;s tables directly!</p></li><li><p>Instead, call the customer API when you need fresh data.</p></li><li><p>Or keep a local read copy, updated from customer events for speed &amp; resilience.</p></li><li><p>Tradeoff: API call is fresh but adds a runtime dependency, while a local copy is fast but slightly stale.</p></li><li><p>For reports across services, feed a separate read store from events.</p></li></ul><h3><strong>Likely follow-up</strong></h3><blockquote><p>&#8220;How do you make sure the event goes out whenever the data changes?&#8221;</p></blockquote><p>Use the transactional outbox pattern<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-2" href="#footnote-2" target="_self">2</a>:</p><p>Write the change and the event into the same local transaction, with the event in an outbox table. A separate relay reads the outbox and publishes each event.</p><p>This avoids a commit that succeeds while the publish fails.</p><h3><strong>Common mistake</strong></h3><p>Saying services could share one database as long as each uses its own tables. A shared database still <em>couples</em> schema changes, load, and outages across services.</p><h2><strong>5. Saga</strong></h2><blockquote><p>&#8220;A customer places an order. You need to reserve stock, charge the card, and book delivery. Yet each step lives on a different service. How do you keep data correct?&#8221;</p></blockquote><p>A saga splits one long transaction into a chain of local transactions, each with an undo step called a &#8220;compensating transaction".</p><p>An order saga might reserve stock, charge the card, and book delivery. Each step commits in its own service right away. If delivery booking fails, the saga runs compensations in reverse.</p><p>It refunds the card &amp; releases the stock.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!v6XR!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0b71e67d-0d6d-4e5b-a63b-c4e0a78b11c5_1536x1024.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!v6XR!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0b71e67d-0d6d-4e5b-a63b-c4e0a78b11c5_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!v6XR!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0b71e67d-0d6d-4e5b-a63b-c4e0a78b11c5_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!v6XR!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0b71e67d-0d6d-4e5b-a63b-c4e0a78b11c5_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!v6XR!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0b71e67d-0d6d-4e5b-a63b-c4e0a78b11c5_1536x1024.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!v6XR!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0b71e67d-0d6d-4e5b-a63b-c4e0a78b11c5_1536x1024.png" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/0b71e67d-0d6d-4e5b-a63b-c4e0a78b11c5_1536x1024.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!v6XR!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0b71e67d-0d6d-4e5b-a63b-c4e0a78b11c5_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!v6XR!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0b71e67d-0d6d-4e5b-a63b-c4e0a78b11c5_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!v6XR!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0b71e67d-0d6d-4e5b-a63b-c4e0a78b11c5_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!v6XR!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0b71e67d-0d6d-4e5b-a63b-c4e0a78b11c5_1536x1024.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><strong>Tips for a strong answer</strong></h3><ul><li><p>Explain why one database transaction cannot span three services.</p></li><li><p>Explain why two-phase commit<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-3" href="#footnote-3" target="_self">3</a> is a poor fit: <em>long locks across services hurt availability.</em></p></li><li><p>Walk through the saga: <em>each step commits locally, each has a compensating step.</em></p></li><li><p>Name both styles of Saga. In &#8220;choreography&#8221;, services react to each other&#8217;s events. In &#8220;orchestration&#8221;, one orchestrator tells each service what to do next.</p></li></ul><h3><strong>Likely follow-up</strong></h3><blockquote><p>&#8220;What happens if the refund itself fails?&#8221;</p></blockquote><p>Retry the compensation until it succeeds; every compensation must be idempotent<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-4" href="#footnote-4" target="_self">4</a>. Plus, store the saga&#8217;s state, so a crash can resume from the last step.</p><p>If retries keep failing, raise an alert for manual handling.</p><h3><strong>Common mistake</strong></h3><p>Saying a saga keeps data consistent at every moment.</p><p>Other requests can see half-finished states between steps. Strong candidates say how they&#8217;d handle this, such as marking the order &#8220;pending&#8221; until the saga completes.</p><h2><strong>6. Circuit breaker</strong></h2><blockquote><p>&#8220;Your order service calls a payment service. Payment starts timing out. What happens to the order service, and how do you protect it?&#8221;</p></blockquote><p>A circuit breaker stops calls to a failing service once errors cross a threshold, then lets a few test calls through later.</p><p>The breaker starts closed and calls flow normally. When failures pass a set rate, it trips open. Every call now fails fast without touching the broken service. After a cooldown, it moves to half-open and allows a few test calls.</p><p>Success closes it again, but failure opens it again.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!DsIx!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fba629b22-6396-4702-b9c0-f85dd8e9bb06_1536x1024.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!DsIx!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fba629b22-6396-4702-b9c0-f85dd8e9bb06_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!DsIx!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fba629b22-6396-4702-b9c0-f85dd8e9bb06_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!DsIx!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fba629b22-6396-4702-b9c0-f85dd8e9bb06_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!DsIx!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fba629b22-6396-4702-b9c0-f85dd8e9bb06_1536x1024.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!DsIx!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fba629b22-6396-4702-b9c0-f85dd8e9bb06_1536x1024.png" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/ba629b22-6396-4702-b9c0-f85dd8e9bb06_1536x1024.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!DsIx!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fba629b22-6396-4702-b9c0-f85dd8e9bb06_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!DsIx!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fba629b22-6396-4702-b9c0-f85dd8e9bb06_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!DsIx!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fba629b22-6396-4702-b9c0-f85dd8e9bb06_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!DsIx!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fba629b22-6396-4702-b9c0-f85dd8e9bb06_1536x1024.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><strong>Tips for a strong answer</strong></h3><ul><li><p>Threads in the order service wait on slow payment calls, pile up, and the order service fails as well.</p></li><li><p>A circuit breaker trips once the failure rate crosses a threshold, so calls fail fast.</p></li><li><p>Name the three states: <em>closed, open, and half-open</em>.</p></li><li><p>Pair the breaker with a timeout and a fallback. On its own, it only turns slow errors into fast ones.</p></li></ul><h3><strong>Likely follow-up</strong></h3><blockquote><p>&#8220;How do you pick the threshold?&#8221;</p></blockquote><p>Start from real traffic data.</p><p>Use a failure rate over a sliding window, not a raw error count, so quiet periods don&#8217;t trip it. Set a minimum number of calls before it can trip. Then tune it from what you see in production.</p><h3><strong>Common mistake</strong></h3><p>Describing the breaker as a retry mechanism. Retries keep calling the broken service. A breaker stops calling it.</p><h2><strong>7. Monolith</strong></h2><blockquote><p>&#8220;You&#8217;re starting a new product with 5 engineers. Would you build a monolith or microservices?&#8221;</p></blockquote><p>A monolith is one application where every module ships together as a single deployable unit.</p><p>Orders, payments, and user accounts live in one codebase. They share one process and usually one database. Modules talk through plain function calls, so each call is fast and fails loudly.</p><p>You build it once, test it once, and deploy it once.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!zQw0!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37232d4e-8559-4546-85c7-397ceb94f5bf_1536x1024.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!zQw0!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37232d4e-8559-4546-85c7-397ceb94f5bf_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!zQw0!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37232d4e-8559-4546-85c7-397ceb94f5bf_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!zQw0!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37232d4e-8559-4546-85c7-397ceb94f5bf_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!zQw0!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37232d4e-8559-4546-85c7-397ceb94f5bf_1536x1024.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!zQw0!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37232d4e-8559-4546-85c7-397ceb94f5bf_1536x1024.png" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/37232d4e-8559-4546-85c7-397ceb94f5bf_1536x1024.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!zQw0!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37232d4e-8559-4546-85c7-397ceb94f5bf_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!zQw0!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37232d4e-8559-4546-85c7-397ceb94f5bf_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!zQw0!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37232d4e-8559-4546-85c7-397ceb94f5bf_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!zQw0!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37232d4e-8559-4546-85c7-397ceb94f5bf_1536x1024.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><strong>Tips for a strong answer</strong></h3><ul><li><p>Pick a monolith first. Five engineers ship faster with one codebase, one pipeline, and one deploy.</p></li><li><p>The product will change shape often, so service boundaries drawn now will probably be wrong.</p></li><li><p>Keep clean module boundaries inside the monolith. Each module owns its own tables, and other modules call it through a clear interface.</p></li><li><p>Name the signals for splitting later: builds get slow, engineers block each other&#8217;s releases, or one module needs to scale on its own.</p></li></ul><h3><strong>Likely follow-up</strong></h3><blockquote><p>&#8220;Your monolith has grown to 60 engineers. What do you do?&#8221;</p></blockquote><p>Find the module with the most pain first, such as the one that changes most often or needs to scale most.</p><p>Extract it into its own service behind the same interface. Route traffic to the new service bit by bit, and keep the old code path until the new one proves stable.</p><p>Repeat one module at a time, never all at once.</p><h3><strong>Common mistake</strong></h3><p>Calling the monolith outdated by default.</p><p>Interviewers want to hear you match the architecture to the size of the group and the product. A rewrite into many services on day one is the answer they&#8217;re testing you against.</p><p>Shopify still runs a very large Rails monolith, so &#8220;monoliths don&#8217;t scale&#8221; sounds like you&#8217;re simply following trends.</p><div class="captioned-button-wrap" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/p/microservices-interview-questions?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;}" data-component-name="CaptionedButtonToDOM"><div class="preamble"><p class="cta-caption"><em><span data-color="#3a7bf9" style="color: rgb(58, 123, 249);">This newsletter has been growing super fast, thanks to your shares.</span></em></p></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/p/microservices-interview-questions?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://newsletter.systemdesign.one/p/microservices-interview-questions?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p></div><h2><strong>8. Sidecar and service mesh</strong></h2><blockquote><p>&#8220;When would you add a service mesh &amp; what does it cost you?&#8221;</p></blockquote><p>A service mesh is a layer of proxies, one beside each service, which handles all traffic between services.</p><p>Each proxy, called a sidecar, runs in the same pod as its service. Every request in and out passes through it. The sidecar handles encryption, retries, timeouts, and metrics. A control plane pushes rules to every sidecar at once.</p><p>You change a retry policy in one place, and the entire fleet follows.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!nyii!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F85050529-2adf-479e-a0b1-3b1555865b39_1536x1024.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!nyii!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F85050529-2adf-479e-a0b1-3b1555865b39_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!nyii!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F85050529-2adf-479e-a0b1-3b1555865b39_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!nyii!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F85050529-2adf-479e-a0b1-3b1555865b39_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!nyii!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F85050529-2adf-479e-a0b1-3b1555865b39_1536x1024.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!nyii!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F85050529-2adf-479e-a0b1-3b1555865b39_1536x1024.png" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/85050529-2adf-479e-a0b1-3b1555865b39_1536x1024.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!nyii!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F85050529-2adf-479e-a0b1-3b1555865b39_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!nyii!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F85050529-2adf-479e-a0b1-3b1555865b39_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!nyii!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F85050529-2adf-479e-a0b1-3b1555865b39_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!nyii!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F85050529-2adf-479e-a0b1-3b1555865b39_1536x1024.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><strong>Tips for a strong answer</strong></h3><ul><li><p>It gives you encryption, retries, timeouts, and traffic metrics without code changes.</p></li><li><p>A service mesh becomes more useful when you have many microservices built with different programming languages.</p></li><li><p>But it adds extra delay on every call, extra memory in every pod, and makes the system complex.</p></li><li><p>Plus, debugging gets harder because a failure could occur in the proxy&#8230;not the service.</p></li></ul><h3><strong>Likely follow-up</strong></h3><blockquote><p>&#8220;How is a service mesh different from an API gateway?&#8221;</p></blockquote><p>A gateway handles traffic coming into the system from clients&#8230;A mesh handles traffic between services inside the system.</p><p>Many large systems run both.</p><h3><strong>Common mistake</strong></h3><p>Adding a mesh to five services because it&#8217;s popular.</p><p>At that size, a library like Resilience4j covers retries and timeouts. Interviewers want to hear you weigh the running costs.</p><h2><strong>9. Observability</strong></h2><blockquote><p>&#8220;Checkout request is slow &amp; it touches ten services. How do you find the cause?&#8221;</p></blockquote><p>Observability is your ability to understand a system&#8217;s inner state from logs, metrics, and traces it produces.</p><p>Logs record single events, like an error with its details. Metrics track numbers over time, like request rate and error rate. Traces follow one request across every service it touches.</p><p>Each service passes a shared trace ID along with the call. Your tracing tool then stitches the pieces into one timeline.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!zAeJ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fde55b45c-35e1-48ec-a56b-bd1a11440902_1536x1024.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!zAeJ!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fde55b45c-35e1-48ec-a56b-bd1a11440902_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!zAeJ!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fde55b45c-35e1-48ec-a56b-bd1a11440902_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!zAeJ!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fde55b45c-35e1-48ec-a56b-bd1a11440902_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!zAeJ!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fde55b45c-35e1-48ec-a56b-bd1a11440902_1536x1024.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!zAeJ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fde55b45c-35e1-48ec-a56b-bd1a11440902_1536x1024.png" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/de55b45c-35e1-48ec-a56b-bd1a11440902_1536x1024.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!zAeJ!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fde55b45c-35e1-48ec-a56b-bd1a11440902_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!zAeJ!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fde55b45c-35e1-48ec-a56b-bd1a11440902_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!zAeJ!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fde55b45c-35e1-48ec-a56b-bd1a11440902_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!zAeJ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fde55b45c-35e1-48ec-a56b-bd1a11440902_1536x1024.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><strong>Tips for a strong answer</strong></h3><ul><li><p>Start with a distributed trace. It shows how much time each service spent on the request.</p></li><li><p>Use metrics to spot which service&#8217;s latency or error rate went up.</p></li><li><p>Search the logs by trace ID to see the exact error.</p></li><li><p>Alert on what users feel, such as slow checkouts, not on every server metric.</p></li></ul><h3><strong>Likely follow-up</strong></h3><blockquote><p>&#8220;Tracing every request costs too much. What do you do?&#8221;</p></blockquote><p>Sample. Keep a small share of normal traces and keep every trace with an error or a slow response. Set retention limits on logs as well.</p><h3><strong>Common mistake</strong></h3><p>Answering &#8220;I&#8217;d check the logs.&#8221;</p><p>Logs from ten services can&#8217;t be joined without a shared trace ID. Name tracing first.</p><h2><strong>10. Service-oriented architecture</strong></h2><blockquote><p>&#8220;How is service-oriented architecture different from microservices?&#8221;</p></blockquote><p>Service-oriented architecture (<strong>SOA</strong>) builds an application from large, shared services connected through a central messaging layer.</p><p>In SOA, one service covers an entire business function, like billing. Several applications reuse the same service. An enterprise service bus (<strong>ESB</strong>) sits in the middle. It routes messages, converts formats, and applies rules between services.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!ViIh!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6cc2263a-d141-4882-bb0b-e17ccb6994ce_1536x1024.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!ViIh!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6cc2263a-d141-4882-bb0b-e17ccb6994ce_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!ViIh!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6cc2263a-d141-4882-bb0b-e17ccb6994ce_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!ViIh!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6cc2263a-d141-4882-bb0b-e17ccb6994ce_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!ViIh!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6cc2263a-d141-4882-bb0b-e17ccb6994ce_1536x1024.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!ViIh!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6cc2263a-d141-4882-bb0b-e17ccb6994ce_1536x1024.png" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/6cc2263a-d141-4882-bb0b-e17ccb6994ce_1536x1024.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!ViIh!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6cc2263a-d141-4882-bb0b-e17ccb6994ce_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!ViIh!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6cc2263a-d141-4882-bb0b-e17ccb6994ce_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!ViIh!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6cc2263a-d141-4882-bb0b-e17ccb6994ce_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!ViIh!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6cc2263a-d141-4882-bb0b-e17ccb6994ce_1536x1024.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><strong>Tips for a strong answer</strong></h3><ul><li><p>Size: SOA services are large and shared across applications. Microservices are small and built around one capability.</p></li><li><p>Logic: SOA puts routing and rules in the bus. Microservices keep the pipes simple and put logic inside each service.</p></li><li><p>Data: SOA services share a database. Each microservice owns its own data.</p></li><li><p>Releases: Shared SOA service needs sign-off from every consumer. A microservice ships on its own.</p></li></ul><h3><strong>Likely follow-up</strong></h3><blockquote><p>&#8220;Isn&#8217;t an API gateway just an enterprise service bus again?&#8221;</p></blockquote><p>Only if you let it grow into one.</p><p>A gateway should route, check tokens, and apply rate limits. Once it starts holding business rules or joining data from many services, it becomes the bottleneck the bus was.</p><p>Watch for gateway changes in every feature release.</p><h3><strong>Common mistake</strong></h3><p>Saying &#8220;microservices are just SOA done right&#8221; without naming a single difference. Interviewers want concrete points. The bus and data ownership are the two clearest ones.</p><p style="text-align: center;"><em><span data-color="#3a7bf9" style="color: rgb(58, 123, 249);">Start a discussion in the comments &amp; connect with fellow readers:</span></em></p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/p/microservices-interview-questions/comments&quot;,&quot;text&quot;:&quot;Leave a comment&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://newsletter.systemdesign.one/p/microservices-interview-questions/comments"><span>Leave a comment</span></a></p><h2><strong>11. Bounded context</strong></h2><blockquote><p>&#8220;How would you split an online shop&#8217;s monolith into services?&#8221;</p></blockquote><p>A <em>bounded context</em> is a boundary inside which one domain model and its terms have one exact meaning.</p><p>Take the word &#8220;customer.&#8221; Sales sees a customer as a lead with a deal value. Support sees someone with open tickets. Shipping only needs a name and an address.</p><p>A bounded context lets each area keep its own small model instead of sharing one giant one.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!X5k0!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5ba5120-1e24-4a2a-af77-6471321622b4_1536x1024.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!X5k0!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5ba5120-1e24-4a2a-af77-6471321622b4_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!X5k0!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5ba5120-1e24-4a2a-af77-6471321622b4_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!X5k0!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5ba5120-1e24-4a2a-af77-6471321622b4_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!X5k0!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5ba5120-1e24-4a2a-af77-6471321622b4_1536x1024.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!X5k0!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5ba5120-1e24-4a2a-af77-6471321622b4_1536x1024.png" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/a5ba5120-1e24-4a2a-af77-6471321622b4_1536x1024.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!X5k0!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5ba5120-1e24-4a2a-af77-6471321622b4_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!X5k0!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5ba5120-1e24-4a2a-af77-6471321622b4_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!X5k0!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5ba5120-1e24-4a2a-af77-6471321622b4_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!X5k0!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5ba5120-1e24-4a2a-af77-6471321622b4_1536x1024.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><strong>Tips for a strong answer</strong></h3><ul><li><p>Start from the business domain, not from database tables or code layers.</p></li><li><p>Find boundaries where words change meaning, such as &#8220;customer&#8221; or &#8220;product.&#8221;</p></li><li><p>Each context owns its model &amp; data, and becomes a candidate service.</p></li><li><p>Map how contexts depend on each other, and which events pass between them.</p></li><li><p>Start with a few large contexts, and split further only when you have a reason.</p></li></ul><h3><strong>Likely follow-up</strong></h3><blockquote><p>&#8220;Two contexts both need customer data. Where does it live?&#8221;</p></blockquote><p>One context owns the source of truth, such as customer service. It publishes an event whenever a customer changes. Other contexts keep their own copy with only the fields they need, updated from those events.</p><h3><strong>Common mistake</strong></h3><p>Splitting by technical layer, with one service for the user interface, one for logic, and one for data. Every feature then touches all three, so nothing ships alone. One service per database table fails the same way.</p><h2><strong>12. Rate limiting</strong></h2><blockquote><p>&#8220;Design rate limiting for a public API running on ten gateway instances.&#8221;</p></blockquote><p>Rate limiting caps the number of requests a caller make in a set time window.</p><p>The gateway counts requests per user, API key, or IP address. Once a caller passes the limit, extra requests get an <code>HTTP 429</code> response. A common method is the token bucket.</p><p>Each caller has a bucket that refills at a steady rate &amp; each request spends one token.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!oVW1!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe773a7bc-7e0b-4b14-9177-f7accf197152_1536x1024.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!oVW1!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe773a7bc-7e0b-4b14-9177-f7accf197152_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!oVW1!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe773a7bc-7e0b-4b14-9177-f7accf197152_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!oVW1!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe773a7bc-7e0b-4b14-9177-f7accf197152_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!oVW1!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe773a7bc-7e0b-4b14-9177-f7accf197152_1536x1024.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!oVW1!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe773a7bc-7e0b-4b14-9177-f7accf197152_1536x1024.png" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/e773a7bc-7e0b-4b14-9177-f7accf197152_1536x1024.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!oVW1!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe773a7bc-7e0b-4b14-9177-f7accf197152_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!oVW1!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe773a7bc-7e0b-4b14-9177-f7accf197152_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!oVW1!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe773a7bc-7e0b-4b14-9177-f7accf197152_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!oVW1!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe773a7bc-7e0b-4b14-9177-f7accf197152_1536x1024.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><strong>Tips for a strong answer</strong></h3><ul><li><p>Limit per user or API key, not just per IP address<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-5" href="#footnote-5" target="_self">5</a>.</p></li><li><p>Pick an algorithm: token bucket allows short bursts; sliding window gives smoother limits.</p></li><li><p>Also keep counters in a shared store like Redis, so all ten instances see the same count.</p></li><li><p>Return a <code>429</code> with a <code>Retry-After</code> header on limit.</p></li><li><p>Plus, set different limits per plan and per endpoint.</p></li></ul><h3><strong>Likely follow-up</strong></h3><blockquote><p>&#8220;Redis goes down. Do you block all traffic or let it through?&#8221;</p></blockquote><p>Usually let it through, using rough limits inside each gateway instance.</p><p>An outage in the limiter then doesn&#8217;t become an outage of the entire API. For sensitive endpoints like login, block instead.</p><h3><strong>Common mistake</strong></h3><p>Counting inside each instance only. With ten instances, the real limit becomes ten times higher. Fixed time windows have a similar flaw since a caller can send double the limit on a window edge.</p><h2><strong>13. REST</strong></h2><blockquote><p>&#8220;Design a REST API for orders.&#8221;</p></blockquote><p>Representational state transfer (<strong>REST</strong>) is an API style where clients act on resources using standard HTTP methods.</p><p>A uniform resource locator (<strong>URL</strong>) like <code>/orders/42</code> names one order. GET reads it, POST creates one, PUT replaces it, and DELETE removes it. Responses normally come back as JSON.</p><p>Each request carries everything the server needs, so servers keep no session state between calls.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!dGmg!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4fb066e5-36a5-44f8-a4a1-ee2eef61f13c_1536x1024.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!dGmg!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4fb066e5-36a5-44f8-a4a1-ee2eef61f13c_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!dGmg!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4fb066e5-36a5-44f8-a4a1-ee2eef61f13c_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!dGmg!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4fb066e5-36a5-44f8-a4a1-ee2eef61f13c_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!dGmg!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4fb066e5-36a5-44f8-a4a1-ee2eef61f13c_1536x1024.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!dGmg!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4fb066e5-36a5-44f8-a4a1-ee2eef61f13c_1536x1024.png" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/4fb066e5-36a5-44f8-a4a1-ee2eef61f13c_1536x1024.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!dGmg!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4fb066e5-36a5-44f8-a4a1-ee2eef61f13c_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!dGmg!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4fb066e5-36a5-44f8-a4a1-ee2eef61f13c_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!dGmg!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4fb066e5-36a5-44f8-a4a1-ee2eef61f13c_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!dGmg!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4fb066e5-36a5-44f8-a4a1-ee2eef61f13c_1536x1024.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><strong>Tips for a strong answer</strong></h3><ul><li><p>Use nouns for resources, such as <code>/orders</code> and <code>/orders/42/items</code>.</p></li><li><p>Use the right methods: GET, POST, PUT, PATCH, and DELETE.</p></li><li><p>Return clear status codes, such as <code>201</code> for created, <code>404</code> for not found, and <code>409</code> for a conflict.</p></li><li><p>Paginate lists, and plan how you&#8217;ll version the API.</p></li></ul><h3><strong>Likely follow-up</strong></h3><blockquote><p>&#8220;A client retries POST <code>/orders</code> after a timeout. How do you avoid creating two orders?&#8221;</p></blockquote><p>Ask the client to send an idempotency key with each POST. The server stores the key with the result. A repeat request with the same key returns the stored result instead of creating a new order.</p><h3><strong>Common mistake</strong></h3><p>Putting verbs in URLs, like <code>/createOrder</code>, or returning <code>200</code> for every error with the real problem hidden in the body.</p><div><hr></div><p>What happens next is up to you. Because the next time an interviewer asks you to design a microservices system, the answer won&#8217;t come from this letter. It&#8217;ll come from what you&#8217;ve learned and practiced.</p><p>&#8212; Neo</p><div><hr></div><p style="text-align: center;"><em><span data-color="#3a7bf9" style="color: rgb(58, 123, 249);">Contribute to the conversation by sharing your insights &amp; experiences:</span></em></p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/p/microservices-interview-questions/comments&quot;,&quot;text&quot;:&quot;Leave a comment&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://newsletter.systemdesign.one/p/microservices-interview-questions/comments"><span>Leave a comment</span></a></p><div><hr></div><div class="captioned-image-container"><figure><a class="image-link image2" target="_blank" href="https://www.linkedin.com/in/nk-systemdesign-one/" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!68aQ!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe4eab3a0-fda8-4df8-b651-0f976b223e9a_320x320.png 424w, https://substackcdn.com/image/fetch/$s_!68aQ!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe4eab3a0-fda8-4df8-b651-0f976b223e9a_320x320.png 848w, https://substackcdn.com/image/fetch/$s_!68aQ!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe4eab3a0-fda8-4df8-b651-0f976b223e9a_320x320.png 1272w, https://substackcdn.com/image/fetch/$s_!68aQ!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe4eab3a0-fda8-4df8-b651-0f976b223e9a_320x320.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!68aQ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe4eab3a0-fda8-4df8-b651-0f976b223e9a_320x320.png" width="160" height="160" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/e4eab3a0-fda8-4df8-b651-0f976b223e9a_320x320.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:320,&quot;width&quot;:320,&quot;resizeWidth&quot;:160,&quot;bytes&quot;:121414,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:&quot;https://www.linkedin.com/in/nk-systemdesign-one/&quot;,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://newsletter.systemdesign.one/i/136161496?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe4eab3a0-fda8-4df8-b651-0f976b223e9a_320x320.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!68aQ!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe4eab3a0-fda8-4df8-b651-0f976b223e9a_320x320.png 424w, https://substackcdn.com/image/fetch/$s_!68aQ!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe4eab3a0-fda8-4df8-b651-0f976b223e9a_320x320.png 848w, https://substackcdn.com/image/fetch/$s_!68aQ!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe4eab3a0-fda8-4df8-b651-0f976b223e9a_320x320.png 1272w, https://substackcdn.com/image/fetch/$s_!68aQ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe4eab3a0-fda8-4df8-b651-0f976b223e9a_320x320.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div></div></div></a><figcaption class="image-caption"><strong>Find me elsewhere: <a href="https://www.linkedin.com/in/nk-systemdesign-one/">LinkedIn</a> | <a href="https://x.com/intent/follow?screen_name=systemdesignone">Twitter</a> | <a href="https://www.instagram.com/systemdesignone/">Instagram</a></strong></figcaption></figure></div><div><hr></div><p><strong>Want to reach 250K+ tech professionals at scale? </strong>&#128240;</p><p>If your company wants to reach 250K+ tech professionals, <a href="https://newsletter.systemdesign.one/p/sponsorship">advertise with me</a>.</p><div><hr></div><p>Thank you for supporting this newsletter.</p><p>You are now 250,001+ readers strong, very close to 251k. Let&#8217;s try to get 251k readers by 27 November. Consider sharing this letter with your friends and get rewards.</p><p>Y&#8217;all are the best.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!6oWl!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e739087-a910-4643-be36-997b6dd5b4af_800x500.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!6oWl!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e739087-a910-4643-be36-997b6dd5b4af_800x500.png 424w, https://substackcdn.com/image/fetch/$s_!6oWl!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e739087-a910-4643-be36-997b6dd5b4af_800x500.png 848w, https://substackcdn.com/image/fetch/$s_!6oWl!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e739087-a910-4643-be36-997b6dd5b4af_800x500.png 1272w, https://substackcdn.com/image/fetch/$s_!6oWl!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e739087-a910-4643-be36-997b6dd5b4af_800x500.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!6oWl!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e739087-a910-4643-be36-997b6dd5b4af_800x500.png" width="590" height="368.75" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/2e739087-a910-4643-be36-997b6dd5b4af_800x500.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:500,&quot;width&quot;:800,&quot;resizeWidth&quot;:590,&quot;bytes&quot;:87878,&quot;alt&quot;:&quot;system design newsletter&quot;,&quot;title&quot;:&quot;system design newsletter&quot;,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://newsletter.systemdesign.one/i/163380418?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e739087-a910-4643-be36-997b6dd5b4af_800x500.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="system design newsletter" title="system design newsletter" srcset="https://substackcdn.com/image/fetch/$s_!6oWl!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e739087-a910-4643-be36-997b6dd5b4af_800x500.png 424w, https://substackcdn.com/image/fetch/$s_!6oWl!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e739087-a910-4643-be36-997b6dd5b4af_800x500.png 848w, https://substackcdn.com/image/fetch/$s_!6oWl!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e739087-a910-4643-be36-997b6dd5b4af_800x500.png 1272w, https://substackcdn.com/image/fetch/$s_!6oWl!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e739087-a910-4643-be36-997b6dd5b4af_800x500.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/p/microservices-interview-questions?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://newsletter.systemdesign.one/p/microservices-interview-questions?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p><div><hr></div><h3>References</h3><ul><li><p>Martin Fowler and James Lewis, <em>Microservices</em>:</p><p><a href="https://martinfowler.com/articles/microservices.html">https://martinfowler.com/articles/microservices.html</a></p></li><li><p>Martin Fowler, <em>MonolithFirst</em>: <a href="https://martinfowler.com/bliki/MonolithFirst.html">https://martinfowler.com/bliki/MonolithFirst.html</a></p></li><li><p>Martin Fowler, <em>BoundedContext</em>: <a href="https://martinfowler.com/bliki/BoundedContext.html">https://martinfowler.com/bliki/BoundedContext.html</a></p></li><li><p>Martin Fowler, <em>CircuitBreaker</em>: <a href="https://martinfowler.com/bliki/CircuitBreaker.html">https://martinfowler.com/bliki/CircuitBreaker.html</a></p></li><li><p>Melvin E. Conway, <em>How Do Committees Invent?</em> <a href="https://www.melconway.com/Home/Committees_Paper.html">https://www.melconway.com/Home/Committees_Paper.html</a></p></li><li><p>Sam Newman, <em>Backends For Frontends</em>: <a href="https://samnewman.io/patterns/architectural/bff/">https://samnewman.io/patterns/architectural/bff/</a></p></li><li><p>Chris Richardson, <em>Pattern: Database per service</em>: <a href="https://microservices.io/patterns/data/database-per-service.html">https://microservices.io/patterns/data/database-per-service.html</a></p></li><li><p>Netflix, <em>Eureka</em>: <a href="https://github.com/Netflix/eureka">https://github.com/Netflix/eureka</a></p></li><li><p>HashiCorp, <em>Consul documentation</em>: <a href="https://developer.hashicorp.com/consul/docs">https://developer.hashicorp.com/consul/docs</a></p></li><li><p>Kubernetes, <em>Service</em>: <a href="https://kubernetes.io/docs/concepts/services-networking/service/">https://kubernetes.io/docs/concepts/services-networking/service/</a></p></li><li><p>Spring, <em>Spring Cloud Config</em>: <a href="https://docs.spring.io/spring-cloud-config/reference/">https://docs.spring.io/spring-cloud-config/reference/</a></p></li><li><p>Istio, <em>What is a service mesh</em>: <a href="https://istio.io/latest/about/service-mesh/">https://istio.io/latest/about/service-mesh/</a></p></li><li><p>OpenTelemetry, <em>Documentation</em>: <a href="https://opentelemetry.io/docs/">https://opentelemetry.io/docs/</a></p></li><li><p>Benjamin H. Sigelman et al., <em>Dapper, a Large-Scale Distributed Systems Tracing Infrastructure</em>: <a href="https://research.google/pubs/dapper-a-large-scale-distributed-systems-tracing-infrastructure/">https://research.google/pubs/dapper-a-large-scale-distributed-systems-tracing-infrastructure/</a></p></li><li><p>RFC 6585, <em>Additional HTTP Status Codes</em>: <a href="https://www.rfc-editor.org/rfc/rfc6585">https://www.rfc-editor.org/rfc/rfc6585</a></p></li><li><p>Roy Fielding, <em>Architectural Styles and the Design of Network-based Software Architectures</em>: <a href="https://ics.uci.edu/~fielding/pubs/dissertation/top.htm">https://ics.uci.edu/~fielding/pubs/dissertation/top.htm</a></p></li><li><p>gRPC, <em>Introduction to gRPC</em>: <a href="https://grpc.io/docs/what-is-grpc/introduction/">https://grpc.io/docs/what-is-grpc/introduction/</a></p></li><li><p>Amazon Builders&#8217; Library, <em>Timeouts, retries, and backoff with jitter: </em><a href="https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/">https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/</a></p></li><li><p>Netflix, <em>Hystrix</em>: <a href="https://github.com/Netflix/Hystrix">https://github.com/Netflix/Hystrix</a></p></li><li><p>Resilience4j, <em>Bulkhead</em>: <a href="https://resilience4j.readme.io/docs/bulkhead">https://resilience4j.readme.io/docs/bulkhead</a></p></li><li><p>PostgreSQL, <em>PREPARE TRANSACTION</em>: <a href="https://www.postgresql.org/docs/current/sql-prepare-transaction.html">https://www.postgresql.org/docs/current/sql-prepare-transaction.html</a></p></li><li><p>Apache Seata, <em>TCC Mode</em>: <a href="https://seata.apache.org/docs/user/mode/tcc/">https://seata.apache.org/docs/user/mode/tcc/</a></p></li><li><p>Hector Garcia-Molina and Kenneth Salem, <em>Sagas</em>: <a href="https://www.cs.cornell.edu/andru/cs711/2002fa/reading/sagas.pdf">https://www.cs.cornell.edu/andru/cs711/2002fa/reading/sagas.pdf</a></p></li><li><p>Werner Vogels, <em>Eventually Consistent</em>: <a href="https://queue.acm.org/detail.cfm?id=1466448">https://queue.acm.org/detail.cfm?id=1466448</a></p></li></ul><div class="footnote" data-component-name="FootnoteToDOM"><a id="footnote-1" href="#footnote-anchor-1" class="footnote-number" contenteditable="false" target="_self">1</a><div class="footnote-content"><p>A service mesh is an infrastructure layer that manages communication between microservices without requiring developers to implement networking features in every service</p></div></div><div class="footnote" data-component-name="FootnoteToDOM"><a id="footnote-2" href="#footnote-anchor-2" class="footnote-number" contenteditable="false" target="_self">2</a><div class="footnote-content"><p>The transactional outbox pattern saves a database change and its event in the same transaction, so the event can be reliably published later even if system crashes.</p></div></div><div class="footnote" data-component-name="FootnoteToDOM"><a id="footnote-3" href="#footnote-anchor-3" class="footnote-number" contenteditable="false" target="_self">3</a><div class="footnote-content"><p>Two-phase commit (<strong>2PC</strong>) makes sure multiple databases either all commit their changes or all roll them back, using a coordinator that asks each database to prepare before committing.</p></div></div><div class="footnote" data-component-name="FootnoteToDOM"><a id="footnote-4" href="#footnote-anchor-4" class="footnote-number" contenteditable="false" target="_self">4</a><div class="footnote-content"><p>Idempotent = performing the same operation multiple times has the same effect as performing it once, such as retrying a payment without charging the customer twice.</p></div></div><div class="footnote" data-component-name="FootnoteToDOM"><a id="footnote-5" href="#footnote-anchor-5" class="footnote-number" contenteditable="false" target="_self">5</a><div class="footnote-content"><p>Because multiple users can share the same IP address, and one user can make requests from different IP addresses.</p></div></div>]]></content:encoded></item><item><title><![CDATA[How GraphRAG Works - A Deep Dive]]></title><description><![CDATA[#204: Understanding GraphRAG Fundamentals]]></description><link>https://newsletter.systemdesign.one/p/graph-retrieval-augmented-generation</link><guid isPermaLink="false">https://newsletter.systemdesign.one/p/graph-retrieval-augmented-generation</guid><dc:creator><![CDATA[Neo Kim]]></dc:creator><pubDate>Thu, 19 Nov 2026 12:40:39 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/f58cc0cf-4b4f-45db-be51-cdb4561839c1_1280x720.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<div class="callout-block" data-callout="true"><h2><a href="https://solutions.cerbos.dev/guide-to-multitenant-authorization?utm_campaign=system_design_nov_2026&amp;utm_source=newsletter&amp;utm_medium=email&amp;utm_content=&amp;utm_term=">Download the &#8220;Multitenant Authorization&#8221; eBook for free</a></h2><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://solutions.cerbos.dev/guide-to-multitenant-authorization?utm_campaign=system_design_nov_2026&amp;utm_source=newsletter&amp;utm_medium=email&amp;utm_content=&amp;utm_term=" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!AR8X!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd0391342-867b-4ebe-849b-819a1571c3f2_1800x960.png 424w, https://substackcdn.com/image/fetch/$s_!AR8X!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd0391342-867b-4ebe-849b-819a1571c3f2_1800x960.png 848w, https://substackcdn.com/image/fetch/$s_!AR8X!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd0391342-867b-4ebe-849b-819a1571c3f2_1800x960.png 1272w, https://substackcdn.com/image/fetch/$s_!AR8X!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd0391342-867b-4ebe-849b-819a1571c3f2_1800x960.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!AR8X!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd0391342-867b-4ebe-849b-819a1571c3f2_1800x960.png" width="1456" height="777" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/d0391342-867b-4ebe-849b-819a1571c3f2_1800x960.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:777,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:2380720,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:&quot;https://solutions.cerbos.dev/guide-to-multitenant-authorization?utm_campaign=system_design_nov_2026&amp;utm_source=newsletter&amp;utm_medium=email&amp;utm_content=&amp;utm_term=&quot;,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://newsletter.systemdesign.one/i/224578240?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd0391342-867b-4ebe-849b-819a1571c3f2_1800x960.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!AR8X!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd0391342-867b-4ebe-849b-819a1571c3f2_1800x960.png 424w, https://substackcdn.com/image/fetch/$s_!AR8X!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd0391342-867b-4ebe-849b-819a1571c3f2_1800x960.png 848w, https://substackcdn.com/image/fetch/$s_!AR8X!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd0391342-867b-4ebe-849b-819a1571c3f2_1800x960.png 1272w, https://substackcdn.com/image/fetch/$s_!AR8X!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd0391342-867b-4ebe-849b-819a1571c3f2_1800x960.png 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Most SaaS products start with Admin, Editor, and Viewer.</p><p>Then a 30,000-person enterprise signs up with matrix reporting, regional compliance, and its own idea of who sees what. Custom roles pile up per tenant, and deals stall on access rules your model can&#8217;t express.<br><br>This <a href="https://solutions.cerbos.dev/guide-to-multitenant-authorization?utm_campaign=system_design_nov_2026&amp;utm_source=newsletter&amp;utm_medium=email&amp;utm_content=&amp;utm_term=">50+ page guide</a> by Cerbos co-founder Emre Baran shows how to implement authorization that fits each tenant without adding roles for every customer.</p><ul><li><p><strong>Why fixed roles break</strong> at enterprise scale, and what role explosion looks like</p></li><li><p><strong>Letting customers manage their own roles</strong> with delegated admin</p></li><li><p><strong>Keeping each tenant&#8217;s policies</strong> <strong>apart</strong> <strong>from</strong> platform-wide rules and out of your application code</p></li><li><p><strong>Real examples from leading SaaS companies</strong> scaling authorization across thousands of tenants</p></li></ul><p>Useful if you&#8217;re building multitenant SaaS or about to sign one of your first large enterprise customers.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://solutions.cerbos.dev/guide-to-multitenant-authorization?utm_campaign=system_design_nov_2026&amp;utm_source=newsletter&amp;utm_medium=email&amp;utm_content=&amp;utm_term=&quot;,&quot;text&quot;:&quot;Download free eBook now&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://solutions.cerbos.dev/guide-to-multitenant-authorization?utm_campaign=system_design_nov_2026&amp;utm_source=newsletter&amp;utm_medium=email&amp;utm_content=&amp;utm_term="><span>Download free eBook now</span></a></p><p><em>(Thanks to <a href="https://solutions.cerbos.dev/guide-to-multitenant-authorization?utm_campaign=system_design_nov_2026&amp;utm_source=newsletter&amp;utm_medium=email&amp;utm_content=&amp;utm_term=">Cerbos</a> for partnering on this newsletter.)</em></p></div><div><hr></div><p>Picture yourself starting a new software project.</p><p>You search the codebase for a function, class, and/or configuration file. But finding those pieces is only <em>half</em> the job&#8230;</p><p>You need to understand how they connect:</p><blockquote><p><em>Which service calls this function?</em></p><p><em>Which database it depend on?</em></p><p><em>Which business rule affects the result?</em></p></blockquote><p>AI systems face a similar problem.</p><p>Traditional Retrieval-Augmented Generation (<strong>RAG</strong>) retrieves relevant chunks from a large document collection. But what happens when many documents contain parts of the answer, and the true answer depends on their relationships?</p><p>That is where GraphRAG takes a unique approach:</p><p>It connects information instead of seeing it as separate pieces. It links entities, concepts, people, events &amp; documents into a knowledge graph. Now the system searches for relevant information and follows connections between them.</p><p>Onward.</p><p></p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption"><span data-color="#3a7bf9" style="color: rgb(58, 123, 249);">Get my system design playbook for FREE on newsletter signup:</span></p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><ul><li><p><em><a href="https://newsletter.systemdesign.one/p/graph-retrieval-augmented-generation/?action=share">Share this letter</a> &amp; I&#8217;ll send you some rewards for the referrals.</em></p></li></ul><h4 style="text-align: center;"><strong>&#167;</strong></h4><p><em><strong>Here&#8217;s what you'll find in this newsletter:</strong></em></p><ul><li><p><strong>Why finding the right text isn&#8217;t always enough: </strong>where traditional RAG and vector search fall short when an answer depends on relationships between people, documents, concepts, and events.</p></li><li><p><strong>How GraphRAG connects the dots. </strong>How knowledge graphs turn scattered information into connected knowledge, and how GraphRAG uses those connections to retrieve better answers.</p></li><li><p><strong>How GraphRAG finds the right path: </strong>From graph queries and traversal to local, global, multi-hop, path, and community-based retrieval. Learn why different questions need different paths.</p></li><li><p><strong>How you know GraphRAG actually works. </strong>How to assess retrieval for relevance, faithfulness, completeness, speed, and overall quality, and what happens when the graph grows very large.</p></li><li><p><strong>What happens when RAG starts making decisions. </strong>How Agentic RAG moves beyond a fixed retrieval pipeline by choosing retrieval strategies, tools, and sources based on the question.</p></li></ul><p>Onward.</p><h4 style="text-align: center;"><strong>&#167;</strong></h4><h2><strong>RAG</strong></h2><p>RAG is a key method for enterprise apps.</p><p>It gives Large Language Models (<strong>LLMs</strong>) access to information outside their training data. It gathers relevant content from sources such as company documents, knowledge bases &amp; databases. Then it forwards that information to the LLM to generate a response.</p><p>Let&#8217;s break down what RAG is:</p><ul><li><p><strong>Retrieval:</strong> Locate information from an outside source</p></li><li><p><strong>Augmented:</strong> Add information that was retrieved to the model context</p></li><li><p><strong>Generation:</strong> Generate an answer based on provided context using LLM</p></li></ul><p>Think of RAG as an information desk for an LLM.</p><p>The retrieval system finds the information and gives it to the LLM so it can answer the question.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!1M-k!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe681d58a-dafc-40a7-bd39-9882b03e8c04_1774x783.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!1M-k!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe681d58a-dafc-40a7-bd39-9882b03e8c04_1774x783.png 424w, https://substackcdn.com/image/fetch/$s_!1M-k!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe681d58a-dafc-40a7-bd39-9882b03e8c04_1774x783.png 848w, https://substackcdn.com/image/fetch/$s_!1M-k!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe681d58a-dafc-40a7-bd39-9882b03e8c04_1774x783.png 1272w, https://substackcdn.com/image/fetch/$s_!1M-k!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe681d58a-dafc-40a7-bd39-9882b03e8c04_1774x783.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!1M-k!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe681d58a-dafc-40a7-bd39-9882b03e8c04_1774x783.png" width="1456" height="643" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/e681d58a-dafc-40a7-bd39-9882b03e8c04_1774x783.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:643,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!1M-k!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe681d58a-dafc-40a7-bd39-9882b03e8c04_1774x783.png 424w, https://substackcdn.com/image/fetch/$s_!1M-k!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe681d58a-dafc-40a7-bd39-9882b03e8c04_1774x783.png 848w, https://substackcdn.com/image/fetch/$s_!1M-k!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe681d58a-dafc-40a7-bd39-9882b03e8c04_1774x783.png 1272w, https://substackcdn.com/image/fetch/$s_!1M-k!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe681d58a-dafc-40a7-bd39-9882b03e8c04_1774x783.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Let&#8217;s look at the standard RAG pipeline (step by step)&#8230;</p><h3><strong>How Standard RAG Works</strong></h3><p>There are 2 phases in a RAG system:</p><ol><li><p>Indexing stage (onetime/periodic)</p></li><li><p>Query stage (for queries from the user)</p></li></ol><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!uySz!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F13bceee2-0d49-403b-a35e-363bc1eeee58_1430x720.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!uySz!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F13bceee2-0d49-403b-a35e-363bc1eeee58_1430x720.png 424w, https://substackcdn.com/image/fetch/$s_!uySz!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F13bceee2-0d49-403b-a35e-363bc1eeee58_1430x720.png 848w, https://substackcdn.com/image/fetch/$s_!uySz!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F13bceee2-0d49-403b-a35e-363bc1eeee58_1430x720.png 1272w, https://substackcdn.com/image/fetch/$s_!uySz!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F13bceee2-0d49-403b-a35e-363bc1eeee58_1430x720.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!uySz!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F13bceee2-0d49-403b-a35e-363bc1eeee58_1430x720.png" width="1430" height="720" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/13bceee2-0d49-403b-a35e-363bc1eeee58_1430x720.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:720,&quot;width&quot;:1430,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!uySz!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F13bceee2-0d49-403b-a35e-363bc1eeee58_1430x720.png 424w, https://substackcdn.com/image/fetch/$s_!uySz!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F13bceee2-0d49-403b-a35e-363bc1eeee58_1430x720.png 848w, https://substackcdn.com/image/fetch/$s_!uySz!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F13bceee2-0d49-403b-a35e-363bc1eeee58_1430x720.png 1272w, https://substackcdn.com/image/fetch/$s_!uySz!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F13bceee2-0d49-403b-a35e-363bc1eeee58_1430x720.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h4><strong>Phase 1: Indexing Stage</strong></h4><p>Let&#8217;s walk through key steps in a RAG pipeline:</p><p><strong>Step A: </strong>Get the documents you want the system to use as references.</p><p>This could be:</p><ul><li><p>Legal documents, precedents, agreements &amp; decisions</p></li><li><p>Literature &amp; clinical guidelines</p></li><li><p>Documentation, code repositories, or API documentation</p></li><li><p>Customer support tickets &amp; FAQ databases</p></li></ul><p><strong>Step B:</strong> Extract useful, machine-readable content from raw sources.</p><p>This includes PDFs, documents, web pages, and databases. It uses PDF parsers, web crawlers, and optical character recognition (<strong>OCR</strong>) tools.</p><p><strong>Step C: </strong>Chunk the documents.</p><p>Even the latest LLM models have a small context window (even 1M tokens don&#8217;t mean you can feed your entire knowledge base into each prompt!). So break documents into tiny pieces, typically 200-1000 tokens long.</p><p>Split text into fixed-size chunks, semantic chunks, or recursive chunks.</p><ul><li><p><em>Fixed-size chunking</em> divides text into equal-length pieces.</p></li><li><p><em>Semantic chunking</em> gathers related content.</p></li><li><p><em>Recursive chunking</em> breaks text into smaller parts step by step.</p></li></ul><p><strong>Step D: </strong>These chunks enter an embedding model.</p><p>The neural network turns each chunk into a dense numerical vector with 768 to 1,536 elements. These vectors depict <em>meaning</em>. The system will cluster similar-looking chunks together, even when they use different words.</p><p>Here are the popular embedding models:</p><ul><li><p>Cohere&#8217;s embed-v3</p></li><li><p>Voyage AI&#8217;s voyage-large-2</p></li><li><p>OpenAI&#8217;s text-embedding-3-large</p></li><li><p>Open-source solutions such as BGE, E5 &amp; GTE</p></li></ul><p>A vector database<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-1" href="#footnote-1" target="_self">1</a> stores these embeddings &amp; enables fast similarity searches.</p><h4><strong>Phase 2: Query Stage</strong></h4><p>Let&#8217;s see how to find information relevant to a user&#8217;s query&#8230;</p><p><strong>Step 1:</strong> When a user asks a question, the system creates an embedding.</p><p>This guarantees the query vector is in the same semantic space as the document vectors.</p><p><strong>Step 2: </strong>The system then sends the query embedding to the vector database.</p><p>The database checks the query vector against the stored document vectors. It then identifies the most semantically similar chunks.</p><p><strong>Step 3:</strong> The vector database runs k-nearest neighbors (<strong>k-NN</strong>) search to identify the K document chunks closest to the query embedding.</p><p>Here are the typical measures of similarity:</p><ul><li><p><strong>Cosine similarity:</strong> Measures the angle between two vectors. The closer their directions, the more similar their embeddings.</p></li><li><p><strong>Dot product:</strong> Fast to compute, appropriate for normalized vectors.</p></li><li><p><strong>Euclidean distance: </strong>Measures straight-line distance.</p></li></ul><p>Yet exact k-NN search is too expensive at scale.</p><p>Vector databases <a href="https://newsletter.systemdesign.one/p/hnsw-vector-search-explained">enable billion-scale searches</a> in milliseconds using approximate nearest neighbor (<strong>ANN</strong>) methods. These include Hierarchical Navigable Small World (<strong>HNSW</strong>) graphs &amp; Inverted File Index (<strong>IVF</strong>).</p><p><strong>Step 4:</strong> Prompt construction.</p><p>The prompt includes the retrieved chunks inserted in the following format:</p><div class="highlighted_code_block" data-attrs="{&quot;language&quot;:&quot;plaintext&quot;,&quot;nodeId&quot;:&quot;6f361c13-7e29-4a3d-aebd-dfb951790dd5&quot;}" data-component-name="HighlightedCodeBlockToDOM"><pre class="shiki"><code class="language-plaintext">You are a helpful assistant.

Answer the user&#8217;s question using only the information
provided in the context below.</code></pre></div><p><strong>Step 5:</strong> The system provides the LLM with the prompt and extra context.</p><p>Then, the LLM creates a response based on the information it has retrieved. A successful RAG implementation must have the model reference its sources.</p><p>This enables verification &amp; builds trust.</p><p>Thus, LLM answers questions it has NOT encountered before, such as your company&#8217;s internal documents, research papers, customer tickets, and so on.</p><p>RAG would find the right pieces of information but miss the relationships between them. And when the answer depends on those connections, simply finding relevant chunks is NOT enough.</p><div class="captioned-button-wrap" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/p/graph-retrieval-augmented-generation?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;}" data-component-name="CaptionedButtonToDOM"><div class="preamble"><p class="cta-caption"><em><span data-color="#3a7bf9" style="color: rgb(58, 123, 249);">This newsletter has been growing super fast, thanks to your shares.</span></em></p></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/p/graph-retrieval-augmented-generation?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://newsletter.systemdesign.one/p/graph-retrieval-augmented-generation?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p></div><h3><strong>Problems with Standard RAG</strong></h3><p>As data size grows, vector search becomes less accurate. (The accuracy could drop by 12% when the collection grows from 10,000 to 100,000 pages.)</p><p>The more documents you add, the more the embeddings overlap, and the more difficult the system will find it to locate the right chunks.</p><p>But scaling is not the entire story&#8230;</p><p>Standard RAG has two fundamental limitations, and they become HARD to ignore as the system grows:</p><h4><strong>1. Chunks are treated as separate</strong></h4><p>Once the system chunks &amp; embeds a document, it converts the document into individual, independent vectors.</p><p>With RAG, there is no explicit link between chunks (even within the same/different documents). Vector search finds relevant chunks based on vector similarity.</p><p>This makes it harder to connect related facts across documents.</p><h4><strong>2. RAG cannot reason across documents</strong></h4><p>Some questions require information from many documents/sources.</p><p>RAG retrieves information chunks relevant to the question. Yet it does not link these chunks to show their relationships. This makes it hard to ask questions about the entire document collection.</p><p>These limitations make it harder for standard RAG to connect related information across chunks &amp; documents&#8230;This is where <em>GraphRAG</em> takes retrieval a step further!</p><h4 style="text-align: center;"><strong>&#167;</strong></h4><h2><strong>GraphRAG</strong></h2><p>GraphRAG adds structure to your documents and does NOT treat them as disjointed chunks in vector space.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!nxK_!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc3b301dd-518b-4e6d-ac93-bff948dc1b5f_1432x955.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!nxK_!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc3b301dd-518b-4e6d-ac93-bff948dc1b5f_1432x955.png 424w, https://substackcdn.com/image/fetch/$s_!nxK_!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc3b301dd-518b-4e6d-ac93-bff948dc1b5f_1432x955.png 848w, https://substackcdn.com/image/fetch/$s_!nxK_!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc3b301dd-518b-4e6d-ac93-bff948dc1b5f_1432x955.png 1272w, https://substackcdn.com/image/fetch/$s_!nxK_!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc3b301dd-518b-4e6d-ac93-bff948dc1b5f_1432x955.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!nxK_!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc3b301dd-518b-4e6d-ac93-bff948dc1b5f_1432x955.png" width="1432" height="955" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/c3b301dd-518b-4e6d-ac93-bff948dc1b5f_1432x955.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:955,&quot;width&quot;:1432,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!nxK_!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc3b301dd-518b-4e6d-ac93-bff948dc1b5f_1432x955.png 424w, https://substackcdn.com/image/fetch/$s_!nxK_!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc3b301dd-518b-4e6d-ac93-bff948dc1b5f_1432x955.png 848w, https://substackcdn.com/image/fetch/$s_!nxK_!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc3b301dd-518b-4e6d-ac93-bff948dc1b5f_1432x955.png 1272w, https://substackcdn.com/image/fetch/$s_!nxK_!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc3b301dd-518b-4e6d-ac93-bff948dc1b5f_1432x955.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>An LLM processes each piece of text, identifies entities<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-2" href="#footnote-2" target="_self">2</a>, and the relationships between them. These entities become nodes in a knowledge graph, and the relationships become edges connecting these nodes.</p><p>So you end up with a neat graph of your entire dataset, showing how your data is related in reality across documents.</p><p>GraphRAG does more than match text to your query:</p><p>It infers facts from related entities. Plus, it traces facts across documents &amp; find information spread across various texts. And answers questions about the entire dataset.</p><p>Let&#8217;s follow the journey from raw documents to a <em>searchable knowledge graph</em>&#8230;</p><h3><strong>How GraphRAG Works</strong></h3><p>It has two phases:</p><ol><li><p>Indexing Stage (constructing knowledge graph)</p></li><li><p>Query Stage (retrieving from the knowledge graph)</p></li></ol><p>Let&#8217;s dive straight in!</p><p style="text-align: justify;">The general pipeline has two extra steps:</p><ul><li><p>Community detection: groups related entities into clusters. For example, people, companies, and products that frequently connect.</p></li><li><p>Community summarization: An LLM creates a short report with the important information inside each community.</p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!krHQ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fda784cfc-5d92-46b0-acab-6e68b5b96a64_1316x850.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!krHQ!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fda784cfc-5d92-46b0-acab-6e68b5b96a64_1316x850.png 424w, https://substackcdn.com/image/fetch/$s_!krHQ!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fda784cfc-5d92-46b0-acab-6e68b5b96a64_1316x850.png 848w, https://substackcdn.com/image/fetch/$s_!krHQ!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fda784cfc-5d92-46b0-acab-6e68b5b96a64_1316x850.png 1272w, https://substackcdn.com/image/fetch/$s_!krHQ!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fda784cfc-5d92-46b0-acab-6e68b5b96a64_1316x850.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!krHQ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fda784cfc-5d92-46b0-acab-6e68b5b96a64_1316x850.png" width="1316" height="850" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/da784cfc-5d92-46b0-acab-6e68b5b96a64_1316x850.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:850,&quot;width&quot;:1316,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!krHQ!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fda784cfc-5d92-46b0-acab-6e68b5b96a64_1316x850.png 424w, https://substackcdn.com/image/fetch/$s_!krHQ!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fda784cfc-5d92-46b0-acab-6e68b5b96a64_1316x850.png 848w, https://substackcdn.com/image/fetch/$s_!krHQ!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fda784cfc-5d92-46b0-acab-6e68b5b96a64_1316x850.png 1272w, https://substackcdn.com/image/fetch/$s_!krHQ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fda784cfc-5d92-46b0-acab-6e68b5b96a64_1316x850.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Think of it like organizing a huge library&#8230;</p><p>Instead of reading every book whenever someone asks a broad question, you first group related books into sections and create a summary for each section.</p><p>At query time, GraphRAG uses these summaries to answer big-picture questions about the dataset without examining every entity &amp; relationship individually.</p><h3><strong>GraphRAG vs Microsoft GraphRAG</strong></h3><p>GraphRAG is a general approach to RAG.</p><p>Instead of relying only on similar text chunks, it uses a Knowledge Graph to connect related entities &amp; information.</p><p>Microsoft GraphRAG is a specific implementation of this idea&#8230;Its pipeline includes:</p><ul><li><p>Community detection</p></li><li><p>Community summaries</p></li><li><p>Entity &amp; relationship extraction</p></li><li><p>Different retrieval methods--Local Search &amp; Global Search</p></li></ul><p>Think of GraphRAG as the idea of making pizza and Microsoft GraphRAG as one specific pizza recipe.</p><p>The idea tells you what you are trying to build&#8230; The recipe gives you specific steps for how to build it.</p><p>Next, let&#8217;s look at Knowledge Graphs &amp; how they &#8220;connect&#8221; information&#8230;</p><h4 style="text-align: center;"><strong>&#167;</strong></h4><h2><strong>Knowledge Graph</strong></h2><p>Let&#8217;s start with the basic building blocks:</p><h3><strong>Nodes, Edges, and Properties</strong></h3><p>A Knowledge Graph<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-3" href="#footnote-3" target="_self">3</a> shows how different pieces of information connect:</p><ul><li><p><strong>Nodes:</strong> Things in the graph, such as a person, company, product, or place</p></li><li><p><strong>Edges:</strong> Relationships connecting those things</p></li><li><p><strong>Properties:</strong> Extra information about nodes/relationships</p></li></ul><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!sbrp!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F00f6aaa5-4ed8-46c1-b2b6-ad1e89edd5d0_1432x377.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!sbrp!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F00f6aaa5-4ed8-46c1-b2b6-ad1e89edd5d0_1432x377.png 424w, https://substackcdn.com/image/fetch/$s_!sbrp!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F00f6aaa5-4ed8-46c1-b2b6-ad1e89edd5d0_1432x377.png 848w, https://substackcdn.com/image/fetch/$s_!sbrp!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F00f6aaa5-4ed8-46c1-b2b6-ad1e89edd5d0_1432x377.png 1272w, https://substackcdn.com/image/fetch/$s_!sbrp!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F00f6aaa5-4ed8-46c1-b2b6-ad1e89edd5d0_1432x377.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!sbrp!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F00f6aaa5-4ed8-46c1-b2b6-ad1e89edd5d0_1432x377.png" width="1432" height="377" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/00f6aaa5-4ed8-46c1-b2b6-ad1e89edd5d0_1432x377.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:377,&quot;width&quot;:1432,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!sbrp!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F00f6aaa5-4ed8-46c1-b2b6-ad1e89edd5d0_1432x377.png 424w, https://substackcdn.com/image/fetch/$s_!sbrp!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F00f6aaa5-4ed8-46c1-b2b6-ad1e89edd5d0_1432x377.png 848w, https://substackcdn.com/image/fetch/$s_!sbrp!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F00f6aaa5-4ed8-46c1-b2b6-ad1e89edd5d0_1432x377.png 1272w, https://substackcdn.com/image/fetch/$s_!sbrp!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F00f6aaa5-4ed8-46c1-b2b6-ad1e89edd5d0_1432x377.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><strong>Graph Models</strong></h3><p>A &#8220;basic graph&#8221; tells you what connects to what&#8230;While a &#8220;property graph&#8221; tells you what those things are &amp; what the connection means.</p><p>Think of a knowledge graph as a map of knowledge and a property graph as a way of putting extra details on that map.</p><p style="text-align: justify;">Let&#8217;s understand it with an example:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!pM-1!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F71a4280a-9fdb-49bd-9165-a022ca1730fa_1608x978.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!pM-1!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F71a4280a-9fdb-49bd-9165-a022ca1730fa_1608x978.png 424w, https://substackcdn.com/image/fetch/$s_!pM-1!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F71a4280a-9fdb-49bd-9165-a022ca1730fa_1608x978.png 848w, https://substackcdn.com/image/fetch/$s_!pM-1!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F71a4280a-9fdb-49bd-9165-a022ca1730fa_1608x978.png 1272w, https://substackcdn.com/image/fetch/$s_!pM-1!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F71a4280a-9fdb-49bd-9165-a022ca1730fa_1608x978.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!pM-1!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F71a4280a-9fdb-49bd-9165-a022ca1730fa_1608x978.png" width="1456" height="886" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/71a4280a-9fdb-49bd-9165-a022ca1730fa_1608x978.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:886,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!pM-1!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F71a4280a-9fdb-49bd-9165-a022ca1730fa_1608x978.png 424w, https://substackcdn.com/image/fetch/$s_!pM-1!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F71a4280a-9fdb-49bd-9165-a022ca1730fa_1608x978.png 848w, https://substackcdn.com/image/fetch/$s_!pM-1!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F71a4280a-9fdb-49bd-9165-a022ca1730fa_1608x978.png 1272w, https://substackcdn.com/image/fetch/$s_!pM-1!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F71a4280a-9fdb-49bd-9165-a022ca1730fa_1608x978.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><div class="callout-block" data-callout="true"><p><strong>Knowledge Graph</strong> = Connected Facts<br><em>John Le Carr&#233; &#8594; WROTE &#8594; Book</em></p><div><hr></div><p><strong>Property Graph</strong> = Connected Facts With Extra Details</p><p><em>John Le Carr&#233; &#8594; WROTE &#8594; Tinker, Tailor, Soldier, Spy &#8592; PURCHASED (03-02-2011) &#8592; Ian</em></p></div><p>Resource Description Framework (<strong>RDF</strong>) is a standard way to represent &amp; link information on the Web using triples.</p><p>A triplet is a basic unit of data in the graph.</p><p>And each triple contains three parts:</p><p><code>Subject &#8594; Predicate &#8594; Object</code></p><ol><li><p><strong>Subject: </strong>Node associated with the triplet</p></li><li><p><strong>Object:</strong> Node receiving the relationship points</p></li><li><p><strong>Predicate: </strong>Relationship between subject &amp; object</p></li></ol><p>So in the above example:</p><ul><li><p><strong>Subject:</strong> John le Carre, Graham Greene, Ian, Alan</p></li><li><p><strong>Predicate:</strong> Wrote, Purchased</p></li><li><p><strong>Object:</strong> Tinker, Tailor, Soldier, Spy</p></li></ul><p>RDF lets users merge data from various sources without losing original details.</p><p>i.e., you query the complete global dataset instead of specific data instances. RDFs let knowledge graphs carry properties from different data management models.</p><ul><li><p><strong>Database:</strong> Organize data from many sources into a unified collection for querying &amp; searching.</p></li><li><p><strong>Graph:</strong> Allow analysis of a networked structure.</p></li><li><p><strong>Knowledge Base:</strong> Store information collected from different sources to answer questions or support decisions.</p></li></ul><h3><strong>RDF Schema and Ontologies</strong></h3><p>RDF Schema (<strong>RDFS</strong>) is a technique to describe classes &amp; relationships with RDF data.</p><p>For example, you can define a person &amp; book as classes and define relationships between them.</p><p>An ontology goes beyond a dictionary&#8230; It formally defines concepts in a domain and shows how they relate to one another.</p><p>A health domain ontology could define:</p><div class="callout-block" data-callout="true"><p>Researcher &#9472;&#9472;conducts&#8594; Clinical trial</p><p>Clinical Trial &#9472;&#9472;studies&#8594; Disease</p><p>Clinical Trial &#9472;&#9472;tests&#8594; Drug</p></div><p>The ontology gives the graph meaning &amp; a common vocabulary.</p><h3><strong>Graph Query Languages</strong></h3><p>Cypher &amp; SPARQL (SPARQL Protocol &amp; RDF Query Language) are 2 graph query languages.</p><p><strong>Cypher</strong> is a query language used with property graphs.</p><p>It has syntax that directly specifies graph patterns:</p><div class="callout-block" data-callout="true"><p><code>MATCH (p:Person {name: "John le Carr&#233;"})-[:WROTE]-&gt;(b:Book)</code></p><p><code>RETURN b.title;</code></p></div><p>This query finds the person named John Le Carr&#233;.</p><p>It follows WROTE relationship to locate the connected book &amp; return the book&#8217;s title.</p><p><strong>SPARQL</strong> is a query language for RDF.</p><p>It operates on RDF triples rather than on property graph patterns:</p><pre><code><code>PREFIX : &lt;http://example.org/&gt;

SELECT ?book
WHERE {
  :JohnLeCarre :wrote ?book .
}</code></code></pre><p>These languages retrieve related information, but they use different graph models.</p><p>This is key for GraphRAG retrieval&#8230; Queries fetch data from relationships, not just similar meanings.</p><h3><strong>Vector Database and Knowledge Graph</strong></h3><p>Vector databases find information based on <em>similarity</em>.</p><p>A knowledge graph is a collection of entities &amp; their relationships. You link entities &amp; explore how they relate to each other.</p><p>i.e., vector search finds relevant information&#8230;while knowledge graphs link the dots.</p><p>Check &#8204;this example:</p><blockquote><p><em>Who has written research papers on Alzheimer&#8217;s disease?</em></p></blockquote><p>This involves multiple relationships:</p><blockquote><p><em>Researcher &#8594; Published &#8594; Paper &#8594; Studies &#8594; Disease &#8594; Treatment</em></p></blockquote><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!lVxL!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5866c706-976f-49e8-8762-6a065a5e081c_1608x978.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!lVxL!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5866c706-976f-49e8-8762-6a065a5e081c_1608x978.png 424w, https://substackcdn.com/image/fetch/$s_!lVxL!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5866c706-976f-49e8-8762-6a065a5e081c_1608x978.png 848w, https://substackcdn.com/image/fetch/$s_!lVxL!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5866c706-976f-49e8-8762-6a065a5e081c_1608x978.png 1272w, https://substackcdn.com/image/fetch/$s_!lVxL!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5866c706-976f-49e8-8762-6a065a5e081c_1608x978.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!lVxL!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5866c706-976f-49e8-8762-6a065a5e081c_1608x978.png" width="1456" height="886" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/5866c706-976f-49e8-8762-6a065a5e081c_1608x978.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:886,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!lVxL!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5866c706-976f-49e8-8762-6a065a5e081c_1608x978.png 424w, https://substackcdn.com/image/fetch/$s_!lVxL!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5866c706-976f-49e8-8762-6a065a5e081c_1608x978.png 848w, https://substackcdn.com/image/fetch/$s_!lVxL!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5866c706-976f-49e8-8762-6a065a5e081c_1608x978.png 1272w, https://substackcdn.com/image/fetch/$s_!lVxL!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5866c706-976f-49e8-8762-6a065a5e081c_1608x978.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Also they could work together:</p><ul><li><p>Vector search finds relevant papers, but it doesn't represent specific relationships in the vector space.</p></li><li><p>A knowledge graph directly stores these relationships, thus making it easier to track multi-step questions.</p></li></ul><h3><strong>How to Choose Between Graph and Vector Databases</strong></h3><p>Before choosing a database, answer these questions<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-4" href="#footnote-4" target="_self">4</a>:</p><ul><li><p>What types of data will you store: documents, embeddings, entities, or relationships?</p></li><li><p>Do queries rely primarily on semantic similarity or explicit relationships?</p></li><li><p>Do you need multi-hop traversal across connected entities?</p></li><li><p>Do you need to represent multiple relationship types &amp; properties?</p></li><li><p>Do queries combine semantic retrieval with graph traversal?</p></li><li><p>What are your latency, throughput &amp; scalability requirements?</p></li><li><p>How frequently will your data &amp; relationships change?</p></li><li><p>Which query languages, APIs, and ecosystem integrations do you need?</p></li></ul><div class="callout-block" data-callout="true"><p><strong>Rule of thumb:</strong></p><ul><li><p>Use vector search for semantic similarity</p></li><li><p>Use graph databases for relationship-heavy queries, and hybrid retrieval when you need both.</p></li></ul></div><h3><strong>Ontology and Schema Design</strong></h3><p>Before building a Knowledge Graph, decide which entities, relationships, and properties it should contain.</p><p>A schema defines the structure of your graph, while an ontology describes the meaning of its concepts and relationships.</p><p>For example, in a medical research knowledge graph:</p><ul><li><p><strong>Entities:</strong> Researcher, Paper, Disease, Treatment</p></li><li><p><strong>Relationships:</strong> AUTHORED, STUDIES, TREATS</p></li><li><p><strong>Properties:</strong> Name, publication date, research field</p></li></ul><p>There are two popular approaches to schema design:</p><h4>1. Top-Down Schema Design</h4><p>Start with a domain model that defines the entities &amp; relationships&#8230;Then build the graph according to that model.</p><p>Think of building a house. You create the blueprint first, then construct the house.</p><p>Similarly, you define the graph&#8217;s structure before adding data.</p><p><em><strong>When to use it:</strong> </em>When you know the domain well and need consistent terminology and relationships.</p><h4>2. Bottom-Up Schema Design</h4><p>Start with the available data.</p><p>Identify important entities and relationships, then use them to develop the schema.</p><p>Imagine having a collection of building materials. You inspect what is available before deciding how to design the house.</p><p><em><strong>When to use it:</strong></em> When you have massive collections of unstructured documents and don&#8217;t know all the relationships in advance.</p><p>In practice, you can combine both approaches&#8230;</p><p>Start with a basic schema, extract entities and relationships from your documents, then refine the schema as you discover new patterns.</p><h3 style="text-align: justify;"><strong>Common Schema Design Pitfalls</strong></h3><p style="text-align: justify;">Without rules for growth, a knowledge graph becomes hard to manage.</p><p style="text-align: justify;">Here are 5 common mistakes:</p><ul><li><p><strong>Duplicate entities:</strong> The same real-world entity appears as multiple nodes. For example, &#8220;John Smith&#8221; and &#8220;J. Smith&#8221; might refer to the same researcher but exist as separate nodes.</p></li><li><p><strong>Missing relationships:</strong> Important connections between entities are absent. For example, a Student and Course exist as nodes, but the ENROLLED_IN relationship between them is missing.</p></li><li><p><strong>Overly complex schemas:</strong> Too many entity types, relationships, and/or properties make the graph difficult to understand &amp; query. For example, storing every student&#8217;s phone number, age, and email address as separate nodes when simple properties would be enough.</p></li><li><p><strong>Inconsistent relationship names:</strong> Different names describe the same relationship. For example, AUTHORED, WROTE, and IS_AUTHOR_OF might represent the same connection between a researcher and a paper.</p></li><li><p><strong>Schema designed for a single query:</strong> A graph built to answer one specific question might struggle with new questions. For example, a university graph designed only to find students enrolled in courses might lack the relationships needed to identify their instructors.</p></li></ul><p>The goal is not to model every possible detail&#8230;</p><p>Instead, design a graph that captures the entities and relationships your application needs, while leaving room for new use cases.</p><p>Let&#8217;s see how GraphRAG uses this structure to retrieve information and generate answers&#8230;</p><h4 style="text-align: center;"><strong>&#167;</strong></h4><h2><strong>Graph RAG Pipeline</strong></h2><p>GraphRAG combines knowledge graphs with RAG to find information, follow relationships &amp; generate answers grounded in relevant evidence.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!4Emr!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb2b5f951-87a1-4cf5-ab7f-42eb1d2c03c1_1059x706.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!4Emr!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb2b5f951-87a1-4cf5-ab7f-42eb1d2c03c1_1059x706.png 424w, https://substackcdn.com/image/fetch/$s_!4Emr!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb2b5f951-87a1-4cf5-ab7f-42eb1d2c03c1_1059x706.png 848w, https://substackcdn.com/image/fetch/$s_!4Emr!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb2b5f951-87a1-4cf5-ab7f-42eb1d2c03c1_1059x706.png 1272w, https://substackcdn.com/image/fetch/$s_!4Emr!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb2b5f951-87a1-4cf5-ab7f-42eb1d2c03c1_1059x706.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!4Emr!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb2b5f951-87a1-4cf5-ab7f-42eb1d2c03c1_1059x706.png" width="1059" height="706" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/b2b5f951-87a1-4cf5-ab7f-42eb1d2c03c1_1059x706.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:706,&quot;width&quot;:1059,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!4Emr!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb2b5f951-87a1-4cf5-ab7f-42eb1d2c03c1_1059x706.png 424w, https://substackcdn.com/image/fetch/$s_!4Emr!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb2b5f951-87a1-4cf5-ab7f-42eb1d2c03c1_1059x706.png 848w, https://substackcdn.com/image/fetch/$s_!4Emr!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb2b5f951-87a1-4cf5-ab7f-42eb1d2c03c1_1059x706.png 1272w, https://substackcdn.com/image/fetch/$s_!4Emr!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb2b5f951-87a1-4cf5-ab7f-42eb1d2c03c1_1059x706.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>The pipeline has 3 key stages:</p><ol><li><p>Build &amp; store the knowledge graph.</p></li><li><p>Detect communities &amp; generate summaries.</p></li><li><p>Retrieve relevant information &amp; generate an answer.</p></li></ol><p>Let&#8217;s dive in!</p><h3>Step 1: Knowledge Graph Creation</h3><p>Convert raw documents into a structured graph of entities &amp; relationships.</p><h4>1. Data Collection</h4><p>Collect information from sources such as books, articles, research papers &amp; internal documents.</p><p>For example, a medical research system processes thousands of papers about diseases, treatments, and clinical trials.</p><h4>2. Entity and Relationship Extraction</h4><p>Identify important entities &amp; relationships between them.</p><p>You could use LLMs, Named Entity Recognition (<strong>NER</strong>), and/or rule-based methods.</p><p>Consider this sentence:</p><blockquote><p><em>&#8220;Dr. Smith authored a paper about a new treatment for Alzheimer&#8217;s disease.&#8221;</em></p></blockquote><p>The extraction process identifies:</p><ul><li><p><strong>Entities:</strong> Dr. Smith, Research Paper, Treatment, Alzheimer&#8217;s Disease</p></li><li><p><strong>Relationships:</strong> AUTHORED, STUDIES, TREATS</p></li></ul><p>Also, the system should resolve duplicate entities &amp; validate extracted relationships before adding them to the graph.</p><h4>3. Graph Construction</h4><p>Turn the extracted information into nodes, edges &amp; properties:</p><ul><li><p><strong>Nodes:</strong> Represent entities such as researchers, papers, diseases &amp; treatments.</p></li><li><p><strong>Edges:</strong> Connect related entities through relationships such as AUTHORED.</p></li><li><p><strong>Properties:</strong> Store extra details such as names, publication dates, and research fields.</p></li></ul><p>For example:</p><p><code>Dr. Smith &#8594; AUTHORED &#8594; Research Paper &#8594; STUDIES &#8594; Alzheimer's Disease</code></p><p>The result is a graph that captures both information &amp; how different facts connect.</p><h4>4. Graph Storage</h4><p>Store the graph in a graph database like Neo4j, Amazon Neptune, TigerGraph, or NebulaGraph.</p><p>Graph databases support relationship traversal &amp; pattern matching.</p><p>You could query a graph to find all researchers who authored papers about treatments for a specific disease.</p><p>Neo4j is a common GraphRAG choice because it supports the Cypher query language, graph algorithms, and vector indexes.</p><p>Plus, you could combine<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-5" href="#footnote-5" target="_self">5</a> a graph database with a specialized vector database such as Milvus, Pinecone, or Weaviate.</p><p>This gives your retrieval system access to semantic similarity &amp; explicit relationships.</p><h3>Step 2: Community Detection and Summaries</h3><p>A large knowledge graph might contain millions of nodes &amp; relationships.</p><p>And searching the entire graph for every question is going to be expensive.</p><p>Community detection organizes the graph into related groups&#8230;</p><h4>1. Community Detection</h4><p>A community is a group of nodes with relatively strong connections to one another.</p><p><em>Think of your social network:</em></p><p>You have friends, family members &amp; coworkers. Each group forms a community because its members often share stronger connections with one another than with people outside the group.</p><p>Knowledge graphs work similarly&#8230;</p><p>A medical research graph might contain communities focused on cancer research, cardiovascular diseases, and neurological disorders.</p><p>Community-detection algorithms identify these groups based on the graph&#8217;s structure.</p><p>Here are the popular approaches:</p><ul><li><p><strong>Leiden:</strong> Finds communities &amp; refines their structure.</p></li><li><p><strong>Louvain</strong><a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-6" href="#footnote-6" target="_self">6</a><strong>:</strong> Groups nodes by optimizing modularity, a measure of community structure.</p></li><li><p><strong>Label Propagation</strong><a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-7" href="#footnote-7" target="_self">7</a><strong>:</strong> Groups nodes by repeatedly propagating labels across their connections.</p></li></ul><h4>2. Community Summaries</h4><p>Once the system identifies communities, an LLM summarizes each group.</p><p>A community about Alzheimer&#8217;s disease might include researchers, treatments, clinical trials &amp; related papers. The LLM examines the available entity descriptions and relationships, then creates a report about the community.</p><p>These reports help GraphRAG answer broad questions without retrieving every individual node &amp; relationship.</p><p>For example:</p><blockquote><p><em>&#8220;What are the major research trends in Alzheimer&#8217;s treatment?&#8221;</em></p></blockquote><p>Instead of examining thousands of disconnected facts at query time, system uses relevant community reports to identify important themes.</p><h3>Step 3: Retrieval and Response Generation</h3><p>Once the graph &amp; community reports are ready, GraphRAG answers user questions.</p><p>The retrieval process has 4 parts:</p><h4>1. Query Processing</h4><p>System analyzes user&#8217;s question to identify relevant entities, relationships &amp; search intent.</p><p>For example:</p><blockquote><p><em>&#8220;Who are friends of my friends?&#8221;</em></p></blockquote><p>In a social network graph, system starts from your user node &amp; follows two friendship relationships.</p><p>You could express this with Cypher:</p><pre><code><code>MATCH (:User {userId: 'referenceUser'})
      -[:FRIEND]-&gt;
      (:User)
      -[:FRIEND]-&gt;
      (fof:User)
RETURN DISTINCT fof</code></code></pre><p>In a real app, you&#8217;d filter out original user &amp; exclude existing direct friends.</p><p>The same principle applies to GraphRAG: <em>start with relevant entities and follow meaningful relationships.</em></p><h4>2. Graph-Based Retrieval</h4><p>The system retrieves a relevant portion of the knowledge graph - <strong>subgraph</strong>.</p><p>It uses 2 techniques:</p><ul><li><p><strong>Subgraph Extraction:</strong> Find entities &amp; relationships directly relevant to question.</p></li><li><p><strong>Context Expansion:</strong> Follow additional relationships to gather supporting evidence.</p></li></ul><p>For example, you ask:</p><blockquote><p><em>&#8220;Which researchers are studying treatments for Alzheimer&#8217;s disease?&#8221;</em></p></blockquote><p>Here&#8217;s what system would do:</p><ol><li><p>Find Alzheimer&#8217;s Disease node.</p></li><li><p>Follow relationships to relevant treatments &amp; research papers.</p></li><li><p>Find the researchers who authored those papers.</p></li><li><p>Retrieve supporting text from the original documents.</p></li></ol><p>You control how far system explores using hop limits, relationship filters, and relevance scores.</p><p>This prevents retrieval from expanding into unrelated parts of the graph.</p><h4>3. Information Aggregation</h4><p>Combine retrieved entities, relationships &amp; supporting text into context for LLM.</p><p>The system would collect:</p><ul><li><p>Researchers &amp; their publications</p></li><li><p>Treatments discussed in those publications</p></li><li><p>Relationships between treatments &amp; diseases</p></li><li><p>Supporting passages from research papers</p></li></ul><p>The goal is to give the LLM enough evidence to answer the question without overwhelming it with <em>irrelevant</em> information.</p><h4>4. Response Generation</h4><p>LLM uses the retrieved context to generate a natural-language answer.</p><p>It identify researchers studying Alzheimer&#8217;s treatments and explain which papers support those connections.</p><p>When the system preserves source references, it also cite documents behind its answer. The key difference is GraphRAG retrieves information through both relevance &amp; relationships.</p><ul><li><p>Vector search helps find semantically similar content.</p></li><li><p>Graph traversal helps uncover how entities connect.</p></li></ul><p>They provide a richer context for questions that depend on many connected facts.</p><p>Now we understand GraphRAG&#8230; when should we use it instead of standard RAG?</p><h3 style="text-align: justify;"><strong>GraphRAG vs Standard RAG: when to use each</strong></h3><p>GraphRAG complements traditional vector RAG&#8230;Each approach serves different retrieval needs.</p><p>Consider using GraphRAG when:</p><ul><li><p>Your documents contain relationships between entities, events, concepts.</p></li><li><p>Questions must draw connections between facts, follow a sequence, or look for patterns.</p></li><li><p>Answers require identifying themes, patterns, trends across an entire dataset.</p></li><li><p>You need to explore dependencies or trace connections between entities.</p></li><li><p>Your app benefits from structured, relationship-aware retrieval, such as legal research, scientific research, fraud investigation.</p></li></ul><p>Consider using Vector RAG when:</p><ul><li><p>Questions require straightforward factual lookups, such as <em>&#8220;What is our annual leave policy?&#8221;</em></p></li><li><p>Relevant information is available in one or a few retrieved chunks.</p></li><li><p>Semantic similarity is sufficient to find the required information.</p></li><li><p>You want a simple retrieval architecture with lower indexing &amp; maintenance overhead.</p></li></ul><h4 style="text-align: center;"><strong>&#167;</strong></h4><div class="callout-block" data-callout="true"><p style="text-align: center;"><em><strong><span data-color="#3a7bf9" style="color: rgb(58, 123, 249);">Reminder: this is a teaser of the subscriber-only newsletter, exclusive to my golden members.</span></strong></em></p><p>When you upgrade, you&#8217;ll get:</p><ul><li><p><strong>High-level architecture of real-world systems.</strong></p></li><li><p>Deep dive into how popular real-world systems work.</p></li><li><p><strong>How real-world systems handle scale, reliability &amp; performance.</strong></p></li></ul><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/subscribe&quot;,&quot;text&quot;:&quot;Unlock Full Access&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://newsletter.systemdesign.one/subscribe"><span>Unlock Full Access</span></a></p><p>(If this newsletter has helped you become a better software engineer, consider subscribing to support my work.)</p></div><h4 style="text-align: center;"><strong>&#167;</strong></h4><h2><strong>GraphRAG Retrieval Strategies</strong></h2><p>Let&#8217;s see how GraphRAG finds the right information:</p>
      <p>
          <a href="https://newsletter.systemdesign.one/p/graph-retrieval-augmented-generation">
              Read more
          </a>
      </p>
   ]]></content:encoded></item><item><title><![CDATA[The Only Way to Debug AI Agents at Scale]]></title><description><![CDATA[#203: Break Into AI Agents]]></description><link>https://newsletter.systemdesign.one/p/ai-agent-observability-opentelemetry</link><guid isPermaLink="false">https://newsletter.systemdesign.one/p/ai-agent-observability-opentelemetry</guid><dc:creator><![CDATA[Neo Kim]]></dc:creator><pubDate>Sat, 14 Nov 2026 11:57:37 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/f24f6d27-7c6a-42c6-9df7-1f3f381b735f_1280x720.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<div class="callout-block" data-callout="true"><h4><a href="https://www.vpdae.com/redirect/ofa0oy6735yxmgym2eohz7btcd2">AI agents need to call APIs, databases, and tools. But how do you give them access without embedding API keys or granting sweeping permissions?</a></h4><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://www.vpdae.com/redirect/ofa0oy6735yxmgym2eohz7btcd2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!ZYu4!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F302fd990-2e23-4c8a-8053-2e2601b945a8_600x400.png 424w, https://substackcdn.com/image/fetch/$s_!ZYu4!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F302fd990-2e23-4c8a-8053-2e2601b945a8_600x400.png 848w, https://substackcdn.com/image/fetch/$s_!ZYu4!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F302fd990-2e23-4c8a-8053-2e2601b945a8_600x400.png 1272w, https://substackcdn.com/image/fetch/$s_!ZYu4!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F302fd990-2e23-4c8a-8053-2e2601b945a8_600x400.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!ZYu4!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F302fd990-2e23-4c8a-8053-2e2601b945a8_600x400.png" width="726" height="484" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/302fd990-2e23-4c8a-8053-2e2601b945a8_600x400.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:400,&quot;width&quot;:600,&quot;resizeWidth&quot;:726,&quot;bytes&quot;:233277,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:&quot;https://www.vpdae.com/redirect/ofa0oy6735yxmgym2eohz7btcd2&quot;,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://newsletter.systemdesign.one/i/195272243?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F302fd990-2e23-4c8a-8053-2e2601b945a8_600x400.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!ZYu4!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F302fd990-2e23-4c8a-8053-2e2601b945a8_600x400.png 424w, https://substackcdn.com/image/fetch/$s_!ZYu4!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F302fd990-2e23-4c8a-8053-2e2601b945a8_600x400.png 848w, https://substackcdn.com/image/fetch/$s_!ZYu4!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F302fd990-2e23-4c8a-8053-2e2601b945a8_600x400.png 1272w, https://substackcdn.com/image/fetch/$s_!ZYu4!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F302fd990-2e23-4c8a-8053-2e2601b945a8_600x400.png 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Learn the design patterns behind secure agent access.</p><p>Join our <strong><a href="https://www.vpdae.com/redirect/ofa0oy6735yxmgym2eohz7btcd2">Nov 18 live webinar</a></strong>, Agent Identity &amp; Access: Building Secure Agentic Systems.</p><p>Explore credential brokering, scoped permissions, and short-lived sessions, with a demo of <a href="https://www.vpdae.com/redirect/ofa0oy6735yxmgym2eohz7btcd2">Infisical</a> Agent Vault, a proxy that keeps API keys out of agents.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://www.vpdae.com/redirect/ofa0oy6735yxmgym2eohz7btcd2&quot;,&quot;text&quot;:&quot;Save your spot&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://www.vpdae.com/redirect/ofa0oy6735yxmgym2eohz7btcd2"><span>Save your spot</span></a></p><p><em>(Thanks to <a href="https://www.vpdae.com/redirect/ofa0oy6735yxmgym2eohz7btcd2">Infisical</a> for partnering on this post.)</em></p></div><div><hr></div><p>Building an AI agent is straightforward.</p><p>But running hundreds of agents reliably in production is HARD.</p><p>An agent might work perfectly during development, yet fail at scale in production. It could get stuck in a reasoning loop, call the wrong tool, exceed its token budget, or return an incorrect answer.</p><p>And when something goes wrong, you need to identify which step caused the failure.</p><p>Traditional application monitoring works, but AI agents create new challenges. Their execution paths vary, they make decisions at runtime, and a single user request could trigger multiple agents, tool calls, and model invocations.</p><p>This is where an AI agent harness comes into play.</p><p>An agent harness provides the infrastructure &amp; controls around a Large Language Model (<strong>LLM</strong>) to manage its execution.</p><p>A production-ready agent harness includes these components:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!B0pV!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4cfef015-a515-40cd-8d1b-680842c962c0_1808x756.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!B0pV!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4cfef015-a515-40cd-8d1b-680842c962c0_1808x756.png 424w, https://substackcdn.com/image/fetch/$s_!B0pV!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4cfef015-a515-40cd-8d1b-680842c962c0_1808x756.png 848w, https://substackcdn.com/image/fetch/$s_!B0pV!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4cfef015-a515-40cd-8d1b-680842c962c0_1808x756.png 1272w, https://substackcdn.com/image/fetch/$s_!B0pV!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4cfef015-a515-40cd-8d1b-680842c962c0_1808x756.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!B0pV!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4cfef015-a515-40cd-8d1b-680842c962c0_1808x756.png" width="1456" height="609" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/4cfef015-a515-40cd-8d1b-680842c962c0_1808x756.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:609,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:&quot;&quot;,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!B0pV!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4cfef015-a515-40cd-8d1b-680842c962c0_1808x756.png 424w, https://substackcdn.com/image/fetch/$s_!B0pV!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4cfef015-a515-40cd-8d1b-680842c962c0_1808x756.png 848w, https://substackcdn.com/image/fetch/$s_!B0pV!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4cfef015-a515-40cd-8d1b-680842c962c0_1808x756.png 1272w, https://substackcdn.com/image/fetch/$s_!B0pV!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4cfef015-a515-40cd-8d1b-680842c962c0_1808x756.png 1456w" sizes="auto, 100vw" loading="lazy" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Fig. 1: Agentic AI Harness Building Blocks</figcaption></figure></div><ul><li><p><strong>Reasoning loop:</strong> Plan tasks, execute actions, observe results, and adjust the execution plan when necessary.</p></li><li><p><strong>Context and memory management:</strong> Manage the agent&#8217;s context window, preserve execution state, and retrieve relevant information from short-term or long-term memory.</p></li><li><p><strong>Evaluations (Evals):</strong> Measure response quality, task completion, and agent behavior using ground-truth datasets, automated checks, or LLM-as-a-Judge.</p></li><li><p><strong>Security:</strong> Enforce authentication, authorization, and access controls across applications, agents, tools, and data sources.</p></li><li><p><strong>Guardrails:</strong> Detect and mitigate risks such as prompt injection, unauthorized tool use, and memory poisoning.</p></li><li><p><strong>Human-in-the-loop (HITL):</strong> Allow users/operators to review decisions, approve sensitive actions, and intervene during execution.</p></li></ul><p>These components make agent execution more reliable.</p><p>Yet you still need visibility into how agents behave once they enter production. That&#8217;s where observability becomes critical. You need to trace every agent execution, identify failures, measure response quality, and understand each workflow's cost.</p><p>Onward.</p><div><hr></div><p><em>Here&#8217;s what&#8217;s inside:</em></p><ul><li><p><strong>Agentic AI architecture:</strong> How reasoning, orchestration, memory, and tools work together in an agentic system.</p></li><li><p><strong>End-to-end observability:</strong> How to trace agent executions across models, tools, and external services.</p></li><li><p><strong>OpenTelemetry:</strong> Which trace identifiers and attributes let you track agent behavior across distributed systems.</p></li><li><p><strong>Agent evaluations:</strong> How to measure task success, response quality, and tool usage using telemetry and automated evaluations.</p></li><li><p><strong>AI FinOps:</strong> How to estimate infrastructure and inference costs, measure token usage, and optimize agent performance at scale.</p></li></ul><p></p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption"><span data-color="#3a7bf9" style="color: rgb(58, 123, 249);">Get my AI engineering playbook for FREE on newsletter signup:</span></p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><ul><li><p><em><a href="https://newsletter.systemdesign.one/p/ai-agent-observability-opentelemetry/?action=share">Share this letter</a> &amp; I&#8217;ll send you some rewards for the referrals.</em></p></li></ul><div><hr></div><h2><strong>Agentic AI Reference Architecture</strong></h2><p>For simple tasks, a single agent<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-1" href="#footnote-1" target="_self">1</a> is enough&#8230;</p><p>But complex workflows require many specialized agents to work together.</p><p>Imagine an AI research assistant tasked with analyzing a company&#8217;s financial performance:</p><ul><li><p>One agent retrieves financial reports,</p></li><li><p>Another agent analyzes the numbers,</p></li><li><p>And a third agent prepares the final summary.</p></li></ul><p>An orchestration engine coordinates their execution &amp; manage task dependencies.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!2p6X!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffaac08a2-7ca4-40b2-bddb-3dde22cb9c2b_1704x924.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!2p6X!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffaac08a2-7ca4-40b2-bddb-3dde22cb9c2b_1704x924.png 424w, https://substackcdn.com/image/fetch/$s_!2p6X!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffaac08a2-7ca4-40b2-bddb-3dde22cb9c2b_1704x924.png 848w, https://substackcdn.com/image/fetch/$s_!2p6X!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffaac08a2-7ca4-40b2-bddb-3dde22cb9c2b_1704x924.png 1272w, https://substackcdn.com/image/fetch/$s_!2p6X!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffaac08a2-7ca4-40b2-bddb-3dde22cb9c2b_1704x924.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!2p6X!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffaac08a2-7ca4-40b2-bddb-3dde22cb9c2b_1704x924.png" width="1456" height="790" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/faac08a2-7ca4-40b2-bddb-3dde22cb9c2b_1704x924.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:790,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:&quot;&quot;,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!2p6X!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffaac08a2-7ca4-40b2-bddb-3dde22cb9c2b_1704x924.png 424w, https://substackcdn.com/image/fetch/$s_!2p6X!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffaac08a2-7ca4-40b2-bddb-3dde22cb9c2b_1704x924.png 848w, https://substackcdn.com/image/fetch/$s_!2p6X!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffaac08a2-7ca4-40b2-bddb-3dde22cb9c2b_1704x924.png 1272w, https://substackcdn.com/image/fetch/$s_!2p6X!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffaac08a2-7ca4-40b2-bddb-3dde22cb9c2b_1704x924.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Fig. 2: Agentic AI Platform Reference Architecture</figcaption></figure></div><p>Let&#8217;s examine the key components:</p><h3>1. Reasoning and Orchestration</h3><p>The reasoning layer determines how an agent approaches a task.</p><p>For a complex request, the planner breaks the original goal into smaller subtasks. The orchestration engine then coordinates their execution, handles dependencies, and collects the results.</p><p>Here are two popular approaches that let agents reason and execute tasks:</p><ul><li><p><strong>Chain-of-Thought (CoT):</strong> A prompting technique that encourages a model to work through intermediate reasoning steps before producing an answer. It supports complex problem-solving but does not provide task orchestration.</p></li><li><p><strong>Reasoning and Acting (ReAct):</strong> An agent pattern that alternates between reasoning, tool execution, and observations. The agent uses tool results to decide its next action and adjust its plan.</p></li></ul><p>A research agent might search for a company&#8217;s annual report, extract its revenue figures, and calculate year-over-year growth. If the report is unavailable, the agent could choose another source.</p><p>The orchestration engine manages this execution flow, with retries, task dependencies, and termination conditions.</p><h3>2. Agent Discovery and Composition</h3><p>When multiple agents collaborate, the system needs a way to discover their capabilities and determine which agent should handle each task.</p><p>An agent registry maintains descriptions of available agents, their supported operations, and their access requirements.</p><p>The Agent2Agent (<strong>A2A</strong>) protocol provides a standardized way for agents to communicate and discover capabilities through an Agent Card.</p><p>An Agent Card is a JSON document containing this data:</p><ul><li><p><strong>Identity:</strong> Agent&#8217;s name, description, and provider information.</p></li><li><p><strong>Service endpoint:</strong> The address where other apps/agents access its A2A service.</p></li><li><p><strong>Capabilities:</strong> Supported features, such as streaming &amp; push notifications.</p></li><li><p><strong>Authentication:</strong> Security requirements for accessing the agent.</p></li><li><p><strong>Skills:</strong> Tasks the agent supports, including descriptions, identifiers, and input/output formats.</p></li></ul><p>An orchestration system discovers a financial analysis agent through its Agent Card and delegates a relevant task to it.</p><p>A2A supports interoperability between agents built using different frameworks.</p><p>A centralized marketplace is optional; agents could also be discovered through configured endpoints or registries.</p><h3>3. Agent Integration and Tool Access</h3><p>Once agents are selected, they need a way to communicate with other agents, applications, and external services.</p><p>The integration layer supports several interaction patterns:</p><ul><li><p><strong>Agent-to-agent:</strong> One agent delegates work to another through an API or an agent communication protocol.</p></li><li><p><strong>Agent-to-application:</strong> An agent exchanges data with an application or returns results for users.</p></li><li><p><strong>Human-to-agent:</strong> A user initiates a task or provides additional instructions.</p></li><li><p><strong>Human-in-the-loop:</strong> An agent requests human approval/intervention before continuing a workflow.</p></li></ul><p>The Model Context Protocol (<strong><a href="https://www.anthropic.com/news/model-context-protocol">MCP</a></strong>) provides a standardized interface for AI applications to interact with external systems.</p><p>MCP uses a client-server architecture and exposes three capabilities:</p><ul><li><p><strong>Resources:</strong> Data and contextual information exposed by an MCP server.</p></li><li><p><strong>Tools:</strong> Functions an AI app could invoke to retrieve information or perform actions.</p></li><li><p><strong>Prompts:</strong> Reusable prompt templates exposed by an MCP server.</p></li></ul><p>A research agent could use an MCP server to retrieve documents from an enterprise knowledge base. Another tool could query a database or create a report.</p><p>MCP standardizes interactions between AI apps and external capabilities, while A2A focuses on communication between agents.</p><p>Both protocols could coexist within the same architecture.</p><h3>4. Context and Memory Management</h3><p>Complex agents execute multiple steps over extended periods.</p><p>So they need to preserve relevant information between actions and across separate sessions.</p><p>Memory management lets agents maintain context without repeatedly processing the entire interaction history.</p><p>Here are three key components:</p><ul><li><p><strong>Working context:</strong> Information currently available within the model&#8217;s context window--instructions, recent messages, and tool results.</p></li><li><p><strong>Execution state:</strong> Persistent information about workflow progress, completed tasks, pending actions, and checkpoints.</p></li><li><p><strong>Long-term memory:</strong> Information retained across interactions, such as user preferences, past outcomes, or retrieved knowledge.</p></li></ul><p>For semantic memory<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-2" href="#footnote-2" target="_self">2</a>, one common approach is to convert text into numerical representations called embeddings and store them in a vector database.</p><p>The database retrieves semantically similar information using similarity measures such as cosine similarity or inner product.</p><p>At scale, Approximate Nearest Neighbor (<strong>ANN</strong>) search accelerates retrieval by trading some accuracy for faster searches. Maximum Inner Product Search (<strong>MIPS</strong><a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-3" href="#footnote-3" target="_self">3</a>) is one possible retrieval objective.</p><p>Yet vector databases are only one part of memory management.</p><p>Execution state belongs in a durable database or checkpoint store. Structured records and conversation histories use other storage systems.</p><p>The architecture should select the appropriate storage mechanism based on the memory type and how the agent needs to access it.</p><h3>5. Security, Governance, and Guardrails</h3><p>AI agents interact with external systems and perform actions on behalf of users.</p><p>This creates security risks when agents access sensitive information or invoke tools that modify data.</p><p>So a production architecture needs security controls at multiple levels:</p><p><strong>I) Authentication and authorization</strong></p><p>Agents should access only the resources &amp; operations required for their tasks.</p><p>A financial analysis agent might need permission to read financial records but should not automatically receive permission to modify them.</p><p>The underlying services and tools must enforce authorization, rather than relying solely on instructions provided to the model.</p><p><strong>II) Agent-specific guardrails</strong></p><p>Guardrails detect/prevent unsafe behaviors:</p><ul><li><p>Unauthorized/inappropriate tool execution.</p></li><li><p>Memory poisoning through malicious stored information.</p></li><li><p>Sensitive data exposure across users, agents, or sessions.</p></li><li><p>Prompt injection through untrusted documents or tool responses.</p></li></ul><p>Human approval is another important control for high-risk operations, such as financial transactions or changes to production infrastructure.</p><p>A central governance layer lets you define policies and monitor compliance. Yet individual components must enforce the controls relevant to their responsibilities.</p><p>A tool gateway should validate permissions before executing an action, while the memory layer should enforce access controls when retrieving stored information.</p><p>This architecture establishes how agents execute tasks and interact with external systems.</p><p>The next challenge is understanding what happens during execution.</p><p>When an agent produces an &#8220;incorrect&#8221; result, you need to identify whether the failure originated in its reasoning, tool execution, retrieved context, or an external service.</p><p>That&#8217;s where end-to-end <em>observability</em> becomes critical&#8230;</p><div><hr></div><h2><strong>Agentic AI Observability Layer</strong></h2><p>Traditional applications follow predefined execution paths.</p><p>AI agents add complexity because their decisions, tool calls, and execution paths can vary from run to run.</p><p>Imagine a research agent tasked with analyzing a company&#8217;s financial performance:</p><p>During one execution, it retrieves an annual report and extracts the required figures. During another, it might search many sources, retry a failed API call, or delegate the analysis to another agent.</p><p>Both executions would produce the same answer, but their latency, resource usage, and costs differ.</p><p>So you need end-to-end observability to understand what happens inside each agent execution.</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!G-MG!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F813df49c-277b-4cef-b0ec-ef1882c8e9a6_1894x494.png 424w, https://substackcdn.com/image/fetch/$s_!G-MG!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F813df49c-277b-4cef-b0ec-ef1882c8e9a6_1894x494.png 848w, https://substackcdn.com/image/fetch/$s_!G-MG!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F813df49c-277b-4cef-b0ec-ef1882c8e9a6_1894x494.png 1272w, https://substackcdn.com/image/fetch/$s_!G-MG!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F813df49c-277b-4cef-b0ec-ef1882c8e9a6_1894x494.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!G-MG!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F813df49c-277b-4cef-b0ec-ef1882c8e9a6_1894x494.png" width="728" height="190" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/813df49c-277b-4cef-b0ec-ef1882c8e9a6_1894x494.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:false,&quot;imageSize&quot;:&quot;normal&quot;,&quot;height&quot;:380,&quot;width&quot;:1456,&quot;resizeWidth&quot;:728,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:&quot;&quot;,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:&quot;center&quot;,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!G-MG!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F813df49c-277b-4cef-b0ec-ef1882c8e9a6_1894x494.png 424w, https://substackcdn.com/image/fetch/$s_!G-MG!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F813df49c-277b-4cef-b0ec-ef1882c8e9a6_1894x494.png 848w, https://substackcdn.com/image/fetch/$s_!G-MG!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F813df49c-277b-4cef-b0ec-ef1882c8e9a6_1894x494.png 1272w, https://substackcdn.com/image/fetch/$s_!G-MG!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F813df49c-277b-4cef-b0ec-ef1882c8e9a6_1894x494.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div></div></div></div><figcaption class="image-caption">Fig. 3: End-to-End Observability of Agentic AI Systems</figcaption></figure></div><p>This visibility offers these capabilities:</p><ul><li><p><strong>End-to-end tracing:</strong> Track the complete execution path across user requests, agent decisions, model invocations, tool calls, and external services.</p></li><li><p><strong>Failure analysis:</strong> Identify the source of errors, incorrect outputs, latency spikes, or unexpected costs. Traces provide diagnostic evidence, while evaluations allow assessing response quality.</p></li><li><p><strong>Auditability and governance:</strong> Record execution history, policy decisions, guardrail outcomes, and resource usage. Durable audit records with appropriate integrity controls support compliance and accountability.</p></li><li><p><strong>Continuous improvement:</strong> Use execution data and evaluation results to identify inefficient workflows, improve prompts, refine agent logic, and measure the impact of changes.</p></li></ul><p>If an agent takes 30 seconds to complete a task, a distributed trace could reveal 20 seconds were spent waiting for an external API.</p><p>You could then optimize the slow dependency instead of changing the model or rewriting the agent&#8217;s reasoning logic.</p><p>But observability should begin before an agent reaches production.</p><h3>Build-Time vs Runtime Observability</h3><p>Agentic AI systems require visibility during development and production lifecycle.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!i2BN!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3f9ec6f3-9b65-4df4-a5d7-eb0ccbb64517_2022x892.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!i2BN!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3f9ec6f3-9b65-4df4-a5d7-eb0ccbb64517_2022x892.png 424w, https://substackcdn.com/image/fetch/$s_!i2BN!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3f9ec6f3-9b65-4df4-a5d7-eb0ccbb64517_2022x892.png 848w, https://substackcdn.com/image/fetch/$s_!i2BN!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3f9ec6f3-9b65-4df4-a5d7-eb0ccbb64517_2022x892.png 1272w, https://substackcdn.com/image/fetch/$s_!i2BN!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3f9ec6f3-9b65-4df4-a5d7-eb0ccbb64517_2022x892.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!i2BN!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3f9ec6f3-9b65-4df4-a5d7-eb0ccbb64517_2022x892.png" width="1456" height="642" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/3f9ec6f3-9b65-4df4-a5d7-eb0ccbb64517_2022x892.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:642,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:&quot;&quot;,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!i2BN!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3f9ec6f3-9b65-4df4-a5d7-eb0ccbb64517_2022x892.png 424w, https://substackcdn.com/image/fetch/$s_!i2BN!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3f9ec6f3-9b65-4df4-a5d7-eb0ccbb64517_2022x892.png 848w, https://substackcdn.com/image/fetch/$s_!i2BN!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3f9ec6f3-9b65-4df4-a5d7-eb0ccbb64517_2022x892.png 1272w, https://substackcdn.com/image/fetch/$s_!i2BN!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3f9ec6f3-9b65-4df4-a5d7-eb0ccbb64517_2022x892.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Fig. 4: Build-Time and Runtime Observability Across the Agentic Lifecycle</figcaption></figure></div><h3>1. Build-Time Observability</h3><p>Build-time observability focuses on examining system behavior during development, testing, and pre-deployment validation.</p><p>It combines telemetry from test executions with evaluation results and system configuration checks.</p><ul><li><p><strong>Agent evaluation:</strong> Test whether agents complete expected tasks using predefined datasets and success criteria.</p></li><li><p><strong>Tool validation:</strong> Verify that agents select correct tools, provide valid parameters, and handle failures.</p></li><li><p><strong>Performance testing:</strong> Measure latency, token consumption, and resource usage under representative workloads.</p></li><li><p><strong>Security validation:</strong> Test access controls, guardrails, and resistance to prompt injection or unauthorized tool execution.</p></li></ul><p>You could run a research agent against 100 predefined tasks and measure its task completion rate, tool call accuracy, and average execution cost.</p><p>These results establish a baseline before deployment.</p><p>Yet passing pre-deployment tests does not guarantee reliable behavior in production.</p><h3>2. Runtime Observability</h3><p>Runtime observability tracks how agents behave during actual execution.</p><p>It relies on three core telemetry signals:</p><ul><li><p><strong>Logs:</strong> Record discrete events such as tool failures, policy decisions, and execution errors.</p></li><li><p><strong>Metrics:</strong> Measure numerical indicators such as request latency, token consumption, error rates, and cost per task.</p></li><li><p><strong>Traces:</strong> Connect operations across agents, models, tools, and external services to reconstruct execution paths.</p></li></ul><p>These signals let you investigate failures and monitor production performance.</p><p>An agent could repeatedly invoke the same tool without making progress. A trace would reveal the repeated calls, while metrics could show the resulting increase in latency and token consumption.</p><p>This matters because agent behavior could change based on user inputs, retrieved information, and intermediate tool results.</p><p>Runtime monitoring lets you identify these risks:</p><ul><li><p><strong>Goal drift:</strong> The agent gradually deviates from the original task.</p></li><li><p><strong>Recursive loops:</strong> Repeated reasoning or tool calls prevent the workflow from completing.</p></li><li><p><strong>Unexpected tool execution:</strong> The agent invokes unnecessary or inappropriate operations.</p></li><li><p><strong>Cost overruns:</strong> Excessive model calls or retries consume more resources than expected.</p></li></ul><p>Observability detects and investigates these behaviors.</p><p>Enforcing execution limits, permissions, and safety policies requires extra controls within the agent architecture.</p><p>Build-time evaluation establishes expected behavior. Runtime observability reveals how the agent performs under real workloads. Thus providing the evidence needed to improve agent reliability and operational efficiency.</p><p>The next step is to connect these observations across distributed components.</p><p>OpenTelemetry collects and correlates telemetry from agent executions, model invocations, and tool calls.</p><div><hr></div><h2><strong>OpenTelemetry for AI Agent Observability</strong></h2><p>When an AI agent executes a task, it could invoke multiple models, call external tools, and delegate work to other agents.</p><p>Each operation generates telemetry.</p><p>But without a common way to connect these events, identifying the cause of a failure becomes hard.</p><p>OpenTelemetry (<strong>OTel</strong>) provides a vendor-neutral framework for collecting and exporting traces, metrics, and logs across distributed systems.</p><p>Distributed tracing lets you reconstruct a task's execution path and understand where time and resources were spent in agentic AI.</p><h3>1. Core Tracing Principles</h3><p>Before instrumenting an agentic system, you need a consistent approach to tracing operations across components.</p><p>Here are four principles to establish the foundation:</p><ul><li><p><strong>Standardized trace propagation:</strong> Use the World Wide Web Consortium (<strong>W3C</strong>) Trace Context standard to propagate tracing information<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-4" href="#footnote-4" target="_self">4</a> across services.</p></li><li><p><strong>One task, multiple spans:</strong> Represent an AI task with a trace. Each model invocation, tool call, or agent operation becomes a span. Record retries as separate operations to preserve their execution history.</p></li><li><p><strong>Span links for related operations:</strong> Use span<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-5" href="#footnote-5" target="_self">5</a> links to connect operations with non-hierarchical relationships, such as asynchronous jobs or workflows that cross trace boundaries. Parent-child relationships remain appropriate for ordinary parallel operations within a trace.</p></li><li><p><strong>Opaque identifiers:</strong> Keep trace and span identifiers free of business meaning. Store task details, agent names, and other contextual information in attributes.</p></li></ul><p>These principles maintain consistent tracing across agent frameworks and external services.</p><h3>2. Trace Identifiers and Relationships</h3><p>OpenTelemetry uses traces and spans to represent distributed operations.</p><p>A trace represents the execution path of a request/operation. Each span records an individual operation within that trace.</p><p>Here are four important identifiers and relationships:</p><ul><li><p><code>trace_id</code><strong>:</strong> Identifies a trace and connects related spans across services.</p></li><li><p><code>span_id</code><strong>:</strong> Uniquely identifies a span within its trace.</p></li><li><p><code>parent_span_id</code><strong>:</strong> Identifies the parent span, establishing the execution hierarchy.</p></li><li><p>Span links<strong>:</strong> Associate a span with other span contexts, including operations in separate traces.</p></li></ul><p>Imagine an AI research agent that receives a request, retrieves documents, and generates a summary.</p><p>The execution could produce the following trace:</p><pre><code><code>TRACE: Research Task
Trace ID: 4bf92f3577b34da6a3ce929d0e0e4736
Start: 10:00:00.000
End:   10:00:02.500
Duration: 2500 ms
&#9474;
&#9492;&#9472;&#9472; SPAN 1: HTTP POST /research
    &#9474;   Span ID: a1 (shortened)
    &#9474;   Parent Span ID: None (root)
    &#9474;   Start: 10:00:00.000
    &#9474;   End:   10:00:02.500
    &#9474;   Duration: 2500 ms
    &#9474;   Attributes: http.request.method=POST
    &#9474;   Status: UNSET
    &#9474;   Events: None
    &#9474;
    &#9492;&#9472;&#9472; SPAN 2: Agent: Research Agent
        &#9474;   Span ID: b2
        &#9474;   Parent Span ID: a1
        &#9474;   Start: 10:00:00.050
        &#9474;   End:   10:00:02.400
        &#9474;   Duration: 2350 ms
        &#9474;   Attributes: gen_ai.agent.name=research_agent
        &#9474;   Status: UNSET
        &#9474;   Events: None
        &#9474;
        &#9500;&#9472;&#9472; SPAN 3: LLM: Generate Plan
        &#9474;   Span ID: c3
        &#9474;   Parent Span ID: b2
        &#9474;   Start: 10:00:00.100
        &#9474;   End:   10:00:00.600
        &#9474;   Duration: 500 ms
        &#9474;   Attributes: gen_ai.request.model=gpt-6
        &#9474;   Status: UNSET
        &#9474;   Events: None
        &#9474;
        &#9500;&#9472;&#9472; SPAN 4: Tool: Search Documents
        &#9474;   Span ID: d4
        &#9474;   Parent Span ID: b2
        &#9474;   Start: 10:00:00.650
        &#9474;   End:   10:00:01.100
        &#9474;   Duration: 450 ms
        &#9474;   Attributes: gen_ai.tool.name=search_documents
        &#9474;   Status: UNSET
        &#9474;   Events: None
        &#9474;   &#9474;
        &#9474;   &#9492;&#9472;&#9472; SPAN 5: HTTP: Search API
        &#9474;       Span ID: e5
        &#9474;       Parent Span ID: d4
        &#9474;       Start: 10:00:00.700
        &#9474;       End:   10:00:01.050
        &#9474;       Duration: 350 ms
        &#9474;       Attributes: http.request.method=GET
        &#9474;       Status: UNSET
        &#9474;       Events: None
        &#9474;
        &#9500;&#9472;&#9472; SPAN 6: Tool: Retrieve Report
        &#9474;   Span ID: f6
        &#9474;   Parent Span ID: b2
        &#9474;   Start: 10:00:01.150
        &#9474;   End:   10:00:01.550
        &#9474;   Duration: 400 ms
        &#9474;   Attributes: gen_ai.tool.name=retrieve_report
        &#9474;   Status: UNSET
        &#9474;   Events: None
        &#9474;   &#9474;
        &#9474;   &#9492;&#9472;&#9472; SPAN 7: HTTP: Document API
        &#9474;       Span ID: g7
        &#9474;       Parent Span ID: f6
        &#9474;       Start: 10:00:01.200
        &#9474;       End:   10:00:01.500
        &#9474;       Duration: 300 ms
        &#9474;       Attributes: http.response.status_code=200
        &#9474;       Status: UNSET
        &#9474;       Events: None
        &#9474;
        &#9492;&#9472;&#9472; SPAN 8: LLM: Generate Summary
            Span ID: h8
            Parent Span ID: b2
            Start: 10:00:01.600
            End:   10:00:02.300
            Duration: 700 ms
            Attributes: gen_ai.request.model=gpt-6
            Status: UNSET
            Events: None</code></code></pre><p>Each operation has its own span, while the spans share a common trace ID.</p><p>If document retrieval fails, you inspect the corresponding span to identify the error, execution duration, and affected service.</p><p>For long-running workflows, a single task could span multiple traces. In such cases, a separate task or workflow identifier correlates executions across trace boundaries.</p><h3>3. OpenTelemetry Attributes for AI Agents</h3><p>Trace identifiers show how operations connect.</p><p>Span attributes describe what happened during each operation.</p><p>For agentic AI systems, here are the categories of useful attributes:</p><ul><li><p><strong>Agent execution:</strong> Agent name, operation type, execution status, and workflow identifiers.</p></li><li><p><strong>Model invocation:</strong> Model provider, model name, token usage, and inference-related information.</p></li><li><p><strong>Tool execution:</strong> Tool name, operation details, execution status, and errors.</p></li><li><p><strong>Performance:</strong> Operation duration, request latency, and other performance measurements.</p></li><li><p><strong>Governance:</strong> Guardrail outcomes, policy decisions, and relevant audit references.</p></li></ul><p>OpenTelemetry provides Generative AI semantic conventions for several model, agent, and tool operations.</p><p>Additional application-specific attributes should be necessary for workflow state, governance, and cost attribution.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!PaRo!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5a5f1bb-d027-43ae-ad8e-759da5f35eea_753x351.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!PaRo!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5a5f1bb-d027-43ae-ad8e-759da5f35eea_753x351.png 424w, https://substackcdn.com/image/fetch/$s_!PaRo!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5a5f1bb-d027-43ae-ad8e-759da5f35eea_753x351.png 848w, https://substackcdn.com/image/fetch/$s_!PaRo!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5a5f1bb-d027-43ae-ad8e-759da5f35eea_753x351.png 1272w, https://substackcdn.com/image/fetch/$s_!PaRo!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5a5f1bb-d027-43ae-ad8e-759da5f35eea_753x351.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!PaRo!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5a5f1bb-d027-43ae-ad8e-759da5f35eea_753x351.png" width="753" height="351" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/a5a5f1bb-d027-43ae-ad8e-759da5f35eea_753x351.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:351,&quot;width&quot;:753,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:&quot;&quot;,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!PaRo!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5a5f1bb-d027-43ae-ad8e-759da5f35eea_753x351.png 424w, https://substackcdn.com/image/fetch/$s_!PaRo!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5a5f1bb-d027-43ae-ad8e-759da5f35eea_753x351.png 848w, https://substackcdn.com/image/fetch/$s_!PaRo!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5a5f1bb-d027-43ae-ad8e-759da5f35eea_753x351.png 1272w, https://substackcdn.com/image/fetch/$s_!PaRo!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5a5f1bb-d027-43ae-ad8e-759da5f35eea_753x351.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">OpenTelemetry Attributes for Agentic AI</figcaption></figure></div><p>Use standardized OpenTelemetry attributes wherever applicable.</p><p>Define custom attributes only when existing conventions do not cover required information.</p><h3>4. Trace Correlation and Auditability</h3><p>Once you collect the relevant spans and attributes, you can reconstruct an AI task's execution history.</p><p>A trace reveals:</p><ol><li><p>Which agent received the original request.</p></li><li><p>Which model generated the execution plan.</p></li><li><p>Where retries, errors, or latency spikes occurred.</p></li><li><p>Which tools were invoked and whether they succeeded.</p></li><li><p>How many tokens were consumed during model invocations.</p></li></ol><p>This information lets you identify failures, analyze resource usage, and investigate unexpected agent behavior.</p><p>Yet telemetry alone does not guarantee a complete or tamper-proof audit trail. Production systems need proper retention policies, access controls, and integrity protections.</p><p>Plus, you must avoid recording sensitive prompts, credentials, or personal information in telemetry unless explicitly required and properly protected.</p><p>The next section explores how to turn agent traces and logs into evaluation datasets to measure task success, response quality, and tool utilization.</p><div><hr></div><h2><strong>Agentic Evaluation and Optimization</strong></h2><p>Observability tells you what happened during an agent&#8217;s execution.</p><p>Evaluation determines whether the agent performed its task correctly.</p><p>A research agent could successfully call every tool and generate a response within two seconds. Yet the response could contain incorrect financial figures or miss crucial information from retrieved documents.</p><p>Traditional performance metrics, such as latency and error rates, would not reveal these quality issues. So you need agent-specific evaluations to measure task success, response quality, and tool usage.</p><p>Let&#8217;s examine key metrics:</p><h3>1. Agent Efficiency</h3><p>Agent efficiency measures how effectively an agent completes its assigned task.</p><p>Some metrics evaluate the final response, while others assess intermediate actions and execution behavior.</p><h4><strong>Task and reasoning quality</strong></h4><ul><li><p><strong>Task success rate:</strong> Measures the percentage of tasks completed according to predefined success criteria. This is one of the most important indicators of agent effectiveness.</p></li><li><p><strong>Reasoning relevance:</strong> Evaluates whether the agent&#8217;s observable plans and actions align with the user&#8217;s request.</p></li><li><p><strong>Reasoning coherence:</strong> Asses whether agent&#8217;s observable planning steps follow a logical sequence without unnecessary or contradictory actions.</p></li><li><p><strong>Task decomposition efficiency:</strong> Measures how effectively an agent divides complex tasks into manageable subtasks, considering dependencies and unnecessary steps.</p></li></ul><p>Reasoning-based metrics require access to explicit plans or other observable reasoning artifacts.</p><p>Many &#8220;hosted&#8221; models do not expose their internal reasoning, so evaluations should focus on available execution evidence.</p><h4><strong>Response quality</strong></h4><ul><li><p><strong>Answer relevance:</strong> Measures whether response address user&#8217;s original request.</p></li><li><p><strong>Groundedness:</strong> Evaluates whether claims in the response are supported by the retrieved documents or trusted evidence.</p></li><li><p><strong>Response fluency:</strong> Assesses grammar, readability, and natural language quality.</p></li><li><p><strong>Response coherence:</strong> Measures whether the response is logically structured and internally consistent.</p></li></ul><p>An agent could produce a fluent financial summary while reporting incorrect revenue figures. Its fluency score might be high, but its groundedness and factual accuracy scores would be low.</p><h4><strong>Reliability and consistency</strong></h4><ul><li><p><strong>Agent robustness:</strong> Measures how reliably an agent handles unexpected inputs, tool failures, and adversarial scenarios.</p></li><li><p><strong>Agent consistency:</strong> Evaluates whether repeated executions with similar inputs produce results that satisfy the same correctness criteria.</p></li></ul><p>These metrics identify cases where an agent appears successful but produces unreliable outcomes.</p><h3>2. Tool Utilization Efficiency</h3><p>AI agents depend on external tools to retrieve information and perform actions.</p><p>Choosing the wrong tool, passing invalid parameters, or making unnecessary calls increases latency and cost.</p><p>Here are four metrics to evaluate tool usage:</p><ul><li><p><strong>Tool selection accuracy:</strong> Measures how often the agent selects an appropriate tool for the task.</p></li><li><p><strong>Tool usage efficiency:</strong> Evaluates whether the agent completes its work with a reasonable number of tool calls and minimal unnecessary operations.</p></li><li><p><strong>Tool call precision:</strong> Measures whether the arguments passed to a tool are valid and appropriate for the requested operation.</p></li><li><p><strong>Tool call success rate:</strong> Measures the percentage of tool invocations that complete successfully.</p></li></ul><p>Imagine a research agent that needs to retrieve a company&#8217;s annual report.</p><p>The agent could invoke the document search tool five times with different queries before retrieving the correct report.</p><p>All five calls might succeed technically, resulting in a 100% tool call success rate. Yet the agent&#8217;s tool usage efficiency would be poor because several calls were unnecessary.</p><p>To measure these metrics accurately, you need evaluation criteria, expected outcomes, or reference examples. Execution logs alone don't show whether a tool was the best choice.</p><h3>3. OpenTelemetry-Based Evaluation Architecture</h3><p>Once you define the evaluation metrics, the next step is to collect execution data and evaluate agent behavior.</p><p>OpenTelemetry provides traces, logs, and metrics to reconstruct agent executions across models, tools, and external services.</p><p>These telemetry signals could support two evaluation approaches:</p><h4><strong>Offline evaluation</strong></h4><p>Offline evaluation analyzes previously recorded executions or runs agents against predefined test datasets.</p><p>An evaluation service retrieves relevant traces and logs, reconstructs the execution history, and calculates quality metrics.</p><p>This approach supports:</p><ul><li><p>Analysis of failed production interactions.</p></li><li><p>Regression testing after prompt/model changes.</p></li><li><p>Identification of recurring tool usage or response quality issues.</p></li><li><p>Comparison of agent versions using the same evaluation dataset.</p></li></ul><p>Offline evaluation does not require evaluation service to run inside live request path.</p><h4><strong>Online evaluation</strong></h4><p>Online evaluation assesses agent behavior during production execution.</p><p>The application or telemetry pipeline sends relevant execution artifacts to an evaluation service. The service then calculates selected metrics and records the results.</p><p>You could evaluate a sample of production responses for groundedness and flag low-scoring interactions for further investigation.</p><p>Online evaluations run synchronously when immediate validation is required or asynchronously to avoid increasing user-facing latency.</p><p>Both approaches rely on evaluation criteria and access to necessary execution artifacts.</p><h3>4. Offline Evaluation Architecture</h3><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!7UOG!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3591f031-d6e3-479a-83ee-8ab01ed50153_1450x646.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!7UOG!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3591f031-d6e3-479a-83ee-8ab01ed50153_1450x646.png 424w, https://substackcdn.com/image/fetch/$s_!7UOG!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3591f031-d6e3-479a-83ee-8ab01ed50153_1450x646.png 848w, https://substackcdn.com/image/fetch/$s_!7UOG!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3591f031-d6e3-479a-83ee-8ab01ed50153_1450x646.png 1272w, https://substackcdn.com/image/fetch/$s_!7UOG!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3591f031-d6e3-479a-83ee-8ab01ed50153_1450x646.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!7UOG!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3591f031-d6e3-479a-83ee-8ab01ed50153_1450x646.png" width="1450" height="646" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/3591f031-d6e3-479a-83ee-8ab01ed50153_1450x646.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:646,&quot;width&quot;:1450,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:&quot;&quot;,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!7UOG!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3591f031-d6e3-479a-83ee-8ab01ed50153_1450x646.png 424w, https://substackcdn.com/image/fetch/$s_!7UOG!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3591f031-d6e3-479a-83ee-8ab01ed50153_1450x646.png 848w, https://substackcdn.com/image/fetch/$s_!7UOG!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3591f031-d6e3-479a-83ee-8ab01ed50153_1450x646.png 1272w, https://substackcdn.com/image/fetch/$s_!7UOG!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3591f031-d6e3-479a-83ee-8ab01ed50153_1450x646.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Fig. 5: Solution Architecture for OTel-Based Offline Evaluation</figcaption></figure></div><p>The evaluation process follows four steps:</p><h4>Step 1: Trigger Evaluation</h4><p>A user or automated workflow initiates an evaluation through the Evaluate API.</p><p>The request specifies the evaluation scope:</p><ul><li><p>Agent name or identifier.</p></li><li><p>Tool descriptions or relevant tool metadata.</p></li><li><p>Date range for retrieving previous interactions.</p></li><li><p>Selected evaluation metrics, such as tool selection accuracy or answer relevance.</p></li></ul><p>Depending on the selected metrics, the evaluator would require reference answers, expected tool calls, or ground-truth data.</p><h4>Step 2: Retrieve Agent Interactions</h4><p>The evaluation service queries the telemetry backend to retrieve agent executions.</p><p>OpenTelemetry trace identifiers correlate operations within each execution, including model invocations, tool calls, and errors.</p><p>The evaluator reconstructs the available execution history from these records. It retrieves the model response, selected tool, tool arguments, and execution outcome for a specific research task.</p><p>The required data must have been captured during execution. Standard telemetry does not automatically contain every prompt, response, or evaluation artifact.</p><h4>Step 3: Evaluate Agent Performance</h4><p>The evaluation engine analyzes retrieved interactions using selected evaluation methods.</p><p>One approach is <strong>LLM-as-a-Judge</strong>, where a language model evaluates an agent&#8217;s output against predefined criteria.</p><p>The evaluator assesses whether a generated financial summary accurately reflects the retrieved annual report. Other metrics could use deterministic checks, reference answers, or direct comparisons with expected tool calls.</p><p>The evaluation engine then generates:</p><ul><li><p>A score for each selected metric.</p></li><li><p>Supporting evidence for the result.</p></li><li><p>An overall evaluation result, where applicable.</p></li></ul><p>The results are stored in persistent storage, such as object or Blob storage, for later analysis.</p><p>LLM-based scores should be validated against human-reviewed examples to assess their reliability.</p><h4>Step 4: Publish Evaluation Results</h4><p>Users retrieve evaluation results through these APIs:</p><ul><li><p><strong>Get ResultList API:</strong> Returns a summary of evaluation runs, including their status and aggregate scores.</p></li><li><p><strong>Get ResultDetails API:</strong> Returns detailed scores, supporting evidence, and explanations for individual interactions and metrics.</p></li></ul><p>These results let you compare agent versions, investigate failures, and identify improvement opportunities.</p><p>If a new prompt increases task success but doubles the average number of tool calls, you can compare the quality improvement against the added cost.</p><p>Once you understand how effectively an agent performs its tasks, the next question is how much each execution costs.</p><p>This brings us to FinOps for agentic AI systems.</p><div><hr></div><h2><strong>OpenTelemetry</strong>-<strong>based Agentic AI FinOps</strong></h2><p>Running AI agents in production creates costs beyond individual model requests.</p><p>An agent might invoke a language model several times, retrieve documents from a vector database, call external APIs, and delegate tasks to other agents before producing a final response.</p><p>Each operation consumes resources.</p><p>FinOps lets you understand these costs and make informed decisions about infrastructure, model selection, and resource allocation.</p><p>For agentic AI systems, the goal is to measure the cost of each execution and optimize spending without compromising task quality or reliability.</p><p>OpenTelemetry (<strong>OTel</strong>) provides execution data that links resource consumption to individual agent workflows.</p><p>Let&#8217;s examine the three crucial dimensions of agentic AI costs:</p>
      <p>
          <a href="https://newsletter.systemdesign.one/p/ai-agent-observability-opentelemetry">
              Read more
          </a>
      </p>
   ]]></content:encoded></item><item><title><![CDATA[Infrastructure as Code - A Deep Dive]]></title><description><![CDATA[#202: Part 6 - DevOps Mastermind]]></description><link>https://newsletter.systemdesign.one/p/how-terraform-works-internally</link><guid isPermaLink="false">https://newsletter.systemdesign.one/p/how-terraform-works-internally</guid><dc:creator><![CDATA[Ayaan]]></dc:creator><pubDate>Thu, 12 Nov 2026 12:49:22 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/241907ef-cfcc-4c24-9d70-4a8281f4b3b6_1280x720.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p></p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption"><span data-color="#3a7bf9" style="color: rgb(58, 123, 249);">Get my system design playbook for FREE on newsletter signup:</span></p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><ul><li><p><em><a href="https://newsletter.systemdesign.one/p/how-terraform-works-internally/?action=share">Share this letter</a> &amp; I&#8217;ll send you some rewards for the referrals.</em></p></li></ul><div><hr></div><p>We&#8217;ve all created infrastructure manually at some point.</p><p>A few clicks in a cloud console, some configuration changes, and our servers, networks &amp; databases are up and running.</p><p>Sounds simple enough, right?</p><p>But what if you need to recreate the same infrastructure for another environment? You&#8217;ll have to go through all those settings &amp; reconfigure the resources. Then you&#8217;ll need to set up their dependencies and make sure that everything matches the original environment.</p><p>And if you miss even one setting, you could spend hours figuring out why something works in staging but fails in production.</p><p>Now imagine managing hundreds of resources across multiple environments&#8230;</p><p>Different engineers are changing these resources regularly. Keeping track of everything manually would become a headache&#8230;!</p><p>You could use scripts to automate some of this work.</p><p>But scripts need to handle resources already running. They must account for changes to existing configurations and failures along the way.</p><p>Wouldn&#8217;t it be easier if you could write down how our infrastructure should look? You could keep it in Git and let a tool figure out what needs to change whenever you update it.</p><p>Well, this is the basic idea behind Infrastructure as Code (<strong>IaC</strong>)&#8230; And Terraform is one of the most widely used tools for managing infrastructure this way.</p><p>With Terraform, you define infrastructure using configuration files.</p><p>You preview changes before applying them and manage existing resources without repeating every step manually.</p><p>But have you ever wondered how Terraform knows which resources already exist? And how it decides what needs updating and what should be created first?</p><p>Onward.</p><h4 style="text-align: center;"><strong>&#167;</strong></h4><div class="callout-block" data-callout="true"><h2><strong><a href="https://prlens.dev/?utm_source=neokim&amp;utm_medium=newsletter&amp;utm_campaign=ad1">Mermaid diagrams are the floor</a></strong></h2><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://prlens.dev/?utm_source=neokim&amp;utm_medium=newsletter&amp;utm_campaign=ad1" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!8piF!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbed88e8-d685-4a85-8795-4805c8781f1a_1200x628.gif 424w, https://substackcdn.com/image/fetch/$s_!8piF!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbed88e8-d685-4a85-8795-4805c8781f1a_1200x628.gif 848w, https://substackcdn.com/image/fetch/$s_!8piF!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbed88e8-d685-4a85-8795-4805c8781f1a_1200x628.gif 1272w, https://substackcdn.com/image/fetch/$s_!8piF!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbed88e8-d685-4a85-8795-4805c8781f1a_1200x628.gif 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!8piF!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbed88e8-d685-4a85-8795-4805c8781f1a_1200x628.gif" width="1200" height="628" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/bbed88e8-d685-4a85-8795-4805c8781f1a_1200x628.gif&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:628,&quot;width&quot;:1200,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:1202568,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:&quot;&quot;,&quot;type&quot;:&quot;image/gif&quot;,&quot;href&quot;:&quot;https://prlens.dev/?utm_source=neokim&amp;utm_medium=newsletter&amp;utm_campaign=ad1&quot;,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://newsletter.systemdesign.one/i/222466719?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbed88e8-d685-4a85-8795-4805c8781f1a_1200x628.gif&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!8piF!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbed88e8-d685-4a85-8795-4805c8781f1a_1200x628.gif 424w, https://substackcdn.com/image/fetch/$s_!8piF!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbed88e8-d685-4a85-8795-4805c8781f1a_1200x628.gif 848w, https://substackcdn.com/image/fetch/$s_!8piF!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbed88e8-d685-4a85-8795-4805c8781f1a_1200x628.gif 1272w, https://substackcdn.com/image/fetch/$s_!8piF!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbed88e8-d685-4a85-8795-4805c8781f1a_1200x628.gif 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Your coding agent can already paste a Mermaid chart into a PR description.</p><p>You still read every file to learn what it does to your system.</p><p><strong><a href="https://prlens.dev/?utm_source=neokim&amp;utm_medium=newsletter&amp;utm_campaign=ad1">PR Lens</a></strong> draws each pull request as a live architecture canvas, right in your pull request. Press play and it walks you through the change, one step at a time.</p><p>On every PR, you see:</p><ul><li><p><strong>What changed</strong>, marked on the parts of the system it touches</p></li><li><p><strong>What else it reaches</strong>, including services the diff never names</p></li><li><p><strong>How a request moves</strong> through the system now</p></li><li><p><strong>What could break</strong>, flagged before you merge</p></li></ul><p>2,200+ GitHub stars and 45,000+ pull requests drawn.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://prlens.dev/?utm_source=neokim&amp;utm_medium=newsletter&amp;utm_campaign=ad1&quot;,&quot;text&quot;:&quot;Add it to your repo&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://prlens.dev/?utm_source=neokim&amp;utm_medium=newsletter&amp;utm_campaign=ad1"><span>Add it to your repo</span></a></p><p>(Thanks to <a href="https://prlens.dev/?utm_source=neokim&amp;utm_medium=newsletter&amp;utm_campaign=ad1">Coldtea</a> for partnering on this newsletter.)</p></div><h4 style="text-align: center;"><strong>&#167;</strong></h4><p>I want to introduce <strong><a href="https://x.com/twtayaan">Ayaan</a></strong> as a guest author.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://x.com/twtayaan" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!GCV8!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d442f35-0ac7-4552-bd87-66962939694d_2048x1024.png 424w, https://substackcdn.com/image/fetch/$s_!GCV8!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d442f35-0ac7-4552-bd87-66962939694d_2048x1024.png 848w, https://substackcdn.com/image/fetch/$s_!GCV8!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d442f35-0ac7-4552-bd87-66962939694d_2048x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!GCV8!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d442f35-0ac7-4552-bd87-66962939694d_2048x1024.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!GCV8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d442f35-0ac7-4552-bd87-66962939694d_2048x1024.png" width="1456" height="728" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/5d442f35-0ac7-4552-bd87-66962939694d_2048x1024.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:728,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:&quot;&quot;,&quot;type&quot;:null,&quot;href&quot;:&quot;https://x.com/twtayaan&quot;,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!GCV8!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d442f35-0ac7-4552-bd87-66962939694d_2048x1024.png 424w, https://substackcdn.com/image/fetch/$s_!GCV8!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d442f35-0ac7-4552-bd87-66962939694d_2048x1024.png 848w, https://substackcdn.com/image/fetch/$s_!GCV8!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d442f35-0ac7-4552-bd87-66962939694d_2048x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!GCV8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d442f35-0ac7-4552-bd87-66962939694d_2048x1024.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>He&#8217;s a <strong>Certified Kubernetes Administrator (CKA)</strong> who writes about Kubernetes, cloud infrastructure, and cloud-native systems. He builds in public and shares practical tips, projects, and lessons learned along the way.</p><p>Check out his work &amp; socials:</p><ul><li><p><a href="https://x.com/twtayaan">Twitter</a></p></li><li><p><a href="https://github.com/Ayaan49">Github</a></p></li></ul><h4 style="text-align: center;"><strong>&#167;</strong></h4><p><em>Inside this newsletter:</em></p><ul><li><p><strong>How Terraform turns a few lines of configuration into real cloud infrastructure, and what happens behind the scenes.</strong></p></li><li><p>The architecture behind Terraform: how Terraform Core, providers, and backends work together to manage infrastructure.</p></li><li><p><strong>How Terraform knows which resources already exist, what needs to change, and which resources to create first.</strong></p></li><li><p>How to stop managing infrastructure manually and start using state, dependency graphs, and reusable modules to manage changes.</p></li><li><p><strong>What really happens when you run </strong><code>terraform init</code><strong>, </strong><code>plan</code><strong>, and </strong><code>apply</code><strong>, and why a failed deployment doesn&#8217;t automatically roll back.</strong></p></li><li><p>How to detect configuration drift, import existing infrastructure, and prevent unexpected resource replacements.</p></li><li><p><strong>How to safely manage Terraform in production with CI/CD pipelines, state locking, plan reviews, and access controls.</strong></p></li></ul><p><em>Let&#8217;s keep going!</em></p><h4 style="text-align: center;"><strong>&#167;</strong></h4><h2>What Is Infrastructure as Code</h2><p>IaC is a way to create &amp; manage infrastructure through configuration files.</p><p>Instead of setting up resources manually, you define them in files and use those files to manage the infrastructure.</p><p>A file could describe a network, a subnet within it, a server connected to the subnet, and the rules that control access. These details stay in Git, making changes visible before they reach the cloud. The configuration remains useful after deployment. It describes what should continue to exist.</p><p>There are two main approaches to managing infrastructure through code: <em>imperative and declarative</em>&#8230;</p><p>The difference lies in how you tell the system what to do.</p><h3>Imperative vs Declarative Infrastructure</h3><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!g1Rq!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd97929be-bdf4-4fab-8c00-e133545d5456_1536x1024.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!g1Rq!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd97929be-bdf4-4fab-8c00-e133545d5456_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!g1Rq!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd97929be-bdf4-4fab-8c00-e133545d5456_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!g1Rq!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd97929be-bdf4-4fab-8c00-e133545d5456_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!g1Rq!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd97929be-bdf4-4fab-8c00-e133545d5456_1536x1024.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!g1Rq!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd97929be-bdf4-4fab-8c00-e133545d5456_1536x1024.png" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/d97929be-bdf4-4fab-8c00-e133545d5456_1536x1024.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!g1Rq!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd97929be-bdf4-4fab-8c00-e133545d5456_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!g1Rq!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd97929be-bdf4-4fab-8c00-e133545d5456_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!g1Rq!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd97929be-bdf4-4fab-8c00-e133545d5456_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!g1Rq!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd97929be-bdf4-4fab-8c00-e133545d5456_1536x1024.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>The <strong>imperative approach</strong> focuses on <em>how</em> to create infrastructure:</p><p>You write a series of instructions explaining each step &amp; order in which they should run. A script could first create a network, save its ID, create a subnet inside it, and finally launch a server&#8230; The script follows these instructions in sequence.</p><p>But what happens if you run the same script again?</p><p>It needs extra checks to avoid creating duplicate resources. You also need to handle situations where some steps succeed while others fail.</p><p>The <strong>declarative approach</strong> works differently:</p><p>Instead of writing instructions for every step, you describe <em>what</em> the final infrastructure should look like.</p><p>Consider a setup where you define a network, a subnet within it, and a server connected to that subnet. Terraform uses these relationships to determine the execution order. Plus, it checks the existing infrastructure to determine which changes are necessary.</p><p>The main difference is simple:</p><p>Imperative tells the system how to do something, while declarative describes the desired final state.</p><h3>What Infrastructure as Code Changes</h3><p>Once the infrastructure is defined in files, changes start with a pull request instead of a series of console clicks.</p><p>You review &amp; check the proposed edit before it reaches production. Git keeps a history of the configuration. The same files also help recreate development, staging, or replacement environments with different values.</p><p>Mistakes are still possible&#8230;</p><p>Someone could change resources directly in the cloud console. Also they could approve a configuration with incorrect settings. But the written definition provides a clear reference. It lets you compare intended infrastructure with what actually exists.</p><p>Terraform uses this description to plan which resources to create, update, or remove.</p><p>Now, let&#8217;s look at how Terraform uses this approach to manage real infrastructure&#8230;</p><h4 style="text-align: center;"><strong>&#167;</strong></h4><h2>Terraform as an Infrastructure as Code Tool</h2><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!cfP8!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc2f2f727-f034-414b-a180-9579c9aefc1b_1774x887.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!cfP8!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc2f2f727-f034-414b-a180-9579c9aefc1b_1774x887.png 424w, https://substackcdn.com/image/fetch/$s_!cfP8!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc2f2f727-f034-414b-a180-9579c9aefc1b_1774x887.png 848w, https://substackcdn.com/image/fetch/$s_!cfP8!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc2f2f727-f034-414b-a180-9579c9aefc1b_1774x887.png 1272w, https://substackcdn.com/image/fetch/$s_!cfP8!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc2f2f727-f034-414b-a180-9579c9aefc1b_1774x887.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!cfP8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc2f2f727-f034-414b-a180-9579c9aefc1b_1774x887.png" width="1456" height="728" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/c2f2f727-f034-414b-a180-9579c9aefc1b_1774x887.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:728,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!cfP8!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc2f2f727-f034-414b-a180-9579c9aefc1b_1774x887.png 424w, https://substackcdn.com/image/fetch/$s_!cfP8!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc2f2f727-f034-414b-a180-9579c9aefc1b_1774x887.png 848w, https://substackcdn.com/image/fetch/$s_!cfP8!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc2f2f727-f034-414b-a180-9579c9aefc1b_1774x887.png 1272w, https://substackcdn.com/image/fetch/$s_!cfP8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc2f2f727-f034-414b-a180-9579c9aefc1b_1774x887.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Terraform uses a declarative approach:</p><p>It helps manage infrastructure across different cloud platforms &amp; services. The same workflow works for networks, databases, and DNS records. It works even when these resources belong to different platforms.</p><p>You define the infrastructure using HashiCorp Configuration Language (<strong>HCL</strong><a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-1" href="#footnote-1" target="_self">1</a>). Terraform reads these files to understand which resources should exist and how they connect.</p><p>Providers then handle platform-specific operations.</p><h3>Terraform Configuration</h3><p>A Terraform configuration consists of plain text <code>.tf files</code>.</p><p>Terraform reads all files in the same working directory as one configuration.</p><p>You could keep network resources in one file and servers in another to organize the setup. The file structure makes the configuration easier to manage. But it does not control the order in which Terraform creates resources.</p><p>Instead, Terraform follows the relationships between resources.</p><p>A server could reference a subnet defined in another file. You don&#8217;t need to copy its ID manually. Also HCL supports expressions &amp; conditional values for configurations that need different settings.</p><h3>The Core Building Blocks</h3><p>Resources are Terraform's main building blocks.</p><p>Each resource represents an object Terraform manages, such as a virtual machine or DNS record.</p><p>Providers connect Terraform to the platforms where these objects exist. They define which resource types are available. They also handle operations such as creating, updating, and/or deleting those resources.</p><p>Other components serve different purposes&#8230;Data sources read information about existing infrastructure without taking ownership of those objects.</p><p>Variables let you pass values into a configuration, while locals help reuse values within it. Outputs expose useful information, such as a server&#8217;s IP address. Modules group related resources into reusable configurations. i.e., you don&#8217;t have to repeat the same setup.</p><p>Terraform also provides meta-arguments<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-2" href="#footnote-2" target="_self">2</a> to control how resources get created and connected.</p><p><code>count</code> and <code>for_each</code> create multiple instances of a resource. Meanwhile, <code>depends_on</code> defines dependencies Terraform cannot identify through references.</p><h3>Configuration, State, and Real Infrastructure</h3><p>A resource block alone does not give Terraform enough information to manage infrastructure&#8230; The configuration describes what should exist.</p><p>But Terraform also needs to know which real objects it already manages.</p><p>The state tracks which resource addresses in the configuration belong to which objects on the remote platform. Providers read those objects to determine their current condition.</p><p>Terraform uses this information to decide what happens to each resource. It determines whether to create, update, replace, remove, or leave it unchanged.</p><p>NOTE: Terraform only checks infrastructure when you run a command. It does not automatically detect &amp; fix every change made outside its configuration.</p><p>Let&#8217;s look at the components responsible for this process&#8230;</p><h4 style="text-align: center;"><strong>&#167;</strong></h4><h2>Terraform Architecture</h2><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!-VW2!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8476d5a8-e6df-462a-8790-589b0447e1d8_1774x887.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!-VW2!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8476d5a8-e6df-462a-8790-589b0447e1d8_1774x887.png 424w, https://substackcdn.com/image/fetch/$s_!-VW2!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8476d5a8-e6df-462a-8790-589b0447e1d8_1774x887.png 848w, https://substackcdn.com/image/fetch/$s_!-VW2!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8476d5a8-e6df-462a-8790-589b0447e1d8_1774x887.png 1272w, https://substackcdn.com/image/fetch/$s_!-VW2!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8476d5a8-e6df-462a-8790-589b0447e1d8_1774x887.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!-VW2!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8476d5a8-e6df-462a-8790-589b0447e1d8_1774x887.png" width="1456" height="728" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/8476d5a8-e6df-462a-8790-589b0447e1d8_1774x887.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:728,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!-VW2!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8476d5a8-e6df-462a-8790-589b0447e1d8_1774x887.png 424w, https://substackcdn.com/image/fetch/$s_!-VW2!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8476d5a8-e6df-462a-8790-589b0447e1d8_1774x887.png 848w, https://substackcdn.com/image/fetch/$s_!-VW2!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8476d5a8-e6df-462a-8790-589b0447e1d8_1774x887.png 1272w, https://substackcdn.com/image/fetch/$s_!-VW2!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8476d5a8-e6df-462a-8790-589b0447e1d8_1774x887.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Terraform needs more than a configuration file to manage infrastructure.</p><p>It needs an engine to plan the work &amp; plugins to communicate with each platform. Plus, it needs a place to keep its state between runs.</p><p>These responsibilities belong to Terraform Core, provider plugins &amp; backend.</p><h3>Terraform Core</h3><p>Terraform Core is the engine behind the terraform command.</p><p>It reads configuration, evaluates variables &amp; expressions, and loads the existing state. Providers also supply Core with schemas. These describe available resources &amp; settings each resource supports.</p><p>With this information, Core determines how resources depend on each other.</p><p>Take a subnet as an example: It needs its network to exist first. Meanwhile, Terraform can create unrelated resources at the same time.</p><p>Using these relationships, Terraform builds a dependency graph<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-3" href="#footnote-3" target="_self">3</a> and creates an execution plan. During <code>terraform apply</code>, Terraform Core coordinates the operations and records their results in state as they succeed.</p><h3>Provider Plugins</h3><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!k-4t!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F601122e5-0a1f-456d-beee-edc6d223980d_1774x887.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!k-4t!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F601122e5-0a1f-456d-beee-edc6d223980d_1774x887.png 424w, https://substackcdn.com/image/fetch/$s_!k-4t!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F601122e5-0a1f-456d-beee-edc6d223980d_1774x887.png 848w, https://substackcdn.com/image/fetch/$s_!k-4t!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F601122e5-0a1f-456d-beee-edc6d223980d_1774x887.png 1272w, https://substackcdn.com/image/fetch/$s_!k-4t!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F601122e5-0a1f-456d-beee-edc6d223980d_1774x887.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!k-4t!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F601122e5-0a1f-456d-beee-edc6d223980d_1774x887.png" width="1456" height="728" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/601122e5-0a1f-456d-beee-edc6d223980d_1774x887.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:728,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!k-4t!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F601122e5-0a1f-456d-beee-edc6d223980d_1774x887.png 424w, https://substackcdn.com/image/fetch/$s_!k-4t!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F601122e5-0a1f-456d-beee-edc6d223980d_1774x887.png 848w, https://substackcdn.com/image/fetch/$s_!k-4t!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F601122e5-0a1f-456d-beee-edc6d223980d_1774x887.png 1272w, https://substackcdn.com/image/fetch/$s_!k-4t!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F601122e5-0a1f-456d-beee-edc6d223980d_1774x887.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Terraform Core follows the same general process across different platforms.</p><p>But each platform has its own APIs &amp; resource types&#8230;</p><p>Provider plugins handle these differences.</p><p>Providers run as separate processes. They communicate with Terraform Core through a plugin protocol based on gRPC<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-4" href="#footnote-4" target="_self">4</a>. This allows Core to request information &amp; operations without knowing each platform&#8217;s API.</p><p>Each provider defines the resources &amp; data sources available for its platform, along with the settings it supports.</p><p>During planning, providers help Terraform check existing resources. This determines whether they need an update/replacement. When you run <code>terraform apply</code>, they communicate with the remote APIs to create, read, update, and/or delete resources.</p><p>This way, Terraform Core manages the overall execution. Providers handle platform-specific operations.</p><h3>Backends</h3><p>Terraform needs to remember which resources it manages after a run ends.</p><p>The backend determines where Terraform stores this state and how it accesses it.</p><p>By default, the local backend saves state on the machine running Terraform. For shared infrastructure, a remote backend allows many engineers and CI/CD pipelines to access the same state.</p><p>Some backends also support state locking<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-5" href="#footnote-5" target="_self">5</a>. It ensures that only one Terraform run can modify the state at a time.</p><p>Features such as encryption, backups &amp; version history depend on the storage service.</p><p>But storing state remotely doesn&#8217;t mean Terraform runs remotely. The command could still run on a laptop or in a pipeline unless you configure remote execution separately.</p><p>So providers communicate with the actual infrastructure. Meanwhile, the backend stores the state Terraform uses to track those resources.</p><p>To understand this better, let&#8217;s look at why Terraform needs state and how it uses it to keep track of infrastructure...</p><h4 style="text-align: center;"><strong>&#167;</strong></h4><h2>Why Terraform Needs State</h2><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!imhn!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fef07fc7f-f524-492d-b83f-4383e6ea12b0_1774x887.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!imhn!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fef07fc7f-f524-492d-b83f-4383e6ea12b0_1774x887.png 424w, https://substackcdn.com/image/fetch/$s_!imhn!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fef07fc7f-f524-492d-b83f-4383e6ea12b0_1774x887.png 848w, https://substackcdn.com/image/fetch/$s_!imhn!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fef07fc7f-f524-492d-b83f-4383e6ea12b0_1774x887.png 1272w, https://substackcdn.com/image/fetch/$s_!imhn!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fef07fc7f-f524-492d-b83f-4383e6ea12b0_1774x887.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!imhn!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fef07fc7f-f524-492d-b83f-4383e6ea12b0_1774x887.png" width="1456" height="728" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/ef07fc7f-f524-492d-b83f-4383e6ea12b0_1774x887.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:728,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!imhn!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fef07fc7f-f524-492d-b83f-4383e6ea12b0_1774x887.png 424w, https://substackcdn.com/image/fetch/$s_!imhn!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fef07fc7f-f524-492d-b83f-4383e6ea12b0_1774x887.png 848w, https://substackcdn.com/image/fetch/$s_!imhn!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fef07fc7f-f524-492d-b83f-4383e6ea12b0_1774x887.png 1272w, https://substackcdn.com/image/fetch/$s_!imhn!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fef07fc7f-f524-492d-b83f-4383e6ea12b0_1774x887.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Terraform needs to recognize the infrastructure from earlier runs.</p><p>This includes resources it created &amp; those it started managing.</p><p>To understand this, you could define an EC2 instance as <code>aws_instance.web</code> in our configuration. But AWS identifies the actual instance using a unique ID. Terraform stores this connection in state so it knows which EC2 instance belongs to the resource defined in our files.</p><h3>Resource Identity and State Data</h3><p>State stores more than resource IDs.</p><p>It contains attributes<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-6" href="#footnote-6" target="_self">6</a> returned by providers &amp; output values. Terraform uses this information to track resources inside modules. When you use <code>count</code> or <code>for_each</code> to create multiple resource instances, Terraform tracks each one separately.</p><p>This information becomes crucial when you modify the configuration.</p><p>Consider what happens when you remove a resource block. The actual resource is not immediately deleted from the cloud.</p><p>Terraform still has a state entry that identifies the remote object associated with the removed block. During <em>planning</em>, it uses this information to determine which resource to delete.</p><h3>Local State vs Remote State</h3><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!z10t!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2c49c416-2577-467e-b490-efe0ea86520b_1536x1024.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!z10t!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2c49c416-2577-467e-b490-efe0ea86520b_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!z10t!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2c49c416-2577-467e-b490-efe0ea86520b_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!z10t!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2c49c416-2577-467e-b490-efe0ea86520b_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!z10t!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2c49c416-2577-467e-b490-efe0ea86520b_1536x1024.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!z10t!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2c49c416-2577-467e-b490-efe0ea86520b_1536x1024.png" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/2c49c416-2577-467e-b490-efe0ea86520b_1536x1024.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!z10t!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2c49c416-2577-467e-b490-efe0ea86520b_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!z10t!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2c49c416-2577-467e-b490-efe0ea86520b_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!z10t!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2c49c416-2577-467e-b490-efe0ea86520b_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!z10t!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2c49c416-2577-467e-b490-efe0ea86520b_1536x1024.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>By default, Terraform stores state in a local file.</p><p>This works well for learning or managing a small project alone.</p><p>But things get more complicated when many engineers or CI/CD pipelines work on the same infrastructure. If everyone maintains a separate copy, one person could make changes without the others having the latest state.</p><p>A remote backend solves this problem by providing authorized Terraform runs with access to a shared state file. Depending on the storage service, it also offers better access control and recovery.</p><p>Moving state to a remote backend only changes where Terraform stores its records. The actual servers, networks, and other infrastructure remain where they are.</p><h3>State Locking and Security</h3><p>When many Terraform runs access same state, there&#8217;s a risk of conflicting changes.</p><p>To prevent this, supported backends provide state locking. Terraform locks the state when needed, which prevents other runs from modifying it at the same time.</p><p>State also needs proper security because it can contain sensitive information. Terraform  hides these values from command output, but they still remain in the state file.</p><p>For production environments, a few precautions are important:</p><ul><li><p>Restrict access to state files.</p></li><li><p>Keep state files &amp; saved plans out of Git.</p></li><li><p>Enable encryption, backups &amp; versioning in the storage service.</p></li><li><p>Avoid the force-unlocking state unless the original operation has ended.</p></li></ul><p>The <code>.terraform.lock.hcl</code> file is different from the state file.</p><p>It records the selected provider versions and checksums<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-7" href="#footnote-7" target="_self">7</a>, so it normally belongs in Git.</p><p>Along with tracking existing resources, Terraform needs to understand how they depend on each other. It uses these relationships to determine how to make changes.</p><h4 style="text-align: center;"><strong>&#167;</strong></h4><h2>How Terraform Understands Resource Dependencies</h2><p>Infrastructure resources often depend on one another.</p><p>A subnet needs a Virtual Private Cloud (<strong>VPC</strong>) to exist before it gets created, while other resources require specific permissions or settings to be in place.</p><p>Terraform identifies these relationships as dependencies. It then uses them to decide the order in which resources should be managed.</p><h3>Implicit Dependencies</h3><p>Most dependencies get identified through references in the configuration.</p><p>When creating a subnet inside a VPC, you need to specify which VPC it belongs to. Instead of manually entering the VPC ID, you reference <code>aws_vpc.main.id</code> in the subnet&#8217;s <code>vpc_id</code> setting.</p><p>Terraform uses this reference to understand which VPC the subnet belongs to. First, it creates the VPC &amp; then proceeds with the subnet.</p><p>The resources don&#8217;t need to be defined in the same <code>.tf file</code> either. Terraform follows their relationships regardless of how the configuration files are organized.</p><h3>Explicit Dependencies</h3><p>Not every dependency involves referencing another resource&#8217;s value.</p><p>An EC2 instance may need an IAM policy attachment completed before its application starts. Yet the instance configuration does not necessarily reference the attachment directly.</p><p>In such cases, you use <code>depends_on</code> to declare the dependency explicitly. Terraform then waits for the required operation to finish before proceeding.</p><p>You should avoid using <code>depends_on</code> unnecessarily&#8230;Resources could end up waiting for each other, even when Terraform could create them in parallel.</p><h3>Dependency Graph</h3><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!hdj4!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F107e87c4-351c-4fc0-860e-ee376f11088c_1774x887.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!hdj4!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F107e87c4-351c-4fc0-860e-ee376f11088c_1774x887.png 424w, https://substackcdn.com/image/fetch/$s_!hdj4!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F107e87c4-351c-4fc0-860e-ee376f11088c_1774x887.png 848w, https://substackcdn.com/image/fetch/$s_!hdj4!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F107e87c4-351c-4fc0-860e-ee376f11088c_1774x887.png 1272w, https://substackcdn.com/image/fetch/$s_!hdj4!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F107e87c4-351c-4fc0-860e-ee376f11088c_1774x887.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!hdj4!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F107e87c4-351c-4fc0-860e-ee376f11088c_1774x887.png" width="1456" height="728" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/107e87c4-351c-4fc0-860e-ee376f11088c_1774x887.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:728,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!hdj4!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F107e87c4-351c-4fc0-860e-ee376f11088c_1774x887.png 424w, https://substackcdn.com/image/fetch/$s_!hdj4!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F107e87c4-351c-4fc0-860e-ee376f11088c_1774x887.png 848w, https://substackcdn.com/image/fetch/$s_!hdj4!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F107e87c4-351c-4fc0-860e-ee376f11088c_1774x887.png 1272w, https://substackcdn.com/image/fetch/$s_!hdj4!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F107e87c4-351c-4fc0-860e-ee376f11088c_1774x887.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Terraform brings these relationships together in a dependency graph.</p><p>This helps it decide which resources to create first.</p><p>If a VPC must exist before a subnet is created, the VPC gets processed first. Meanwhile, independent resources could be created in parallel because they don&#8217;t rely on each other.</p><p>Yet a circular dependency creates a problem. If two resources depend on each other, neither has a valid starting point, so Terraform reports an error instead of proceeding.</p><p>During destruction, the order reverses. Dependent resources get removed before the infrastructure they rely on.</p><p>Terraform uses these dependencies to order the changes in a plan.</p><p>Now, let&#8217;s look at what happens to each resource throughout its lifecycle.</p><h4 style="text-align: center;"><strong>&#167;</strong></h4><div class="callout-block" data-callout="true"><p style="text-align: center;"><em><strong><span data-color="#3a7bf9" style="color: rgb(58, 123, 249);">Reminder: this is a teaser of the subscriber-only newsletter, exclusive to my golden members.</span></strong></em></p><p>When you upgrade, you&#8217;ll get:</p><ul><li><p><strong>High-level architecture of real-world systems.</strong></p></li><li><p>Deep dive into how popular real-world systems work.</p></li><li><p><strong>How real-world systems handle scale, reliability &amp; performance.</strong></p></li></ul><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/subscribe&quot;,&quot;text&quot;:&quot;Unlock Full Access&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://newsletter.systemdesign.one/subscribe"><span>Unlock Full Access</span></a></p><p>(If this newsletter has helped you become a better software engineer, consider subscribing to support my work.)</p></div><h4 style="text-align: center;"><strong>&#167;</strong></h4><h2>How Terraform Manages Resource Lifecycle</h2>
      <p>
          <a href="https://newsletter.systemdesign.one/p/how-terraform-works-internally">
              Read more
          </a>
      </p>
   ]]></content:encoded></item><item><title><![CDATA[High Performance LLM Inference]]></title><description><![CDATA[#201: How to make LLM Inference Faster and More Efficient]]></description><link>https://newsletter.systemdesign.one/p/how-to-optimize-llm-inference</link><guid isPermaLink="false">https://newsletter.systemdesign.one/p/how-to-optimize-llm-inference</guid><dc:creator><![CDATA[Neo Kim]]></dc:creator><pubDate>Mon, 09 Nov 2026 10:32:38 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/b49272a0-cb6e-40d0-9257-a7486c996738_1280x720.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p></p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption"><span data-color="#3a7bf9" style="color: rgb(58, 123, 249);">Get my AI engineering playbook for FREE on newsletter signup:</span></p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><ul><li><p><em><a href="https://newsletter.systemdesign.one/p/how-to-optimize-llm-inference/?action=share">Share this letter</a> &amp; I&#8217;ll send you some rewards for the referrals.</em></p></li></ul><div><hr></div><p>We were taught that making LLM inference faster means using more powerful GPUs.</p><p>That advice is exactly why we have expensive hardware underused while users wait longer for responses.</p><p>Here&#8217;s what works instead<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-1" href="#footnote-1" target="_self">1</a>:</p><div><hr></div><div class="callout-block" data-callout="true"><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://blog.n8n.io/introducing-n8n-agents/?utm_source=passionfroot&amp;utm_medium=newsletter&amp;utm_campaign=global_global_prospecting_paid-social_linkedin_feed_all-products_traffic_20261001" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!JvT_!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F87b6fdc0-988a-439e-b615-74309dbaba00_1200x675.png 424w, https://substackcdn.com/image/fetch/$s_!JvT_!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F87b6fdc0-988a-439e-b615-74309dbaba00_1200x675.png 848w, https://substackcdn.com/image/fetch/$s_!JvT_!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F87b6fdc0-988a-439e-b615-74309dbaba00_1200x675.png 1272w, https://substackcdn.com/image/fetch/$s_!JvT_!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F87b6fdc0-988a-439e-b615-74309dbaba00_1200x675.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!JvT_!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F87b6fdc0-988a-439e-b615-74309dbaba00_1200x675.png" width="1200" height="675" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/87b6fdc0-988a-439e-b615-74309dbaba00_1200x675.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:675,&quot;width&quot;:1200,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:523317,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:&quot;https://blog.n8n.io/introducing-n8n-agents/?utm_source=passionfroot&amp;utm_medium=newsletter&amp;utm_campaign=global_global_prospecting_paid-social_linkedin_feed_all-products_traffic_20261001&quot;,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://newsletter.systemdesign.one/i/223933546?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F87b6fdc0-988a-439e-b615-74309dbaba00_1200x675.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!JvT_!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F87b6fdc0-988a-439e-b615-74309dbaba00_1200x675.png 424w, https://substackcdn.com/image/fetch/$s_!JvT_!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F87b6fdc0-988a-439e-b615-74309dbaba00_1200x675.png 848w, https://substackcdn.com/image/fetch/$s_!JvT_!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F87b6fdc0-988a-439e-b615-74309dbaba00_1200x675.png 1272w, https://substackcdn.com/image/fetch/$s_!JvT_!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F87b6fdc0-988a-439e-b615-74309dbaba00_1200x675.png 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>n8n just launched a <strong><a href="https://blog.n8n.io/introducing-n8n-agents/?utm_source=passionfroot&amp;utm_medium=newsletter&amp;utm_campaign=global_global_prospecting_paid-social_linkedin_feed_all-products_traffic_20261001">new type of AI agent</a></strong>, one you can control.</p><p>It works toward a goal and can act through tools like Gmail or Slack. You can also give it n8n workflows as tools that run exactly as you built them, so you decide which actions the agent handles on its own and which follow a set process. Add approvals before sensitive actions and see every action it takes.</p><p>Built for teams that need agents they can rely on.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://blog.n8n.io/introducing-n8n-agents/?utm_source=passionfroot&amp;utm_medium=newsletter&amp;utm_campaign=global_global_prospecting_paid-social_linkedin_feed_all-products_traffic_20261001&quot;,&quot;text&quot;:&quot;See it in action&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://blog.n8n.io/introducing-n8n-agents/?utm_source=passionfroot&amp;utm_medium=newsletter&amp;utm_campaign=global_global_prospecting_paid-social_linkedin_feed_all-products_traffic_20261001"><span>See it in action</span></a></p><p><em>(Thanks to <a href="https://blog.n8n.io/introducing-n8n-agents/?utm_source=passionfroot&amp;utm_medium=newsletter&amp;utm_campaign=global_global_prospecting_paid-social_linkedin_feed_all-products_traffic_20261001">n8n</a> for partnering on this newsletter.)</em></p></div><div><hr></div><h2><strong>Batching</strong></h2><p>Batching processes many requests together instead of running each request through the model separately.</p><p>A real AI service might have many users waiting for tokens at the same time. Processing each request separately would leave much of the GPU&#8217;s available capacity unused.</p><p>Batching allows several sequences to be processed together.</p><p>During decode<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-2" href="#footnote-2" target="_self">2</a>, the GPU repeatedly needs access to the model&#8217;s weights. When only one request is being processed, those weights are being used to advance only one sequence.</p><p>Imagine four users are waiting for their next token:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!kOjm!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F71882dfe-8d82-4cea-ab5e-b2fc75f7088a_1601x742.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!kOjm!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F71882dfe-8d82-4cea-ab5e-b2fc75f7088a_1601x742.png 424w, https://substackcdn.com/image/fetch/$s_!kOjm!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F71882dfe-8d82-4cea-ab5e-b2fc75f7088a_1601x742.png 848w, https://substackcdn.com/image/fetch/$s_!kOjm!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F71882dfe-8d82-4cea-ab5e-b2fc75f7088a_1601x742.png 1272w, https://substackcdn.com/image/fetch/$s_!kOjm!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F71882dfe-8d82-4cea-ab5e-b2fc75f7088a_1601x742.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!kOjm!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F71882dfe-8d82-4cea-ab5e-b2fc75f7088a_1601x742.png" width="1456" height="675" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/71882dfe-8d82-4cea-ab5e-b2fc75f7088a_1601x742.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:675,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!kOjm!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F71882dfe-8d82-4cea-ab5e-b2fc75f7088a_1601x742.png 424w, https://substackcdn.com/image/fetch/$s_!kOjm!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F71882dfe-8d82-4cea-ab5e-b2fc75f7088a_1601x742.png 848w, https://substackcdn.com/image/fetch/$s_!kOjm!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F71882dfe-8d82-4cea-ab5e-b2fc75f7088a_1601x742.png 1272w, https://substackcdn.com/image/fetch/$s_!kOjm!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F71882dfe-8d82-4cea-ab5e-b2fc75f7088a_1601x742.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>The inference system groups those requests into a &#8220;batch&#8221;.</p><p>During the next decode step, the GPU processes the current position for ALL four sequences together. The model weights are still needed, but the work done with those weights is spread across several requests. This gives the GPU more computation to perform for the data it accesses and makes better use of its parallel processing capacity.</p><p>The same idea applies to arithmetic intensity<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-3" href="#footnote-3" target="_self">3</a>.</p><p>Processing more sequences together increases the amount of useful computation performed while the model weights are being used. But batching does not make additional requests free.</p><p>A batch of thirty-two requests requires more computation than a single request.</p><p>Each sequence also has its own attention<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-4" href="#footnote-4" target="_self">4</a> work, intermediate values, and KV cache. As the batch grows, both compute &amp; memory usage increase. The advantage is the GPU can use its resources much more efficiently.</p><p>This introduces an important distinction between latency &amp; throughput.</p><ul><li><p>Latency describes how long an individual request takes.</p></li><li><p>Throughput describes how much total work the system can complete in a time period, typically measured in tokens per second across all requests.</p></li></ul><p>Batching mainly improves throughput because the system advances many sequences together instead of one at a time.</p><p>But there are limits to how large a batch could become&#8230;Memory is one of them.</p><p>Every active request needs space for its KV cache, and the cache grows as the sequence gets longer. A large batch of long conversations could eventually consume the available GPU memory.</p><p>Timing creates another problem&#8230;</p><p>Real requests do not arrive/finish together.</p><p>Imagine four requests enter a batch:</p><div class="callout-block" data-callout="true"><p>Request A &#8594; finishes after 20 tokens<br>Request B &#8594; finishes after 100 tokens<br>Request C &#8594; finishes after 50 tokens<br>Request D &#8594; finishes after 200 tokens</p></div><p>Request A finishes much earlier than Request D.</p><p>With &#8220;static&#8221; batching, the group gets formed before processing begins. When shorter requests finish, their places are not immediately filled with new requests. So newer requests wait while the remaining requests in the current batch continue.</p><p>This reduces some of the efficiency batching was meant to provide.</p><p>Real traffic is constantly changing&#8230;</p><ul><li><p>Users send prompts at different times.</p></li><li><p>Plus prompts have different lengths.</p></li><li><p>Also generated answers finish after different numbers of tokens.</p></li></ul><p>i.e., an efficient serving system needs a batch that changes while the model is generating.</p><p>That&#8217;s why we need continuous batching...</p><h2><strong>Continuous Batching</strong></h2><p>Continuous batching means changing the group of requests being processed as requests finish &amp; new ones become ready.</p><p>Imagine 32 requests starting together:</p><p>One request finishes after 20 tokens, another after 100, and another keeps generating for 500 tokens.</p><p>With static batching, the group gets decided before generation begins. Shorter requests finish while longer ones are still running. New requests then have to wait for the current batch to complete.</p><p>Continuous batching changes how the system schedules this work. (Remember, decode happens one step at a time.)</p><p>The model generates the next token for the active sequences, then moves to the next generation step. Instead of scheduling an entire request from beginning to end, the system makes scheduling decisions between these iterations.</p><p>This is called &#8220;iteration-level&#8221; scheduling.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!789h!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb1c040ab-a9fb-432d-a5eb-589749e906f6_1353x819.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!789h!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb1c040ab-a9fb-432d-a5eb-589749e906f6_1353x819.png 424w, https://substackcdn.com/image/fetch/$s_!789h!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb1c040ab-a9fb-432d-a5eb-589749e906f6_1353x819.png 848w, https://substackcdn.com/image/fetch/$s_!789h!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb1c040ab-a9fb-432d-a5eb-589749e906f6_1353x819.png 1272w, https://substackcdn.com/image/fetch/$s_!789h!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb1c040ab-a9fb-432d-a5eb-589749e906f6_1353x819.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!789h!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb1c040ab-a9fb-432d-a5eb-589749e906f6_1353x819.png" width="1353" height="819" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/b1c040ab-a9fb-432d-a5eb-589749e906f6_1353x819.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:819,&quot;width&quot;:1353,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!789h!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb1c040ab-a9fb-432d-a5eb-589749e906f6_1353x819.png 424w, https://substackcdn.com/image/fetch/$s_!789h!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb1c040ab-a9fb-432d-a5eb-589749e906f6_1353x819.png 848w, https://substackcdn.com/image/fetch/$s_!789h!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb1c040ab-a9fb-432d-a5eb-589749e906f6_1353x819.png 1272w, https://substackcdn.com/image/fetch/$s_!789h!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb1c040ab-a9fb-432d-a5eb-589749e906f6_1353x819.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Suppose the active batch looks like this:</p><div class="callout-block" data-callout="true"><p>A B C D</p></div><p>After a generation step, Request A finishes.</p><p>The scheduler removes A and uses the available capacity for another waiting request when the system has enough compute and memory available.</p><p>The batch can therefore change while generation continues.</p><div class="callout-block" data-callout="true"><p>Step 1 &#8594; A B C D<br>Step 2 &#8594; A B C D</p><p>&#8230;</p><p>A finishes</p><p>&#8230;</p><p>Step 3 &#8594; E B C D</p></div><p>i.e., the system does not need to wait for B, C &amp; D to finish before it considers E.</p><p>This makes continuous batching better suited to real traffic, where requests arrive at different times and generate answers of different lengths. The idea became important in LLM serving through systems such as Orca, which introduced iteration-level scheduling for generative Transformer models. Modern inference engines such as vLLM use continuous batching as part of their serving systems.</p><p>But adding a new request is not free/automatic.</p><p>Every active request consumes resources, including space for its KV cache. Also, the scheduler has a limit on how many sequences and tokens it can process in one iteration. A new request only joins when the system has enough capacity to schedule its work.</p><p>Besides new requests need prefill before they continue through decode.</p><p>Modern inference engines must decide how to schedule prefill work alongside requests already generating tokens.</p><p>So continuous batching is about more than filling empty slots&#8230;</p><p>The scheduler repeatedly decides which requests should use the GPU next instead of locking the GPU into one batch until every request finishes. This keeps the hardware busy even when users arrive &amp; leave at unpredictable times.</p><p>Plus different requests could share more than the same GPU.</p><p>Many begin with the same tokens. Recomputing those tokens every time creates another unnecessary workload.</p><p>This is where you need prefix caching...</p><h2><strong>Prefix Caching</strong></h2><p>Prefix caching means reusing the KV cache for a prompt beginning the model has already processed.</p><p>Many requests sent to an AI app begin with the same tokens.</p><ul><li><p>A chatbot would use the same system instructions at the beginning of every request.</p></li><li><p>A Retrieval-Augmented Generation (<strong>RAG</strong>) application would repeatedly send the same document/context before asking different questions about it.</p></li></ul><p>Even a normal multi-turn conversation contains repeated text.</p><p>When you send another message, the app sends much of the earlier conversation back to the model as part of the new prompt. Without caching, the model would process those repeated tokens again during prefill.</p><p>But the KV cache could help.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!rLw3!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F94ae9001-9fce-4a64-bb32-ff618723f421_1420x841.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!rLw3!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F94ae9001-9fce-4a64-bb32-ff618723f421_1420x841.png 424w, https://substackcdn.com/image/fetch/$s_!rLw3!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F94ae9001-9fce-4a64-bb32-ff618723f421_1420x841.png 848w, https://substackcdn.com/image/fetch/$s_!rLw3!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F94ae9001-9fce-4a64-bb32-ff618723f421_1420x841.png 1272w, https://substackcdn.com/image/fetch/$s_!rLw3!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F94ae9001-9fce-4a64-bb32-ff618723f421_1420x841.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!rLw3!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F94ae9001-9fce-4a64-bb32-ff618723f421_1420x841.png" width="1420" height="841" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/94ae9001-9fce-4a64-bb32-ff618723f421_1420x841.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:841,&quot;width&quot;:1420,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!rLw3!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F94ae9001-9fce-4a64-bb32-ff618723f421_1420x841.png 424w, https://substackcdn.com/image/fetch/$s_!rLw3!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F94ae9001-9fce-4a64-bb32-ff618723f421_1420x841.png 848w, https://substackcdn.com/image/fetch/$s_!rLw3!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F94ae9001-9fce-4a64-bb32-ff618723f421_1420x841.png 1272w, https://substackcdn.com/image/fetch/$s_!rLw3!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F94ae9001-9fce-4a64-bb32-ff618723f421_1420x841.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>During prefill, the model computes keys &amp; values for prompt tokens and stores them in the KV cache.</p><p>If another request begins with exactly the same sequence of tokens, the system reuses the keys and values already computed for that shared beginning.</p><p>That shared beginning is called a &#8220;prefix&#8221;.</p><p>Imagine 2 requests look like this:</p><div class="callout-block" data-callout="true"><p>Request A: [System instructions] + [Document] + &#8220;Summarize this&#8221;</p><p>Request B: [System instructions] + [Document] + &#8220;What are the main arguments?&#8221;</p></div><p>Both requests begin with same system instructions &amp; document.</p><p>With prefix caching, the inference system reuses cached KV data for the matching prefix instead of recomputing it. Prefill only needs to process the part that is not already available in the cache.</p><p>The keyword here is <em>prefix</em>.</p><p>The matching tokens have to appear at the beginning of the request. Once the token sequence differs, later KV states depend on that different sequence and cannot simply be reused as part of the same cached prefix.</p><p>This is why prompt order matters for caching.</p><p>If an app has a large block of instructions/context that stays the same across requests, placing that stable content before the parts that change increases how much of the prompt is reusable.</p><p>Prefix caching reduces repeated prefill computation.</p><p>For long shared prefixes, this reduces time to first token because the model has fewer uncached prompt tokens to process before generation begins. Also it reduces cost when a provider charges <em>less</em> for cached input tokens than for uncached input tokens.</p><p>But a cached prefix does NOT make the entire request free.</p><p>Any new part of the prompt still needs processing, and output still has to go through decode.</p><p>Now many requests share the GPU, batches change while generation runs, and cached prefixes avoid repeated work.</p><p>But when more requests are waiting than the system can handle at once, the request scheduler decides which one runs next&#8230;</p><div><hr></div><div class="captioned-button-wrap" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/p/how-to-optimize-llm-inference?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;}" data-component-name="CaptionedButtonToDOM"><div class="preamble"><p class="cta-caption"><em><span data-color="#3a7bf9" style="color: rgb(58, 123, 249);">If something resonated with you today, share this letter. Because one idea, one action, can change everything.</span></em></p></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/p/how-to-optimize-llm-inference?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://newsletter.systemdesign.one/p/how-to-optimize-llm-inference?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p></div><div><hr></div><h2><strong>Request Scheduling</strong></h2><p>Request scheduling decides which requests get to use the model &amp; when they get to run.</p><p>Continuous batching lets requests enter &amp; leave while generation is happening.</p><p>But the GPU can only handle so much work at once&#8230;</p><p>Imagine 100 requests waiting, but the inference server currently has enough capacity to process only part of them&#8230;The remaining requests have to wait. This creates a &#8220;queue&#8221;. The scheduler sits between that queue &amp; the model. Its job is to decide which requests move from waiting into active processing.</p><p>The simplest approach is first-come-first-served (<strong>FCFS</strong>):</p><ul><li><p>The first request to arrive is considered first.</p></li><li><p>If Request A arrives before Request B, the scheduler tries to process A before B.</p></li></ul><p>But arrival time is not the only way to make this decision&#8230;</p><p>Some systems also support priority scheduling. A request can be assigned a priority, and the scheduler considers higher-priority work before lower-priority work. Arrival time can then decide between requests with the same priority.</p><p>Choosing a request is only part of the scheduler&#8217;s job.</p><p>Also it has to check whether enough resources are available to run it. Every active request needs KV-cache memory, and the model also has a limit on how many sequences &amp; tokens it can process during an iteration.</p><p>So even if a request is next in the queue, the scheduler cannot keep adding work.</p><p>Suppose the GPU is already processing several long conversations:</p><p>Their KV caches occupy most of the available cache memory. Another request is waiting... Before admitting it, the serving system has to check whether enough capacity remains.</p><p>Admitting too many requests could create memory pressure and force the system to interrupt and/or rearrange already running work.</p><p>&#8220;Admission control&#8221; decides whether new work should enter the active system based on currently available resources. The exact rules depend on the inference engine, but the goal is the same: avoid accepting more active work than the system can handle efficiently.</p><p>Scheduling therefore involves more than asking, <em>&#8220;Who arrived first?&#8221;</em></p><p>The system has to balance waiting requests against available compute, token budgets &amp; KV-cache memory.</p><p>This matters even more because the KV cache is NOT a fixed size.</p><p>Every new token makes it grow. With many active users, those caches have to share a limited amount of GPU memory. How that memory is allocated determines how many requests the server can keep running at once.</p><p>That brings us to PagedAttention&#8230;</p><h2><strong>PagedAttention</strong></h2><p>PagedAttention is a technique that manages the KV cache in small blocks instead of storing each request&#8217;s cache in one large, continuous area of GPU memory.</p><p>KV cache grows as a model processes more tokens... The problem is the inference server does NOT know exactly how large each request&#8217;s KV cache will become&#8230;</p><p>Imagine a user sends a prompt &amp; model is allowed to generate up to 1,000 tokens:</p><p>The answer might finish after 100 tokens, or it might use almost the entire limit. One approach is to reserve enough continuous GPU memory for the maximum possible sequence length before generation begins.</p><p>But much of that reserved space could remain unused if the request finishes early.</p><p>There is another problem called &#8220;fragmentation&#8221;.</p><p>Imagine GPU memory as a row of storage spaces:</p><p>Requests enter &amp; leave, creating free spaces in different places over time. There may be enough free memory in total for a new KV cache, but not enough &#8220;continuous&#8221; free space where the system expects to place it.</p><p>Both problems waste memory that could otherwise hold more active requests.</p><p>PagedAttention solves this by borrowing an idea from operating-system virtual memory: <em>paging</em>.</p><p>Instead of requiring a request's KV cache to occupy one continuous area of physical memory, PagedAttention divides it into &#8220;fixed-size&#8221; blocks.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!XJ6s!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F06a5bed0-07a1-4e39-acee-c397fb1ac5ae_1239x809.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!XJ6s!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F06a5bed0-07a1-4e39-acee-c397fb1ac5ae_1239x809.png 424w, https://substackcdn.com/image/fetch/$s_!XJ6s!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F06a5bed0-07a1-4e39-acee-c397fb1ac5ae_1239x809.png 848w, https://substackcdn.com/image/fetch/$s_!XJ6s!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F06a5bed0-07a1-4e39-acee-c397fb1ac5ae_1239x809.png 1272w, https://substackcdn.com/image/fetch/$s_!XJ6s!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F06a5bed0-07a1-4e39-acee-c397fb1ac5ae_1239x809.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!XJ6s!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F06a5bed0-07a1-4e39-acee-c397fb1ac5ae_1239x809.png" width="1239" height="809" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/06a5bed0-07a1-4e39-acee-c397fb1ac5ae_1239x809.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:809,&quot;width&quot;:1239,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!XJ6s!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F06a5bed0-07a1-4e39-acee-c397fb1ac5ae_1239x809.png 424w, https://substackcdn.com/image/fetch/$s_!XJ6s!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F06a5bed0-07a1-4e39-acee-c397fb1ac5ae_1239x809.png 848w, https://substackcdn.com/image/fetch/$s_!XJ6s!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F06a5bed0-07a1-4e39-acee-c397fb1ac5ae_1239x809.png 1272w, https://substackcdn.com/image/fetch/$s_!XJ6s!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F06a5bed0-07a1-4e39-acee-c397fb1ac5ae_1239x809.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>A request then uses blocks located in different places in GPU memory.</p><p>The blocks do not have to sit next to each other physically.</p><p>The system tracks their locations using a &#8220;block table&#8221;. This gives the model a logical view of the request&#8217;s KV cache while the memory manager handles where those blocks actually live.</p><p>Also more blocks could be allocated as the sequence grows&#8230;</p><p>Instead of reserving memory for a 1,000-token answer that might stop after 100 tokens, the system allocates KV-cache blocks as they&#8217;re needed.</p><p>Some <em>unused</em> space may remain in the final block.</p><p>A sequence does not always end exactly at a block boundary. But the amount of wasted memory is much smaller than reserving a large continuous region for every request.</p><p>This matters because KV-cache memory limits how many requests remain active on a GPU.</p><p>If each request wastes less memory, the server could keep more sequences in memory at the same time. This enables larger effective batches, thus increasing throughput.</p><p>Plus, paged blocks make sharing KV-cache data easier.</p><p>If two requests share the same cached prefix, their block tables refer to the same physical KV-cache blocks instead of storing another copy for each request. (This connects PagedAttention directly to the prefix caching.)</p><p>The original vLLM work reported near-zero waste in KV-cache memory. Also it improved throughput two to four times over the serving systems it compared against while maintaining similar latency.</p><p>PagedAttention does not make the KV cache smaller by changing what the model stores. Instead, it changes how that cache is placed and managed in memory, thus allowing the available GPU memory to be used more efficiently.</p><p>But KV-cache storage is only one part of the attention problem.</p><p>Even with efficient memory allocation, the model has to perform the attention computation itself.</p><p>This brings us to FlashAttention&#8230;</p><h2><strong>FlashAttention</strong></h2><p>FlashAttention<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-5" href="#footnote-5" target="_self">5</a> is a faster way to compute attention by <em>reducing</em> how much data moves between different GPU memory levels.</p><p>When the model processes a sequence, each token needs a way to decide which earlier tokens are relevant. Attention does this by comparing tokens &amp; producing scores that describe how strongly one position should attend to another.</p><p>For a sequence with 2,000 tokens, this could involve millions of attention scores.</p><p>A straightforward implementation creates a large matrix containing these intermediate scores.</p><p>The GPU writes intermediate data to its high-bandwidth memory (<strong>HBM</strong>), reads it back for another operation, writes another result, and continues through the attention calculation.</p><p>The calculations themselves are fast on a GPU.</p><p>But moving all those intermediate values between HBM &amp; GPU&#8217;s much smaller on-chip memory would become extremely expensive.</p><p>FlashAttention changes how this calculation is performed.</p><p>Instead of computing &amp; storing the entire attention matrix in HBM, it divides the work into smaller pieces called &#8220;tiles&#8221;. Each tile is small enough for the GPU to work on using its faster on-chip SRAM.</p><p>The GPU loads a portion of the required data, performs several parts of the attention calculation there, and combines the results without writing the full attention matrix to HBM.</p><p>This keeps more of the work close to the GPU&#8217;s compute units.</p><p>Instead of repeatedly moving large amounts of intermediate data in and out of HBM, FlashAttention reduces these expensive memory transfers.</p><p>There is an important detail here:</p><p>FlashAttention does not approximate standard attention to make it faster. It computes exact attention while changing how the computation is organized around the GPU&#8217;s memory hierarchy.</p><p>Also it does not remove the quadratic amount of arithmetic required by standard dense attention.</p><p>If the sequence length doubles, the number of token-to-token comparisons still grows roughly with the square of the sequence length. FlashAttention reduces the intermediate memory it needs and the number of expensive reads and writes to HBM.</p><p>Its auxiliary memory requirement grows linearly, not quadratically, with sequence length.</p><p>This distinction explains why FlashAttention makes exact attention much faster without changing what the attention operation computes.</p><p>The original FlashAttention work showed that paying attention to where data moves, not only how many calculations are performed, produces large speed improvements.</p><p>PagedAttention &amp; FlashAttention therefore solve two different memory problems:</p><ul><li><p>PagedAttention manages how the KV cache gets stored while serving requests.</p></li><li><p>FlashAttention improves how the attention calculation itself moves data through GPU memory.</p></li></ul><p>But even an efficient algorithm still has to become instructions the GPU can execute.</p><p>Those instructions run inside GPU kernels&#8230;</p><div><hr></div><div class="callout-block" data-callout="true"><p style="text-align: center;"><em><strong><span data-color="#3a7bf9" style="color: rgb(58, 123, 249);">Reminder: this is a teaser of the subscriber-only newsletter, exclusive to my golden members.</span></strong></em></p><p>When you upgrade, you&#8217;ll get:</p><ul><li><p><strong>High-level architecture of real-world systems.</strong></p></li><li><p>Deep dive into how popular real-world systems work.</p></li><li><p><strong>How real-world systems handle scale, reliability &amp; performance.</strong></p></li></ul><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/subscribe&quot;,&quot;text&quot;:&quot;Unlock Full Access&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://newsletter.systemdesign.one/subscribe"><span>Unlock Full Access</span></a></p><p>(If this newsletter has helped you become a better software engineer, consider subscribing to support my work.)</p></div><div><hr></div><h2><strong>GPU Kernels</strong></h2>
      <p>
          <a href="https://newsletter.systemdesign.one/p/how-to-optimize-llm-inference">
              Read more
          </a>
      </p>
   ]]></content:encoded></item><item><title><![CDATA[System Design Interview: Design Live Streaming System]]></title><description><![CDATA[#200: System Design Interview]]></description><link>https://newsletter.systemdesign.one/p/live-streaming-system-design</link><guid isPermaLink="false">https://newsletter.systemdesign.one/p/live-streaming-system-design</guid><dc:creator><![CDATA[Neo Kim]]></dc:creator><pubDate>Fri, 06 Nov 2026 12:31:39 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/ae102526-d92c-4c18-b667-25b0586d7d4a_1280x720.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p></p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption"><span data-color="#3a7bf9" style="color: rgb(58, 123, 249);">Get my AI engineering playbook for FREE on newsletter signup:</span></p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><ul><li><p><em><a href="https://newsletter.systemdesign.one/p/live-streaming-system-design/?action=share">Share this letter</a> &amp; I&#8217;ll send you some rewards for the referrals.</em></p></li></ul><div><hr></div><p>50 million people hit play, and your live streaming system has only a few seconds to respond&#8230;!</p><p>A live streaming platform must capture a high-quality video feed, process it in near real time, and deliver it to millions of viewers across phones, laptops &amp; TVs.</p><p>Traffic could surge within seconds when a match starts/key moment occurs.</p><p>Every stage adds latency, and every bottleneck puts the stream at risk.</p><p>In this newsletter, we&#8217;ll break down the system design of a live streaming platform built for 50 million <em>concurrent</em> viewers.</p><p>Let&#8217;s start with the system requirements&#8230;</p><div><hr></div><div class="callout-block" data-callout="true"><h2><a href="https://prlens.dev/?utm_source=neokim&amp;utm_medium=newsletter&amp;utm_campaign=ad2">A walkthrough of every agent PR</a></h2><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://prlens.dev/?utm_source=neokim&amp;utm_medium=newsletter&amp;utm_campaign=ad2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!8piF!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbed88e8-d685-4a85-8795-4805c8781f1a_1200x628.gif 424w, https://substackcdn.com/image/fetch/$s_!8piF!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbed88e8-d685-4a85-8795-4805c8781f1a_1200x628.gif 848w, https://substackcdn.com/image/fetch/$s_!8piF!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbed88e8-d685-4a85-8795-4805c8781f1a_1200x628.gif 1272w, https://substackcdn.com/image/fetch/$s_!8piF!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbed88e8-d685-4a85-8795-4805c8781f1a_1200x628.gif 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!8piF!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbed88e8-d685-4a85-8795-4805c8781f1a_1200x628.gif" width="1200" height="628" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/bbed88e8-d685-4a85-8795-4805c8781f1a_1200x628.gif&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:628,&quot;width&quot;:1200,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:1202568,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:&quot;&quot;,&quot;type&quot;:&quot;image/gif&quot;,&quot;href&quot;:&quot;https://prlens.dev/?utm_source=neokim&amp;utm_medium=newsletter&amp;utm_campaign=ad2&quot;,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://newsletter.systemdesign.one/i/222466719?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbed88e8-d685-4a85-8795-4805c8781f1a_1200x628.gif&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!8piF!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbed88e8-d685-4a85-8795-4805c8781f1a_1200x628.gif 424w, https://substackcdn.com/image/fetch/$s_!8piF!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbed88e8-d685-4a85-8795-4805c8781f1a_1200x628.gif 848w, https://substackcdn.com/image/fetch/$s_!8piF!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbed88e8-d685-4a85-8795-4805c8781f1a_1200x628.gif 1272w, https://substackcdn.com/image/fetch/$s_!8piF!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbed88e8-d685-4a85-8795-4805c8781f1a_1200x628.gif 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Every agent PR comes with a screenshot. Agents open them faster than anyone can read them.</p><p><strong><a href="https://prlens.dev/?utm_source=neokim&amp;utm_medium=newsletter&amp;utm_campaign=ad2">PR Lens</a></strong> adds the architecture. It draws each pull request as a canvas of your system, marks every part new, changed, or removed, and walks you through how it fits.</p><p>On every PR, you see:</p><ul><li><p><strong>What changed</strong>, marked on the parts of the system it touches</p></li><li><p><strong>What else it reaches</strong>, including services the diff never names</p></li><li><p><strong>How a request moves</strong> through the system now</p></li><li><p><strong>What could break</strong>, flagged before you merge</p></li></ul><p>2,200+ GitHub stars and 45,000+ pull requests drawn.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://prlens.dev/?utm_source=neokim&amp;utm_medium=newsletter&amp;utm_campaign=ad2&quot;,&quot;text&quot;:&quot;Add it to your repo&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://prlens.dev/?utm_source=neokim&amp;utm_medium=newsletter&amp;utm_campaign=ad2"><span>Add it to your repo</span></a></p><p>(Thanks to <a href="https://prlens.dev/?utm_source=neokim&amp;utm_medium=newsletter&amp;utm_campaign=ad2">Coldtea</a> for partnering on this newsletter.)</p></div><div><hr></div><p>I want to introduce <a href="https://www.linkedin.com/in/msdeep14/">Mandeep Singh</a> as a guest author.</p><div class="captioned-image-container"><figure><a class="image-link image2" target="_blank" href="https://www.linkedin.com/in/msdeep14/" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!H-hM!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F34171e56-0f79-49f3-8271-55f6d7fb39a8_1584x396.png 424w, https://substackcdn.com/image/fetch/$s_!H-hM!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F34171e56-0f79-49f3-8271-55f6d7fb39a8_1584x396.png 848w, https://substackcdn.com/image/fetch/$s_!H-hM!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F34171e56-0f79-49f3-8271-55f6d7fb39a8_1584x396.png 1272w, https://substackcdn.com/image/fetch/$s_!H-hM!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F34171e56-0f79-49f3-8271-55f6d7fb39a8_1584x396.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!H-hM!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F34171e56-0f79-49f3-8271-55f6d7fb39a8_1584x396.png" width="1456" height="364" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/34171e56-0f79-49f3-8271-55f6d7fb39a8_1584x396.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:364,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:852266,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:&quot;https://www.linkedin.com/in/msdeep14/&quot;,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://newsletter.systemdesign.one/i/209259249?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F34171e56-0f79-49f3-8271-55f6d7fb39a8_1584x396.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!H-hM!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F34171e56-0f79-49f3-8271-55f6d7fb39a8_1584x396.png 424w, https://substackcdn.com/image/fetch/$s_!H-hM!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F34171e56-0f79-49f3-8271-55f6d7fb39a8_1584x396.png 848w, https://substackcdn.com/image/fetch/$s_!H-hM!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F34171e56-0f79-49f3-8271-55f6d7fb39a8_1584x396.png 1272w, https://substackcdn.com/image/fetch/$s_!H-hM!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F34171e56-0f79-49f3-8271-55f6d7fb39a8_1584x396.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div></div></div></a></figure></div><p>He&#8217;s an author, educator, and software engineer with a strong passion for building simple, scalable systems.</p><p>He authored the O&#8217;Reilly book &#8220;System Design on AWS&#8221; and is working on the upcoming O&#8217;Reilly book &#8220;System Design From Scratch&#8221;. He also supports others&#8217; learning and development by sharing lessons on Cloud and System Design on <a href="https://www.youtube.com/@MsDeepSingh">YouTube</a>.</p><p>Checkout his work:</p><ul><li><p><a href="https://oreil.ly/ruQbc">System Design on AWS Book</a> (Also find on <a href="https://amzn.to/3D3BIBJ">Amazon</a>)</p></li><li><p><a href="https://systemdesignfromscratch.org/welcome">Newsletter</a></p></li><li><p><a href="https://www.linkedin.com/in/msdeep14/">LinkedIn</a></p></li><li><p><a href="https://x.com/msdeep14">X</a></p></li></ul><div><hr></div><h2><span>System Requirements &amp; Scale</span></h2><p>Before you design the architecture, you must define what the system supports.</p><p>i.e., separate functional requirements from non-functional requirements</p><p>Functional requirements describe what viewers can do, while non-functional requirements define how well the system must perform at scale.</p><p>Let&#8217;s get into it:</p><h3>Functional and Non-Functional Requirements</h3><p>Here are the key &#8220;functional&#8221; requirements:</p><ul><li><p>Viewers can watch live matches.</p></li><li><p>Video must be streamed based on device type &amp; network bandwidth.</p></li></ul><p>Let&#8217;s <em>leave out</em> these features to limit the scope:</p><ul><li><p>Multiple audio languages (keep only English),</p></li><li><p>Live chat &amp; user reactions,</p></li><li><p>Ads pipeline.</p></li></ul><p>&#8220;Non-functional&#8221; requirements define how well the platform must perform. Here&#8217;s an essential list for smooth operations:</p><ul><li><p>Low latency: the stream should reach viewers in real time without buffering. For this design, we&#8217;ll target 4 to 30 seconds of end-to-end latency.</p></li><li><p>Scalability: traffic could exponentially rise/fall sharply during a live event; the system must absorb sudden spikes in concurrent viewers.</p></li><li><p>Extensibility: architecture should support new event types &amp; device types without major redesigns.</p></li></ul><p>End-to-end latency comes from several stages between capture &amp; playback.</p><p>(We&#8217;ll examine those stages separately later &amp; see where each one adds delay.)</p><p>Latency is only half the problem&#8230; Scale also drives many architecture choices&#8230;</p><p>So let&#8217;s put some numbers behind it&#8230;</p><h3><span>System Scale</span></h3><p>Live events create sharp traffic spikes&#8230;</p><p>i.e., peak concurrent viewers matter more than average traffic.</p><p>For this design, assume:</p><ul><li><p>Maximum concurrent viewers: 50 million</p></li><li><p>Average stream bitrate: 5 Mbps</p></li></ul><p>(<em>Bitrate</em> is the amount of video data transferred each second.)</p><p>So the total outgoing bandwidth for a 1080p video is:</p><div class="highlighted_code_block" data-attrs="{&quot;language&quot;:&quot;plaintext&quot;,&quot;nodeId&quot;:&quot;764656f4-08d7-4365-9c70-e4aecdd745a0&quot;}" data-component-name="HighlightedCodeBlockToDOM"><pre class="shiki"><code class="language-plaintext">Total Bandwidth = Peak Viewers &#215; Average Bitrate

50,000,000 &#215; 5 Mbps = 250,000,000 Mbps = 250 Tbps</code></pre></div><p>This massive number shows why a central server cannot efficiently deliver the stream to every viewer<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-1" href="#footnote-1" target="_self">1</a>.</p><p>You need video compression, adaptive bitrate streaming, and a Content Delivery Network (<strong>CDN</strong>) to distribute the traffic across edge locations closer to viewers.</p><p>Let&#8217;s move on to the high-level architecture&#8230;</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/p/live-streaming-system-design?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://newsletter.systemdesign.one/p/live-streaming-system-design?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p><div><hr></div><h2><span>High-Level System Design</span></h2><p>The system streams video from cameras at the event venue to users&#8217; screens.</p><p>There are three main phases:</p><ul><li><p>Video creation/ingestion,</p></li><li><p>Video processing,</p></li><li><p>Video distribution.</p></li></ul><p>Here&#8217;s the high-level workflow:</p><ol><li><p>High-resolution cameras record the video at the event venue.</p></li><li><p>The system ingests the video at the venue and prepares it for transfer to application servers for extra processing.</p></li><li><p>Video processing prepares the stream to support different device types.</p></li><li><p>Finally, video distribution delivers the stream to users&#8217; devices.</p></li></ol><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!ZieU!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbde1f970-8d3f-4abd-bbf8-87feb520b137_552x135.png 424w, https://substackcdn.com/image/fetch/$s_!ZieU!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbde1f970-8d3f-4abd-bbf8-87feb520b137_552x135.png 848w, https://substackcdn.com/image/fetch/$s_!ZieU!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbde1f970-8d3f-4abd-bbf8-87feb520b137_552x135.png 1272w, https://substackcdn.com/image/fetch/$s_!ZieU!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbde1f970-8d3f-4abd-bbf8-87feb520b137_552x135.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!ZieU!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbde1f970-8d3f-4abd-bbf8-87feb520b137_552x135.png" width="552" height="135" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/bde1f970-8d3f-4abd-bbf8-87feb520b137_552x135.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:135,&quot;width&quot;:552,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!ZieU!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbde1f970-8d3f-4abd-bbf8-87feb520b137_552x135.png 424w, https://substackcdn.com/image/fetch/$s_!ZieU!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbde1f970-8d3f-4abd-bbf8-87feb520b137_552x135.png 848w, https://substackcdn.com/image/fetch/$s_!ZieU!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbde1f970-8d3f-4abd-bbf8-87feb520b137_552x135.png 1272w, https://substackcdn.com/image/fetch/$s_!ZieU!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbde1f970-8d3f-4abd-bbf8-87feb520b137_552x135.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div></div></div></div><figcaption class="image-caption">High-Level View of Live Streaming Platform</figcaption></figure></div><p>Each phase addresses a different part of the journey from camera to screen.</p><p>Let&#8217;s start with video ingestion&#8230;</p><h3><span>Video Ingestion</span></h3><p>The journey starts with cameras at the live event&#8230;</p><p>These cameras capture high-quality video and create a large amount of data. Transferring the original video directly for further processing would require significant bandwidth &amp; take more time. So the system first reduces the video size while keeping the quality in check.</p><p>A video &#8220;encoder&#8221; at the event location handles this step.</p><p>The event location is typically called the Production Control Room (<strong>PCR</strong>), although the exact terminology varies by company.</p><p>The encoder uses video codecs to compress the video. </p><p>Here are the popular options:</p><ul><li><p>H.264 - Advanced Video Coding (<strong>AVC</strong>).</p></li><li><p>H.265 - High Efficiency Video Coding (<strong>HEVC</strong>), the successor to H.264.</p></li></ul><p>Once encoded, the smaller video stream needs to reach a central location for further processing &amp; delivery. This infrastructure could run in the cloud or in your own on-premises data center.</p><p>Next comes an important design decision: <em>how do you transfer this live video stream with low latency?</em></p><h3><span>Live Data Transfer</span></h3><p>Once the video gets encoded, the system must transfer it from the PCR to a central processing location.</p><p>Think of it like uploading a large video file to Google Drive:</p><p>The upload takes time, especially for large files. But for live streaming, video must reach the processing servers with minimal delay.</p><p>Transferring video over the public internet creates 2 challenges:</p><ul><li><p><strong>Variable network conditions:</strong> Network congestion &amp; packet loss could delay video delivery.</p></li><li><p><strong>Bandwidth limitations:</strong> Available bandwidth could fluctuate, especially during peak traffic.</p></li></ul><p>For major live events, a &#8220;dedicated&#8221; network connection helps address these challenges.</p><p>A dedicated connection links the event location to the central processing infrastructure without relying on the public internet. It provides more predictable bandwidth &amp; network performance.</p><p>Yet it cannot eliminate every source of delay/failure. Plus, setting up and managing dedicated network links requires significant effort.</p><p>Cloud providers offer services to simplify this process:</p><ul><li><p><strong>AWS Direct Connect:</strong> Provides dedicated network connectivity between your infrastructure and AWS.</p></li><li><p><strong>Azure ExpressRoute:</strong> Provides private connectivity between your infrastructure and Microsoft Azure.</p></li></ul><p>Both services help establish more predictable network paths for transferring live video to cloud infrastructure.</p><p>But network connectivity is only part of the solution. You also need a protocol to transport a continuous video stream efficiently.</p><p>Let&#8217;s look at the networking protocols used for live video transfer&#8230;</p><h3><span>Networking Protocols</span></h3><p>Live video requires a continuous data flow between the event location &amp; central processing servers.</p><p>In a typical HTTP request, the client sends a request, receives a response, and completes the exchange. But repeatedly establishing connections for a continuous video stream would introduce unnecessary overhead.</p><p>A &#8220;persistent&#8221; connection helps avoid this problem.</p><p>WebSockets might seem like an obvious choice&#8230; They support persistent, bidirectional communication between clients and servers.</p><p>But WebSockets do not provide built-in media transport features such as video codecs, timestamps, or packet-loss recovery. You&#8217;d need to implement additional mechanisms to handle live audio &amp; video reliably.</p><p>WebSockets are better suited to features such as live chat &amp; viewer reactions.</p><p>For transporting live video, we need protocols designed specifically for media delivery.</p><h4>Real-Time Messaging Protocol</h4><p>Real-Time Messaging Protocol (<strong>RTMP</strong>) is a protocol designed to transfer audio &amp; video streams with low latency.</p><p>(Historically, RTMP supported video delivery from streaming servers to client devices through Adobe Flash Player. But Flash Player reached end of life in December 2020.)</p><p>RTMP remains widely used for the first phase of live streaming: transferring video from the event location to ingestion servers.</p><p><em>Here&#8217;s how it works:</em></p><ol><li><p>Client initiates a handshake with the RTMP server.</p></li><li><p>After the handshake succeeds, the client establishes a persistent connection.</p></li><li><p>RTMP divides media stream into smaller chunks &amp; transfers them over connection.</p></li><li><p>The server receives the chunks &amp; reconstructs the media stream for further processing.</p></li></ol><p>RTMP runs over TCP, which provides reliable, ordered data delivery.</p><p>Yet TCP introduces a trade-off:</p><p>When packets get lost, TCP retransmits them. Subsequent data may be delayed until the missing data arrives, a behavior known as <em>head-of-line blocking</em>. And this could increase latency or cause playback interruptions under poor network conditions.</p><p>A dedicated network connection helps reduce congestion and packet loss, thus making RTMP a practical choice for transporting live video from the event venue.</p><p>Another limitation is codec compatibility:</p><p>Traditional RTMP implementations primarily support H.264 video, while H.265 support requires newer extensions and compatible software.</p><h4>Secure Reliable Transport</h4><p>Secure Reliable Transport (<strong>SRT</strong>) is a protocol designed for low-latency video transport.</p><p>SRT runs over UDP. SRT adds reliability mechanisms on top of UDP, including packet-loss detection, selective retransmission, and configurable latency buffers.</p><p><em>Here&#8217;s the key difference:</em></p><ul><li><p><strong>RTMP:</strong> Uses TCP to deliver data reliably and in order. Packet loss can delay subsequent data while TCP recovers missing packets.</p></li><li><p><strong>SRT:</strong> Uses UDP with selective retransmission. It attempts to recover lost packets within a configured latency window and avoids waiting indefinitely for packets that arrive too late.</p></li></ul><p>This makes SRT useful when transferring live video over networks with fluctuating bandwidth or packet loss.</p><p>Also RTMP has broad support for existing streaming hardware &amp; software. But SRT support depends on the capabilities of the encoder, decoder, and streaming infrastructure.</p><p>Both protocols primarily transport video from the source to the streaming backend.</p><p>But delivering video from the backend to <em>millions</em> of viewers requires a different approach&#8230;</p><h4>HTTP Live Streaming</h4><p>HTTP Live Streaming (<strong>HLS</strong>) is a streaming protocol by Apple.</p><p>Instead of maintaining a continuous media connection between the streaming server and every viewer, HLS divides encoded video into smaller media segments.</p><p><em>Here&#8217;s what a typical HLS workflow looks like:</em></p><ol><li><p>The encoder continuously produces compressed video.</p></li><li><p>A packager divides the encoded stream into media segments, typically 2 to 4 seconds long.</p></li><li><p>The system creates a playlist, called a manifest, containing references to available segments.</p></li><li><p>The video player downloads the playlist and requests segments over HTTP.</p></li><li><p>As new segments become available, the player continues downloading them to maintain playback.</p></li></ol><p>The camera does not need to stop recording after each segment. Video capture and encoding continue while the packager creates segments from the incoming stream.</p><p>Yet segment duration introduces an important trade-off.</p><p>Shorter segments could reduce live-stream latency but increase request overhead &amp; place extra demands on the streaming infrastructure. Longer segments reduce request frequency but increase playback delay.</p><p>Low-Latency HLS extends this approach by letting players access partial segments before a segment is complete.</p><p>(Netflix uses HLS with 2-second segments for its live streaming architecture.)</p><p>HLS also offers broad compatibility with modern playback devices and CDNs.</p><p>For our architecture, we can use RTMP/SRT to transport video from the event location to the central processing servers, then use HLS to distribute the processed stream to viewers.</p><p>If ultra-low-latency playback becomes a requirement, we should evaluate alternatives such as WebRTC or specialized low-latency delivery protocols. SRT remains a strong option for contribution feeds, but it is not a direct replacement for HLS across millions of consumer devices.</p><h3>Video Delivery Workflow</h3><p>Let&#8217;s summarize the first phase of our architecture:</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!U0Nj!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3d1fc7af-d3e8-45b1-97ab-0b064f7b54ab_572x145.png 424w, https://substackcdn.com/image/fetch/$s_!U0Nj!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3d1fc7af-d3e8-45b1-97ab-0b064f7b54ab_572x145.png 848w, https://substackcdn.com/image/fetch/$s_!U0Nj!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3d1fc7af-d3e8-45b1-97ab-0b064f7b54ab_572x145.png 1272w, https://substackcdn.com/image/fetch/$s_!U0Nj!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3d1fc7af-d3e8-45b1-97ab-0b064f7b54ab_572x145.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!U0Nj!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3d1fc7af-d3e8-45b1-97ab-0b064f7b54ab_572x145.png" width="572" height="145" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/3d1fc7af-d3e8-45b1-97ab-0b064f7b54ab_572x145.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:145,&quot;width&quot;:572,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!U0Nj!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3d1fc7af-d3e8-45b1-97ab-0b064f7b54ab_572x145.png 424w, https://substackcdn.com/image/fetch/$s_!U0Nj!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3d1fc7af-d3e8-45b1-97ab-0b064f7b54ab_572x145.png 848w, https://substackcdn.com/image/fetch/$s_!U0Nj!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3d1fc7af-d3e8-45b1-97ab-0b064f7b54ab_572x145.png 1272w, https://substackcdn.com/image/fetch/$s_!U0Nj!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3d1fc7af-d3e8-45b1-97ab-0b064f7b54ab_572x145.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div></div></div></div><figcaption class="image-caption">Video journey from cameras to centralized servers</figcaption></figure></div><ol><li><p><strong>Video capture:</strong> Multiple high-resolution cameras record the event from different angles.</p></li><li><p><strong>Production Control Room:</strong> Camera feeds reach the PCR or operations center, where the production system inspects and selects the video streams for further processing. </p></li><li><p><strong>Encoding and compression:</strong> Once the production feed is finalized, video encoders compress it using codecs such as Advanced Video Coding (AVC/H.264) or High Efficiency Video Coding (HEVC/H.265). This reduces the bandwidth required for transmission. Encoding can happen on-site with hardware or software encoders, or through cloud-based services. For example, AWS Elemental MediaLive provides live video encoding &amp; processing in the cloud.</p></li><li><p><strong>Video transfer:</strong> The compressed stream travels from the event location to centralized processing infrastructure over a dedicated network connection, using a protocol such as RTMP/SRT.</p></li></ol><h3>Choosing Video Segment Size</h3><p>One final consideration is how the continuous video stream gets divided into segments for downstream processing &amp; delivery.</p><p><em>The segment duration affects several factors:</em></p><ul><li><p><strong>Live-stream latency:</strong> Shorter segments reduce the delay between the live event and viewer playback.</p></li><li><p><strong>Encoding and packaging:</strong> Segment boundaries must align with the encoded video structure, and the system needs time to prepare segments for delivery.</p></li><li><p><strong>Network overhead:</strong> Smaller segments generate more frequent HTTP requests, increasing request volume across clients, servers, and CDNs.</p></li></ul><p>For conventional HTTP-based live streaming, segment durations range from around 2 to 10 seconds, depending on the configuration.</p><p>The appropriate value depends on the event&#8217;s latency requirements, encoding configuration, network conditions, and geographic distribution of viewers. Once the video gets transferred to the central servers, the next phase prepares it for playback across different devices and network speeds.</p><p>Let&#8217;s examine video processing next&#8230;</p><h3><span>Video Processing</span></h3><p>Once the video stream reaches the central server, next step is to prepare it for different devices &amp; network conditions.</p><p>A viewer watching on a mobile phone with a slow internet connection has different requirements from someone watching on a 4K television with high-speed broadband.</p><p>The system must support both without interrupting playback.</p><p><em>Transcoding</em> converts the incoming video stream into multiple encoded versions, each with a specific resolution, bitrate, and codec.</p><p>Two important parameters determine the output:</p><ul><li><p><strong>Bitrate:</strong> The amount of video data transmitted per second, typically measured in megabits per second (Mbps). Higher bitrates allow better video quality but require more bandwidth.</p></li><li><p><strong>Resolution:</strong> The number of pixels in each video frame, such as 4K, 1080p, 720p, or 480p. Higher resolutions contain more visual detail but typically require higher bitrates to maintain quality.</p></li></ul><p>For example, an MP4 video could be transcoded into many quality levels:</p><ul><li><p><strong>High bitrate:</strong> 4K, 1080p, or 720p.</p></li><li><p><strong>Medium bitrate:</strong> 1080p, 720p, or 480p.</p></li><li><p><strong>Low bitrate:</strong> 480p, 360p, or 144p.</p></li></ul><p>Resolution &amp; bitrate are separate parameters, so each output needs its own bitrate configuration. These versions form an adaptive bitrate (<strong>ABR</strong>) ladder.</p><p>The video player then selects the appropriate version based on the viewer&#8217;s available bandwidth, device capabilities, and playback conditions.</p><p>i.e., someone watching on a fast Wi-Fi connection might receive a 1080p stream. If their connection slows down, player can switch to 720p/480p to reduce buffering.</p><h4>Managed Services vs Custom Transcoding</h4><p>At this stage, you must decide whether to build your own transcoding infrastructure or use a managed service.</p><p>Cloud providers offer services for video encoding &amp; format conversion.</p><p>AWS Elemental MediaConvert converts video files into different formats, resolutions, and bitrates. It&#8217;s designed for file-based video processing.</p><p>For continuous live streams, AWS Elemental MediaLive is an appropriate managed service because it supports real-time video encoding &amp; transcoding.</p><p>The output is a collection of video renditions ready for packaging &amp; distribution.</p><h4>Video Packaging: HLS and DASH</h4><p>After transcoding, the system must package the video into a format supported by viewers&#8217; devices.</p><p>Here are 2 popular protocols:</p><ul><li><p><strong>HTTP Live Streaming (HLS):</strong> Widely supported across modern streaming platforms.</p></li><li><p><strong>Dynamic Adaptive Streaming over HTTP (DASH):</strong> An open standard for delivering adaptive video streams over HTTP.</p></li></ul><p>Both support adaptive bitrate streaming, thus allowing video players to switch between quality levels as network conditions change.</p><p>HLS is commonly used on Apple devices, while many other players and platforms support DASH. Also HLS works on many non-Apple devices, so the choice depends on player and platform compatibility.</p><p>Let&#8217;s examine how HLS works&#8230;</p><h4>How HLS Organizes Video Segments</h4><p>HLS divides the encoded video into smaller media segments and generates playlists that tell the player where to find them.</p><p>Each segment is typically stored in one of two formats:</p><ul><li><p><code>.ts</code><strong>:</strong> MPEG-2 Transport Stream, a traditional HLS segment format.</p></li><li><p><code>.m4s</code><strong>:</strong> Fragmented MP4, commonly used with modern HLS implementations.</p></li></ul><p>The system also generates two types of playlist files, both using the <code>.m3u8</code> extension.</p><p><strong>1. Master Playlist</strong></p><p>The master playlist lists the available video renditions, such as 4K, 1080p &amp; 720p.</p><p>Each entry includes required bandwidth, resolution, and codec.</p><p>The player uses this information to identify compatible renditions and select an appropriate quality level.</p><p><strong>2. Media Playlist</strong></p><p>Each rendition has its own media playlist.</p><p>This playlist contains an ordered list of video segments, along with their durations and locations.</p><p>For a live event, the playlist updates as new segments become available. The player repeatedly fetches the latest playlist &amp; downloads appropriate segments.</p><p>This structure allows the player to switch between quality levels during playback without restarting the stream.</p><h4>Video Storage</h4><p>The system also needs to store video for later playback.</p><p>We can retain both the original ingested video &amp; transcoded outputs.</p><p>This serves 2 purposes:</p><ul><li><p><strong>Original video:</strong> Preserves source material for future processing or re-encoding.</p></li><li><p><strong>Transcoded video:</strong> Allows viewers to replay the event without requiring the system to transcode it again.</p></li></ul><p>Object storage is a common choice for storing these large video files.</p><p>Amazon Simple Storage Service (S3) stores the original recordings, transcoded renditions, media segments, and manifests.</p><p>We also maintain a metadata store to map each event and video rendition to its corresponding file locations in Amazon S3.</p><p>When someone wants to watch a completed event, the system uses this metadata to locate the appropriate video assets.</p><h4>Digital Rights Management</h4><p>Live sports content typically comes with strict licensing &amp; distribution requirements.</p><p>The platform must protect video streams from unauthorized access &amp; playback.</p><p>Digital Rights Management (<strong>DRM</strong>) encrypts video content and requires an authorized player to obtain a valid license before decrypting and playing it.</p><p>Here are the popular DRM systems:</p><ul><li><p><strong>Google Widevine:</strong> Supported across Android &amp; many browser-based playback environments.</p></li><li><p><strong>Microsoft PlayReady:</strong> Used across various streaming devices, browsers &amp; television platforms.</p></li><li><p><strong>Apple FairPlay Streaming:</strong> Used to protect HLS content on supported Apple devices.</p></li></ul><p>AWS Elemental MediaPackage can package live video into HLS/DASH formats and integrate with DRM systems to encrypt content.</p><p>But DRM license issuance and key management require integration with a compatible DRM provider or license service.</p><p>This ensures authorized viewers access protected streams through supported players.</p><h4>From Video Processing to Distribution</h4><p>Let&#8217;s summarize what happens after the live video reaches the central processing servers.</p><ol><li><p><strong>Transcoding:</strong> The incoming stream gets converted into multiple resolutions and bitrates to support different devices &amp; network conditions.</p></li><li><p><strong>Packaging:</strong> Each rendition gets divided into media segments and organized into HLS/DASH manifests.</p></li><li><p><strong>Storage:</strong> Original recordings, transcoded video, and metadata get stored for live delivery &amp; future playback.</p></li><li><p><strong>Content protection:</strong> DRM encryption and license services help restrict playback to authorized viewers.</p></li></ol><p>The processed video is now ready for distribution.</p><p>But our system must deliver it to 50 million concurrent viewers without overwhelming the central servers. This is where a Content Delivery Network (<strong>CDN</strong>) becomes essential.</p><p>Let&#8217;s examine video distribution next&#8230;</p><h3><span>Video Distribution</span></h3><p>The video is now encoded, transcoded &amp; packaged.</p><p>The next challenge is delivering it to 50 million concurrent viewers.</p><p>Sending every video segment directly from the central servers would require enormous bandwidth &amp; put massive pressure on the infrastructure.</p><p>A CDN caches video segments at edge locations closer to viewers. When thousands of people in the same region watch a match, they receive the same video segments from a nearby edge server instead of repeatedly requesting them from the origin.</p><p>But caching introduces a tradeoff:</p><p>Cached content could become stale, while live streaming requires viewers to receive new segments with minimal delay. The system must balance cache efficiency with live-stream freshness.</p><p>Live video segments are typically immutable once published. So this makes them suitable for caching, while frequently updated manifests require shorter cache durations.</p><h4>How CDN Distribution Works</h4><p>Once the video gets transcoded &amp; packaged, the distribution system makes video segments and manifests available through the CDN.</p><p>In a typical CDN architecture, edge servers retrieve content from the origin if it&#8217;s not already cached. Subsequent requests for the same segment get then served directly from the edge.</p><p><em>Here&#8217;s how the process works:</em></p><ol><li><p><strong>Video packaging:</strong> The processing system generates video segments and HLS/DASH manifests.</p></li><li><p><strong>Origin storage:</strong> Packaged content becomes available on an origin server or object storage service.</p></li><li><p><strong>CDN retrieval:</strong> When an edge location receives a request for uncached content, it fetches the required segment from the origin or upstream CDN cache.</p></li><li><p><strong>Edge caching:</strong> CDN stores the segment temporarily so nearby viewers can access it without another origin request.</p></li><li><p><strong>Video delivery:</strong> Viewers download the segments from edge locations through Internet Service Providers (<strong>ISPs</strong>) or mobile networks.</p></li></ol><p>For live sports, viewer demand concentrates in specific geographic regions.</p><p>The distribution system must account for these traffic patterns &amp; ensure sufficient CDN capacity in regions where large audiences are expected.</p><p>Here are CDN solutions that support this architecture:</p><ul><li><p><strong>Amazon CloudFront:</strong> delivers content through a globally distributed network of edge locations.</p></li><li><p><strong>Cloudflare CDN:</strong> A distributed caching &amp; content delivery service.</p></li><li><p><strong>Netflix Open Connect:</strong> Netflix&#8217;s own CDN infrastructure, designed to deliver video efficiently to its viewers.</p></li></ul><p>The goal is to serve video from locations close to viewers while reducing traffic to the central processing infrastructure.</p><h4>Video Pipeline Coordination</h4><p>Consider 3 major phases we&#8217;ve covered:</p><ul><li><p>Video ingestion</p></li><li><p>Video processing</p></li><li><p>Video distribution</p></li></ul><p>These phases depend on one another&#8230;</p><p>The system must receive the incoming video before it can process it. Transcoding begins as encoded video becomes available, and distribution starts as soon as the required segments and manifests are ready.</p><p>For live streaming, these stages operate continuously as a pipeline rather than waiting for the entire event recording to finish.</p><p>But failures can occur at any stage&#8230;</p><p>The system might generate the 720p rendition while the 1080p transcoding process fails.</p><p>An <em>orchestration layer</em> helps coordinate the services involved in the pipeline. Its responsibilities include:</p><ul><li><p><strong>Task coordination:</strong> Ensure ingestion, transcoding, packaging, and distribution happen in the correct order.</p></li><li><p><strong>Failure handling:</strong> Detect failed processing tasks &amp; trigger necessary recovery actions.</p></li><li><p><strong>Dependency management:</strong> Track which outputs are ready for downstream services.</p></li><li><p><strong>Distribution readiness:</strong> Ensure video segments &amp; manifests become available to CDN as processing progresses.</p></li></ul><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!kJwj!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F018314da-d920-408f-a688-d486c84e07ca_697x203.png 424w, https://substackcdn.com/image/fetch/$s_!kJwj!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F018314da-d920-408f-a688-d486c84e07ca_697x203.png 848w, https://substackcdn.com/image/fetch/$s_!kJwj!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F018314da-d920-408f-a688-d486c84e07ca_697x203.png 1272w, https://substackcdn.com/image/fetch/$s_!kJwj!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F018314da-d920-408f-a688-d486c84e07ca_697x203.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!kJwj!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F018314da-d920-408f-a688-d486c84e07ca_697x203.png" width="697" height="203" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/018314da-d920-408f-a688-d486c84e07ca_697x203.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:203,&quot;width&quot;:697,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!kJwj!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F018314da-d920-408f-a688-d486c84e07ca_697x203.png 424w, https://substackcdn.com/image/fetch/$s_!kJwj!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F018314da-d920-408f-a688-d486c84e07ca_697x203.png 848w, https://substackcdn.com/image/fetch/$s_!kJwj!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F018314da-d920-408f-a688-d486c84e07ca_697x203.png 1272w, https://substackcdn.com/image/fetch/$s_!kJwj!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F018314da-d920-408f-a688-d486c84e07ca_697x203.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div></div></div></div><figcaption class="image-caption">Video Ingestion to Video Distribution Pipeline</figcaption></figure></div><p>If the 1080p rendition fails, the system can attempt recovery while continuing to deliver other available renditions.</p><p>This prevents a failure in one quality level from unnecessarily interrupting the entire live stream.</p><h4>Bringing Video Closer to Viewers</h4><p>The network path between origin infrastructure &amp; CDN locations affects delivery performance.</p><p>CDN providers use private backbone networks, peering connections, and optimized routing to transfer content efficiently.</p><p>Yet the exact network path depends on the CDN provider and deployment architecture. Traffic does not always travel only over dedicated private connections.</p><p>Once the content reaches an edge server, viewers access it through their ISPs or mobile networks.</p><p>This final delivery path uses the public internet.</p><p>To reduce latency and network congestion, CDN providers place caching infrastructure close to major concentrations of viewers.</p><p>Netflix takes this approach with <em>Open Connect</em>.</p><p>Its Open Connect Appliances are caching servers that can be deployed inside participating ISP networks. This lets Netflix serve content closer to subscribers while reducing traffic across upstream networks.</p><p>AWS offers a related capability through Amazon CloudFront Embedded Points of Presence (<strong>POPs</strong>).</p><p>These smaller CDN locations extend content delivery infrastructure deeper into ISP networks, bringing cached content closer to end users.</p><p>Both approaches help reduce the distance video data must travel before reaching viewers.</p><p>For a live event with 50 million concurrent viewers, this distributed architecture is essential for handling traffic without overwhelming the origin servers.</p><p>The CDN delivers the video segments&#8230;The next step is understanding how a viewer&#8217;s device discovers the stream, selects the appropriate quality &amp; plays video.</p><p>Let&#8217;s examine video playback next&#8230;</p><h3><span>Video Playback</span></h3><p>The player must locate the live stream, select the right video quality &amp; download the segments needed for playback.</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!mUY2!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F951b3ef4-778b-4b6f-b455-294e9db33dfc_315x152.png 424w, https://substackcdn.com/image/fetch/$s_!mUY2!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F951b3ef4-778b-4b6f-b455-294e9db33dfc_315x152.png 848w, https://substackcdn.com/image/fetch/$s_!mUY2!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F951b3ef4-778b-4b6f-b455-294e9db33dfc_315x152.png 1272w, https://substackcdn.com/image/fetch/$s_!mUY2!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F951b3ef4-778b-4b6f-b455-294e9db33dfc_315x152.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!mUY2!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F951b3ef4-778b-4b6f-b455-294e9db33dfc_315x152.png" width="423" height="204.11428571428573" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/951b3ef4-778b-4b6f-b455-294e9db33dfc_315x152.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:152,&quot;width&quot;:315,&quot;resizeWidth&quot;:423,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!mUY2!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F951b3ef4-778b-4b6f-b455-294e9db33dfc_315x152.png 424w, https://substackcdn.com/image/fetch/$s_!mUY2!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F951b3ef4-778b-4b6f-b455-294e9db33dfc_315x152.png 848w, https://substackcdn.com/image/fetch/$s_!mUY2!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F951b3ef4-778b-4b6f-b455-294e9db33dfc_315x152.png 1272w, https://substackcdn.com/image/fetch/$s_!mUY2!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F951b3ef4-778b-4b6f-b455-294e9db33dfc_315x152.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div></div></div></div><figcaption class="image-caption">Video Playback Request Flow</figcaption></figure></div><p><em>Here&#8217;s the end-to-end request workflow:</em></p><ol><li><p><strong>Request the stream:</strong> App/site sends a request to the playback service, along with the event details.</p></li><li><p><strong>Retrieve the master manifest:</strong> Playback service provides the master manifest URL, which the player uses to retrieve the available video quality options.</p></li><li><p><strong>Select video quality:</strong> The player chooses an initial quality level based on the available network bandwidth &amp; device capabilities.</p></li><li><p><strong>Retrieve the media manifest:</strong> The player requests the media manifest for the selected quality. This manifest contains references to video segments available through CDN.</p></li><li><p><strong>Download &amp; play:</strong> The player downloads the video segments from the CDN, buffers enough data to begin playback, and starts displaying the live stream.</p></li></ol><p>NOTE: We&#8217;ve intentionally excluded services responsible for user authentication and authorization, which control access to specific content. Our focus is on delivering live video to viewers with minimal delay.</p><h4>How Adaptive Bitrate Streaming Works</h4><p>The viewer&#8217;s network conditions could change throughout the event&#8230; Someone watching on a mobile device might move from a fast Wi-Fi connection to a <em>slower</em> cellular network.</p><p>If the player keeps downloading high-bitrate video segments, playback could stall because the network cannot deliver data fast enough.</p><p>The video player continuously monitors factors such as available network throughput and playback buffer levels with adaptive bitrate (<strong>ABR</strong>) streaming. It then uses this information to select an appropriate video rendition.</p><ul><li><p><strong>When bandwidth increases:</strong> Player switches to a higher-bitrate rendition for better video quality.</p></li><li><p><strong>When bandwidth decreases:</strong> The player switches to a lower-bitrate rendition to reduce buffering.</p></li><li><p><strong>When network conditions stabilize:</strong> Player continues downloading segments at a suitable quality level.</p></li></ul><p>These switches typically happen at segment boundaries, allowing the player to change video quality without restarting playback.</p><p>The goal is to maintain smooth playback while delivering the highest practical video quality for each viewer.</p><p>We&#8217;ve now covered the complete journey from cameras at the event venue to video playback on viewers&#8217; devices.</p><p>But supporting 50 million concurrent viewers creates extra challenges, especially when traffic increases sharply during a major live event.</p><p>Let&#8217;s examine how the architecture handles these situations&#8230;</p><div><hr></div><div class="callout-block" data-callout="true"><p style="text-align: center;"><em><strong><span data-color="#3a7bf9" style="color: rgb(58, 123, 249);">Reminder: this is a teaser of the subscriber-only newsletter, exclusive to my golden members.</span></strong></em></p><p>When you upgrade, you&#8217;ll get:</p><ul><li><p><strong>High-level architecture of real-world systems.</strong></p></li><li><p>Deep dive into how popular real-world systems work.</p></li><li><p><strong>How real-world systems handle scale, reliability &amp; performance.</strong></p></li></ul><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/subscribe&quot;,&quot;text&quot;:&quot;Unlock Full Access&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://newsletter.systemdesign.one/subscribe"><span>Unlock Full Access</span></a></p><p>(If this newsletter has helped you become a better software engineer, consider subscribing to support my work.)</p></div><div><hr></div><h2><span>Live Streaming Operations</span></h2>
      <p>
          <a href="https://newsletter.systemdesign.one/p/live-streaming-system-design">
              Read more
          </a>
      </p>
   ]]></content:encoded></item><item><title><![CDATA[How LLM Inference Works]]></title><description><![CDATA[#199: The Secret to LLM&#8217;s Superpower]]></description><link>https://newsletter.systemdesign.one/p/how-llm-inference-works</link><guid isPermaLink="false">https://newsletter.systemdesign.one/p/how-llm-inference-works</guid><dc:creator><![CDATA[Neo Kim]]></dc:creator><pubDate>Tue, 03 Nov 2026 11:41:27 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/f480de20-bc62-4de6-ad88-25228bca603a_1280x720.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<div class="callout-block" data-callout="true"><h2><strong><a href="https://inngest.link/jtH41MP">Don&#8217;t go back to the starting line</a></strong></h2><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://inngest.link/jtH41MP" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!n1yx!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9c57394c-2437-46cf-b50d-9636a4facbe3_2048x992.png 424w, https://substackcdn.com/image/fetch/$s_!n1yx!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9c57394c-2437-46cf-b50d-9636a4facbe3_2048x992.png 848w, https://substackcdn.com/image/fetch/$s_!n1yx!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9c57394c-2437-46cf-b50d-9636a4facbe3_2048x992.png 1272w, https://substackcdn.com/image/fetch/$s_!n1yx!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9c57394c-2437-46cf-b50d-9636a4facbe3_2048x992.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!n1yx!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9c57394c-2437-46cf-b50d-9636a4facbe3_2048x992.png" width="1456" height="705" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/9c57394c-2437-46cf-b50d-9636a4facbe3_2048x992.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:705,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:&quot;https://inngest.link/jtH41MP&quot;,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!n1yx!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9c57394c-2437-46cf-b50d-9636a4facbe3_2048x992.png 424w, https://substackcdn.com/image/fetch/$s_!n1yx!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9c57394c-2437-46cf-b50d-9636a4facbe3_2048x992.png 848w, https://substackcdn.com/image/fetch/$s_!n1yx!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9c57394c-2437-46cf-b50d-9636a4facbe3_2048x992.png 1272w, https://substackcdn.com/image/fetch/$s_!n1yx!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9c57394c-2437-46cf-b50d-9636a4facbe3_2048x992.png 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><strong><a href="https://inngest.link/jtH41MP">Inngest</a></strong> makes your code durable by default.</p><p>Simply wrapping your workflows in Inngest&#8217;s <code>step.run</code> provides automatic step-level checkpointing &amp; retries, with zero infra to manage.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://inngest.link/jtH41MP&quot;,&quot;text&quot;:&quot;Keep running with Inngest&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://inngest.link/jtH41MP"><span>Keep running with Inngest</span></a></p><p>(Thanks to <a href="https://inngest.link/jtH41MP">Inngest</a> for partnering on this newsletter.)</p><p><em>Onward&#8230;</em></p></div><div><hr></div><p>You type a question into ChatGPT:</p><blockquote><p><em>&#8220;Why is the sky blue?&#8221;</em></p></blockquote><p>&#8230;  and hit Enter!</p><p>A moment later, the answer starts appearing.</p><p>From your side, that is the entire interaction&#8230;</p><p>But behind the screen, an entire &#8220;inference&#8221; system has already started working.</p><p>And it has 3 HARD problems to solve:</p><ul><li><p><em>How to generate tokens quickly when each step needs access to a huge model?</em></p></li><li><p><em>How to serve many users without running out of GPU memory &amp; capacity?</em></p></li><li><p><em>How to make responses fast for users without wasting expensive hardware?</em></p></li></ul><p>To answer those questions, we&#8217;ll follow one request from the moment you hit Enter to the moment the answer reaches your screen.</p><p>Along the way, we will see how tokenization, prefill, decode, KV cache, batching, memory management, and distributed serving all fit together.</p><p>Let&#8217;s start with the request itself&#8230;</p><div><hr></div><p></p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption"><span data-color="#3a7bf9" style="color: rgb(58, 123, 249);">Get my AI engineering playbook for FREE on newsletter signup:</span></p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><ul><li><p><em><a href="https://newsletter.systemdesign.one/p/how-llm-inference-works/?action=share">Share this letter</a> &amp; I&#8217;ll send you some rewards for the referrals.</em></p></li></ul><div><hr></div><h2><strong>Inference</strong></h2><p>A model goes through two main stages: <em><strong>training &amp; inference</strong></em>.</p><p>During <em>training</em>, the model learns its parameters--weights. It processes large amounts of data &amp; repeatedly adjusts those weights to get better at predicting what comes next. Once a trained model gets deployed, those weights are then typically fixed!</p><p>Then the <em>inference</em> begins:</p><p>When you send the model a prompt, it does not retrain itself on your question. It uses the weights it already learned to produce a response. This process is called inference.</p><blockquote><p>Training creates the model&#8230;Inference runs it.</p></blockquote><p>(This distinction becomes important at scale.)</p><p>Training requires enormous computing power, but it occurs during model development. Inference occurs every time someone sends a request. The same trained checkpoint can therefore perform inference again and again for as long as it is being served.</p><p>Our request is one of those inference jobs.</p><p>But &#8220;Why is the sky blue?&#8221; cannot be handed directly to the model as a string of English text. It first has to be converted into the representation the model expects.</p><p>That begins with tokenization&#8230;</p><div><hr></div><h2><strong>Tokenization</strong></h2><blockquote><p>A token is a piece of text. Sometimes, it is a complete word, and sometimes it is only part of one. Spaces &amp; punctuation also appear as part of tokens.</p></blockquote><p>Large Language Models (<strong>LLMs</strong>) work with tokens, not words.</p><p>So our question, &#8220;Why is the sky blue?&#8221;, could be split like this:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!UVpM!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd8f7f58b-dfa1-4825-ae6d-847be59b3dda_1642x699.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!UVpM!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd8f7f58b-dfa1-4825-ae6d-847be59b3dda_1642x699.png 424w, https://substackcdn.com/image/fetch/$s_!UVpM!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd8f7f58b-dfa1-4825-ae6d-847be59b3dda_1642x699.png 848w, https://substackcdn.com/image/fetch/$s_!UVpM!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd8f7f58b-dfa1-4825-ae6d-847be59b3dda_1642x699.png 1272w, https://substackcdn.com/image/fetch/$s_!UVpM!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd8f7f58b-dfa1-4825-ae6d-847be59b3dda_1642x699.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!UVpM!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd8f7f58b-dfa1-4825-ae6d-847be59b3dda_1642x699.png" width="1456" height="620" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/d8f7f58b-dfa1-4825-ae6d-847be59b3dda_1642x699.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:620,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!UVpM!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd8f7f58b-dfa1-4825-ae6d-847be59b3dda_1642x699.png 424w, https://substackcdn.com/image/fetch/$s_!UVpM!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd8f7f58b-dfa1-4825-ae6d-847be59b3dda_1642x699.png 848w, https://substackcdn.com/image/fetch/$s_!UVpM!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd8f7f58b-dfa1-4825-ae6d-847be59b3dda_1642x699.png 1272w, https://substackcdn.com/image/fetch/$s_!UVpM!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd8f7f58b-dfa1-4825-ae6d-847be59b3dda_1642x699.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><div class="callout-block" data-callout="true"><p>[&#8221;Why&#8221;, &#8220; is&#8221;, &#8220; the&#8221;, &#8220; sky&#8221;, &#8220; blue&#8221;, &#8220;?&#8221;]</p></div><p>The exact split depends on the tokenizer.</p><p>Another model may divide the same sentence differently. Each token maps to an integer in the model&#8217;s vocabulary. These integers (token IDs) become the model&#8217;s discrete input before it maps them to learned vector representations.</p><p>That&#8217;s why tokens appear everywhere when you use an LLM.</p><ul><li><p>API pricing get typically measured in tokens.</p></li><li><p>Context windows get measured in tokens.</p></li><li><p>Generation speed gets measured in tokens per second.</p></li></ul><p>The tokenizer still needs a method for deciding where to split the text.</p><p>One common method is Byte Pair Encoding (<strong>BPE</strong><a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-1" href="#footnote-1" target="_self">1</a>). It builds a vocabulary by repeatedly finding common adjacent units &amp; merging them. Frequent patterns could eventually become individual tokens, while less common words often get split into several pieces.</p><p>This gives the tokenizer a useful compromise.</p><p>It does not need a vocabulary containing every possible word. Plus, it does not have to process normal text one character at a time.</p><p>Another tokenization framework you may come across is SentencePiece<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-2" href="#footnote-2" target="_self">2</a>.</p><p>SentencePiece applies algorithms such as BPE/Unigram<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-3" href="#footnote-3" target="_self">3</a> directly to raw text, including whitespace. This makes it useful for languages where spaces are not reliable word boundaries.</p><p>By the end of tokenization, our sentence is no longer just text.</p><p>It has become a sequence of token IDs. Before the model processes those tokens, however, we need to consider a limit.</p><p>That limit is the context window.</p><div><hr></div><div class="captioned-button-wrap" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/p/how-llm-inference-works?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;}" data-component-name="CaptionedButtonToDOM"><div class="preamble"><p class="cta-caption"><em><span data-color="#3a7bf9" style="color: rgb(58, 123, 249);">If something resonated with you today, share this letter. Because one idea, one action, can change everything.</span></em></p></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/p/how-llm-inference-works?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://newsletter.systemdesign.one/p/how-llm-inference-works?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p></div><div><hr></div><h2><strong>Context Window</strong></h2><blockquote><p>It&#8217;s the maximum number of tokens a model works with at one time.</p></blockquote><p>Your visible prompt is not necessarily the only thing taking up this space.</p><p>A chat request also contains earlier messages, system instructions, retrieved documents, tool definitions, or other information the application sends to the model. Depending on the model &amp; API, generated output also needs to fit within the applicable sequence or output limits. This is why long conversations eventually need some form of context management.</p><p>What happens at the limit depends on the application:</p><p>A system could remove older messages, summarize them, retrieve only relevant parts, or reject a request that is too large. The model&#8217;s context window should NOT be confused with a permanent memory of everything that has happened in conversation.</p><p>Context length matters for another reason: <em>longer prompts require more work.</em></p><p>During attention<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-4" href="#footnote-4" target="_self">4</a>, the model uses information from different positions in the sequence to understand how the tokens relate to one another. As the sequence grows, the model has more information to process &amp; store. This increases computation and memory requirements, although the exact cost depends on the model architecture and attention implementation.</p><p>A larger context window also does not mean the model will use every part equally well.</p><p>Information buried in the middle of a long prompt can be harder to retrieve than information closer to the beginning/end.</p><p>Our question is tiny by comparison: <em>&#8220;Why is the sky blue?&#8221;</em></p><p>Once those tokens are inside the available context, the model begins processing them.</p><div><hr></div><h2><strong>Prefill</strong></h2><blockquote><p>Prefill is the stage where the model processes tokens in your prompt before it starts generating an answer.</p></blockquote><p>LLMs are built from many layers stacked on top of one another.</p><p>Each layer takes the representation produced by the previous layer, processes it, and passes a new representation forward. By the time the data reaches final layers, the model has built a richer representation of the prompt.</p><p>Inside these layers is a mechanism called <em>attention</em>.</p><p>Attention helps the model decide which earlier tokens are useful when processing the current token. In our question, for example, when the model processes blue, information from sky could be relevant to understanding what the sentence is asking.</p><p>LLMs used for text generation normally use causal attention.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!W7VO!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9dbd8e22-7a28-4b6c-8ec5-f6613fb60b6e_1480x757.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!W7VO!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9dbd8e22-7a28-4b6c-8ec5-f6613fb60b6e_1480x757.png 424w, https://substackcdn.com/image/fetch/$s_!W7VO!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9dbd8e22-7a28-4b6c-8ec5-f6613fb60b6e_1480x757.png 848w, https://substackcdn.com/image/fetch/$s_!W7VO!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9dbd8e22-7a28-4b6c-8ec5-f6613fb60b6e_1480x757.png 1272w, https://substackcdn.com/image/fetch/$s_!W7VO!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9dbd8e22-7a28-4b6c-8ec5-f6613fb60b6e_1480x757.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!W7VO!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9dbd8e22-7a28-4b6c-8ec5-f6613fb60b6e_1480x757.png" width="1456" height="745" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/9dbd8e22-7a28-4b6c-8ec5-f6613fb60b6e_1480x757.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:745,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!W7VO!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9dbd8e22-7a28-4b6c-8ec5-f6613fb60b6e_1480x757.png 424w, https://substackcdn.com/image/fetch/$s_!W7VO!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9dbd8e22-7a28-4b6c-8ec5-f6613fb60b6e_1480x757.png 848w, https://substackcdn.com/image/fetch/$s_!W7VO!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9dbd8e22-7a28-4b6c-8ec5-f6613fb60b6e_1480x757.png 1272w, https://substackcdn.com/image/fetch/$s_!W7VO!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9dbd8e22-7a28-4b6c-8ec5-f6613fb60b6e_1480x757.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Causal attention means a token uses information from earlier positions, but not from positions ahead of it.</p><p>This preserves the left-to-right structure needed for predicting the next token. During prefill, however, every token in our prompt already exists: &#8220;<em>Why is the sky blue?&#8221;</em>. The GPU can therefore process work across the prompt positions in <em>parallel</em> rather than waiting for each token to be generated first.</p><p>While processing the prompt, attention creates keys &amp; values for each token.</p><p>You can think of the key as information used to determine when a token is relevant. The value contains the information the model can retrieve. The model keeps these keys &amp; values in the KV cache.</p><p>The KV cache will be useful again during generation.</p><p>Storing the keys &amp; values means the model does not need to recompute the same key and value projections for every previous token at every generation step.</p><p>(We will look much more closely at the KV cache later.)</p><p>Once the prompt has passed through the model, it must decide which token comes next.</p><p>The model assigns a raw score to every possible token in its vocabulary. These scores are called <em>logits</em>. A decoding or sampling strategy uses the logits to select the first output token.</p><p>Depending on the strategy, this can mean choosing the strongest candidate or sampling from several likely candidates.</p><p>The work done before the first generated token contributes to a metric called Time to First Token (<strong>TTFT</strong>).</p><p>TTFT measures the time between sending a request and receiving the first part of the response. Prefill contributes to this delay, but it is not the only source. A request can also spend time traveling over the network, waiting for available GPU capacity, or sitting in a queue while the serving system decides which request should run next.</p><p>Once the first output token has been chosen, inference enters a different phase&#8230;</p><div><hr></div><div class="captioned-button-wrap" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/p/how-llm-inference-works?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;}" data-component-name="CaptionedButtonToDOM"><div class="preamble"><p class="cta-caption"><em><span data-color="#3a7bf9" style="color: rgb(58, 123, 249);">Share this letter &amp; get some rewards for the referrals.</span></em></p></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/p/how-llm-inference-works?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://newsletter.systemdesign.one/p/how-llm-inference-works?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p></div><div><hr></div><h2><strong>Decode</strong></h2><blockquote><p>Decode is the stage where the model generates a response one token at a time.</p></blockquote><p>Suppose the response begins: <em>&#8220;The sky appears blue because...&#8221;</em></p><p>The model cannot generate the entire sentence in one shot. It first chooses a token such as &#8216;<em>The</em>&#8217;. That token becomes part of the sequence. The model then uses the updated sequence to choose the next token.</p><p>This is autoregressive generation<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-5" href="#footnote-5" target="_self">5</a>.</p><p>Each newly generated token depends on the tokens before it.</p><p>The process looks roughly like this:</p><div class="callout-block" data-callout="true"><p>The<br>The sky<br>The sky appears<br>The sky appears blue</p></div><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!C1uh!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F59611ae9-3f6a-4b7d-a9bd-497d31180792_1054x512.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!C1uh!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F59611ae9-3f6a-4b7d-a9bd-497d31180792_1054x512.png 424w, https://substackcdn.com/image/fetch/$s_!C1uh!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F59611ae9-3f6a-4b7d-a9bd-497d31180792_1054x512.png 848w, https://substackcdn.com/image/fetch/$s_!C1uh!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F59611ae9-3f6a-4b7d-a9bd-497d31180792_1054x512.png 1272w, https://substackcdn.com/image/fetch/$s_!C1uh!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F59611ae9-3f6a-4b7d-a9bd-497d31180792_1054x512.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!C1uh!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F59611ae9-3f6a-4b7d-a9bd-497d31180792_1054x512.png" width="1054" height="512" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/59611ae9-3f6a-4b7d-a9bd-497d31180792_1054x512.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:512,&quot;width&quot;:1054,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!C1uh!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F59611ae9-3f6a-4b7d-a9bd-497d31180792_1054x512.png 424w, https://substackcdn.com/image/fetch/$s_!C1uh!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F59611ae9-3f6a-4b7d-a9bd-497d31180792_1054x512.png 848w, https://substackcdn.com/image/fetch/$s_!C1uh!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F59611ae9-3f6a-4b7d-a9bd-497d31180792_1054x512.png 1272w, https://substackcdn.com/image/fetch/$s_!C1uh!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F59611ae9-3f6a-4b7d-a9bd-497d31180792_1054x512.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>At each decode step, the model still needs information from tokens it has already processed.</p><p>Instead of recomputing the keys &amp; values for all previous tokens, it reuses them from the KV cache created during prefill. The model computes the key and value for the newest token and adds them to the cache. It then passes the newest token through its layers and produces another set of logits: scores for the possible next token.</p><p>The decoding strategy uses those logits to select the next token.</p><p>The selected token joins the sequence, and another decode step begins. This process continues until the model reaches a stopping condition, such as an end-of-sequence token or a configured output limit.</p><p>During prefill, ALL prompt tokens are already available, so the hardware can process work across many positions in parallel. During ordinary autoregressive decode, the next token does not exist until the previous one has been selected. This sequential dependency makes output generation behave differently from input processing.</p><p>There is also a hardware cost inside this loop.</p><p>Every decode step runs through the model&#8217;s layers &amp; repeatedly accesses a large set of model weights. For large models, moving this data through GPU memory could become a major bottleneck.</p><p>This is where the simple story of &#8220;model predicts the next token&#8221; starts turning into a system design problem&#8230;</p><div><hr></div><h2><strong>Streaming Responses</strong></h2><blockquote><p>Streaming is the process of sending parts of the model&#8217;s response to the user while the rest of the response is still being generated.</p></blockquote><p>We have already seen how decode works&#8230;</p><p>The model generates one token, adds it to the sequence, then generates the next one. Without streaming, the server could wait for the entire decode process to finish before returning the answer.</p><p>That would leave you looking at an empty screen while the model generates the full response.</p><p>Streaming changes when the output reaches you.</p><p>Suppose the model is generating: <em>&#8220;The sky appears blue because...&#8221;</em></p><p>As parts of the response become available, the server could start sending them back to the application. Your browser then displays the beginning while the model is still generating the rest&#8230;</p><p>Streaming does not make the model generate tokens faster.</p><p>It changes when you receive those tokens. For an interactive application such as a chatbot, this makes the response feel more responsive.</p><h3><strong>How the response reaches your browser</strong></h3><p>The server needs a way to keep sending data after the request has started.</p><p>One common option is <em>Server-Sent Events</em> (<strong>SSE</strong>):</p><p>SSE keeps an HTTP connection open so server can continue sending events to the client instead of returning one response and immediately closing the exchange.</p><p>The communication moves in one direction: server sends updates &amp; client receives them.</p><div class="callout-block" data-callout="true"><p>Server &#8594; Client</p></div><p>This works well for text generation because most of the response data travels from the server to the client after you send your prompt.</p><p>Another option is <strong>WebSockets</strong>.</p><p>A WebSocket creates a persistent connection where both sides can send data:</p><div class="callout-block" data-callout="true"><p>Server &#8596; Client</p></div><p>This two-way communication is useful when the client needs to keep sending information while receiving results. A real-time voice assistant is one example because the user can continue sending audio while the server sends information back.</p><p>There is one more detail between decode &amp; what appears on your screen: <em>buffering</em>.</p><p>The server does not have to send every generated token as a separate update. It could collect several tokens and send them together as one chunk.</p><p>For example:</p><div class="callout-block" data-callout="true"><p>Token 1 &#8594; send</p><p>Token 2 &#8594; send</p><p>Token 3 &#8594; send</p></div><p>OR</p><div class="callout-block" data-callout="true"><p>Tokens 1&#8211;3 &#8594; send together</p></div><p>The choice affects how the response feels.</p><p>Sending smaller chunks more frequently makes the response appear smoother. Grouping more tokens into each chunk reduces some of the overhead from repeatedly sending tiny updates.</p><p>So there is a tradeoff between responsiveness &amp; communication overhead.</p><p>Our request has now completed the first part of its journey<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-6" href="#footnote-6" target="_self">6</a>:</p><div class="callout-block" data-callout="true"><p>Text &#8594; Tokens &#8594; Prefill &#8594; Decode &#8594; Stream</p></div><p>We started with: <em>&#8220;Why is the sky blue?&#8221;</em></p><p>We have followed it from ordinary text to the response appearing on the screen. For one request, this process is manageable. A production AI service has to repeat it for many users while their requests compete for the same hardware.</p><p>This is where inference starts becoming a HARD systems problem.</p><div><hr></div><div class="callout-block" data-callout="true"><p style="text-align: center;"><em><strong><span data-color="#3a7bf9" style="color: rgb(58, 123, 249);">Reminder: this is a teaser of the subscriber-only newsletter, exclusive to my golden members.</span></strong></em></p><p>When you upgrade, you&#8217;ll get:</p><ul><li><p><strong>High-level architecture of real-world systems.</strong></p></li><li><p>Deep dive into how popular real-world systems work.</p></li><li><p><strong>How real-world systems handle scale, reliability &amp; performance.</strong></p></li></ul><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/subscribe&quot;,&quot;text&quot;:&quot;Unlock Full Access&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://newsletter.systemdesign.one/subscribe"><span>Unlock Full Access</span></a></p><p>(If this newsletter has helped you become a better software engineer, consider subscribing to support my work.)</p></div><div><hr></div><h2><strong>Memory Problem</strong></h2>
      <p>
          <a href="https://newsletter.systemdesign.one/p/how-llm-inference-works">
              Read more
          </a>
      </p>
   ]]></content:encoded></item><item><title><![CDATA[How to Cut Your Claude Code Token Usage]]></title><description><![CDATA[#198: Part 5 - Claude Code 24/7]]></description><link>https://newsletter.systemdesign.one/p/claude-code-token-usage</link><guid isPermaLink="false">https://newsletter.systemdesign.one/p/claude-code-token-usage</guid><dc:creator><![CDATA[Neo Kim]]></dc:creator><pubDate>Sat, 31 Oct 2026 16:00:47 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/3008c11a-7223-4c72-b555-490d7d392173_1280x720.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p></p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption"><span data-color="#3a7bf9" style="color: rgb(58, 123, 249);">Get my AI engineering playbook for FREE on newsletter signup:</span></p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><ul><li><p><em><a href="https://newsletter.systemdesign.one/p/claude-code-token-usage/?action=share">Share this letter</a> &amp; I&#8217;ll send you some rewards for the referrals.</em></p></li></ul><div><hr></div><p><span>Every Claude session starts with context:</span></p><ul><li><p>conversation,</p></li><li><p><span>your prompts,</span></p></li><li><p>your CLAUDE.md,</p></li><li><p>instructions added by the Anthropic team,</p></li><li><p><span>and everything Claude knows about the task.</span></p></li></ul><p><span>But Claude has a &#8220;limit&#8221;&#8230;</span></p><p><span>Every file you paste, every tool you connect, and every long conversation you keep alive uses part of that limit.</span></p><p><span>So if your context is draining quickly, the simplest &amp; best option is to start a new conversation instead of waiting for Claude to compact everything.</span></p><p><span>In this newsletter, we&#8217;ll cover habits that help you use Claude&#8217;s context more efficiently &amp; wisely.</span></p><p><span>To do that, we won&#8217;t just follow practical tips.</span></p><p><span>(We&#8217;ll also test different approaches &amp; compare the results side by side.)</span></p><p><span>Let&#8217;s get started!</span></p><h4 style="text-align: center;"><strong>&#167;</strong></h4><div class="callout-block" data-callout="true"><h2><a href="https://psxid.figma.com/neo_agent">If you have multiple interests, please take advantage of AI agents</a></h2><p>Software engineers were encouraged to specialize for years.</p><p>Backend. Frontend. Infrastructure.</p><p>But AI agents are making it easier to explore disciplines outside your expertise.</p><p>So how do you actually use them in a real-world workflow?</p><p>I focus on backend engineering, but recently wanted to try full-stack development for a personal project. I needed a consistent design system, polished screens, and all the UI states a frontend needs.</p><p>So I used <strong><a href="https://psxid.figma.com/neo_agent">Figma agent</a></strong> to help review and refine my existing designs.</p><p>I started with my design context and ran the <strong><a href="https://www.figma.com/community/skill/60839/ui-state-expander">/ui-state-expander skill</a></strong> to identify missing loading, empty, error, and interaction states.</p><p>The result was a workflow I could use to move from existing designs toward implementation:</p><ol><li><p>Start with an existing design</p></li><li><p>Give Figma agent your design context</p></li><li><p>Expand missing UI states</p></li><li><p>Review the generated output</p></li><li><p>Refine the designs directly in Figma</p></li><li><p>Build the frontend</p></li></ol><p>I still reviewed the output and made the design decisions myself.</p><p>The agent helped me work through parts of the process where I had less experience. If you&#8217;re a software engineer interested in exploring full-stack development, I recommend you give it a try:</p><p><strong><a href="https://psxid.figma.com/neo_agent"><mark data-color="#ffff00" style="background-color: rgb(255, 255, 0); color: rgb(0, 0, 0);">Try Figma Agent Today &#187;</mark></a></strong></p><p><em>(Thanks to <a href="https://psxid.figma.com/neo_agent">Figma</a> for partnering on this newsletter.)</em></p></div><h4 style="text-align: center;"><strong>&#167;</strong></h4><p>I want to introduce <strong><a href="https://www.learnwithmeai.com/welcome">Gencay</a></strong> as the guest author.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://www.learnwithmeai.com/welcome" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!i2sU!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F79774ca3-8968-45f9-bb13-fb23bc404b4f_1442x668.png 424w, https://substackcdn.com/image/fetch/$s_!i2sU!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F79774ca3-8968-45f9-bb13-fb23bc404b4f_1442x668.png 848w, https://substackcdn.com/image/fetch/$s_!i2sU!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F79774ca3-8968-45f9-bb13-fb23bc404b4f_1442x668.png 1272w, https://substackcdn.com/image/fetch/$s_!i2sU!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F79774ca3-8968-45f9-bb13-fb23bc404b4f_1442x668.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!i2sU!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F79774ca3-8968-45f9-bb13-fb23bc404b4f_1442x668.png" width="1442" height="668" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/79774ca3-8968-45f9-bb13-fb23bc404b4f_1442x668.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:668,&quot;width&quot;:1442,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:&quot;&quot;,&quot;type&quot;:null,&quot;href&quot;:&quot;https://www.learnwithmeai.com/welcome&quot;,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!i2sU!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F79774ca3-8968-45f9-bb13-fb23bc404b4f_1442x668.png 424w, https://substackcdn.com/image/fetch/$s_!i2sU!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F79774ca3-8968-45f9-bb13-fb23bc404b4f_1442x668.png 848w, https://substackcdn.com/image/fetch/$s_!i2sU!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F79774ca3-8968-45f9-bb13-fb23bc404b4f_1442x668.png 1272w, https://substackcdn.com/image/fetch/$s_!i2sU!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F79774ca3-8968-45f9-bb13-fb23bc404b4f_1442x668.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>He has been building with AI since ChatGPT first shipped in 2022.</p><p>For years, he built for clients on Upwork: 7,300+ hours of it.</p><p>Now he builds in the open and shares every piece of it on <strong><a href="https://www.learnwithmeai.com/welcome">LearnAIWithMe</a></strong>, every Monday, Wednesday &amp; Friday.</p><p>Not just theory for its own sake. One real build at a time, from idea to running system.</p><h4 style="text-align: center;"><strong>&#167;</strong></h4><p><em>Here&#8217;s what you&#8217;ll find inside this newsletter:</em></p><ul><li><p><strong>The budget most engineers blow in an hour.</strong> What quietly drains your context window, and how fast your daily and weekly limits go with it.</p></li><li><p><strong>The signal that a conversation is past saving.</strong> When to close it, and the one prompt to run first so nothing important gets lost.</p></li><li><p><strong>A project memory that lives outside the chat.</strong> Nested CLAUDE.md files, Auto Memory, and path-scoped rules, so the conversation itself stays disposable.</p></li><li><p><strong>A setup that gets smarter every week.</strong> The correction, rule, skill loop, and what happens when the setup meant to save tokens becomes the thing spending them.</p></li><li><p><strong>A real audit of a real project.</strong> An actual token bill, a 5-step cleanup, a token-saving skill tested head to head &amp; five skills worth installing.</p></li></ul><p></p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption"><em><span data-color="#3a7bf9" style="color: rgb(58, 123, 249);">Golden members get all letters like these!&#8230;</span></em></p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><p></p><h4 style="text-align: center;"><strong>&#167;</strong></h4><h2><span>Context window is a budget</span></h2><p><span>The context window is Claude&#8217;s working memory.</span></p><p><span>It holds your instructions, files, &amp; entire conversation&#8230; and every model has a &#8220;</span><em><span>fixed&#8221;</span></em><span> size&#8230;</span></p><ul><li><p><span>Opus 5 carries a 1M context on the Max plan,</span></p></li><li><p><span>Fable 5 draws down your usage faster than the older models.</span></p></li></ul><p><span>So the window &amp; your limits are connected.</span></p><p><span>Claude has daily &amp; weekly limits, and some models spend them quicker than others.</span></p><p><span>This is why management matters&#8230;!</span></p><p><span>If you fill the window with noise, your limit drops early, and you sit there waiting for the reset instead of working.</span></p><h4 style="text-align: center;"><strong>&#167;</strong></h4><h2><span>Token saving habits that compound</span></h2><p><span>If you repeat a job in every conversation, put it in a </span><code>CLAUDE.md</code></p><p><em><span>Here is why:</span></em></p><p><span>Even a 1M window fills up, and when it does, Claude compacts the conversation on its own. Compacting is a summary, and summaries drop details. So the instructions you typed at the start may not survive into the rest of the conversation.</span></p><p><span>But a good </span><code>CLAUDE.md</code><span> never gets compacted away, because Claude reads it fresh from disk every session.</span></p><p><span>Plus, do not stretch one conversation forever. The bigger it grows, the more each compact costs, and those compacts eat into the same limit you wanted to save.</span></p><p><span>A new conversation is the </span><em><span>cheapest</span></em><span> fix&#8230;and here is the signal for when to switch:</span></p><p><span>When Claude starts compacting every few messages, or the answers begin forgetting rules you gave an hour ago, the conversation is past saving. Each compact now eats your limit while dropping details, so closing it costs less than keeping it alive.</span></p><p><span>But you do not want to lose what the old one knows, so ask this before you close:</span></p><div class="highlighted_code_block" data-attrs="{&quot;language&quot;:&quot;plaintext&quot;,&quot;nodeId&quot;:&quot;06363aa6-d0d0-4f46-b71d-64b1c9492ba1&quot;}" data-component-name="HighlightedCodeBlockToDOM"><pre class="shiki"><code class="language-plaintext">Summarize this conversation in 5 sentences.
But first, ask me 5 questions to make sure the things I care about end up in the summary.</code></pre></div><p><span>Then paste the summary into the new conversation &amp; you continue where you left off at a fraction of the cost.</span></p><p><span>These three commands help you manage the window without leaving it:</span></p><ul><li><p><code>/clear</code><span> resets the context to zero.</span></p></li><li><p><code>/compact</code><span> condenses the conversation, and you steer it with an instruction like </span><code>/compact</code><span> focus on the code changes.</span></p></li><li><p><code>/context</code><strong><span> </span></strong><span>shows what is sitting in your window right now, so you see the problem before it costs you.</span></p></li></ul><h4 style="text-align: center;"><strong>&#167;</strong></h4><h2><span>Grow your project in MD files, not in the chat</span></h2><p><span>A big project inside one chat gets heavy.</span></p><p><span>Some users hold onto the conversation because valuable information lives there, and they are afraid to lose it.</span></p><p><span>The better way is a folder &amp; an </span><code>MD</code><span> file.</span></p><p><span>Add a folder to the conversation, and then use this:</span></p><div class="highlighted_code_block" data-attrs="{&quot;language&quot;:&quot;plaintext&quot;,&quot;nodeId&quot;:&quot;c199807f-36d9-47be-95fb-243edf8190ef&quot;}" data-component-name="HighlightedCodeBlockToDOM"><pre class="shiki"><code class="language-plaintext">Create a CLAUDE.md for this project.

Make sure to include the decisions we made in this conversation
and the rules I gave you.</code></pre></div><p><span>Now the knowledge lives in the file instead of the chat.</span></p><p><span>Every time you open this project, the saved context comes with it, and the conversation itself stays disposable.</span></p><p><span>You could even switch models/tools, and the work survives because the files are the project.</span></p><h4 style="text-align: center;"><strong>&#167;</strong></h4><h2><span>Memory at scale</span></h2><p><span>Let&#8217;s say your project is big, with tens of folders.</span></p><p><span>A nested </span><code>CLAUDE.md</code><span> system is a good road here.</span></p><p><span>Your main </span><code>CLAUDE.md</code><span> stays short &amp; it says something like this:</span></p><div class="highlighted_code_block" data-attrs="{&quot;language&quot;:&quot;plaintext&quot;,&quot;nodeId&quot;:&quot;3e4ef337-fbb4-4baf-909a-734b6c051538&quot;}" data-component-name="HighlightedCodeBlockToDOM"><pre class="shiki"><code class="language-plaintext">This project has 10 folders.
Each folder holds its own CLAUDE.md explaining it.
Read the nested file only when you work inside that folder.</code></pre></div><p><span>And each folder carries its own </span><code>CLAUDE.md</code><span> with the details.</span></p><p><span>The gain is double&#8230;</span></p><p><span>In a normal conversation, the nested files stay unread, so your context stays light &amp; the session runs longer.</span></p><p><span>And when you need the details, one prompt like</span></p><div class="highlighted_code_block" data-attrs="{&quot;language&quot;:&quot;plaintext&quot;,&quot;nodeId&quot;:&quot;808d43fb-a544-42e9-95f8-c05f122cd236&quot;}" data-component-name="HighlightedCodeBlockToDOM"><pre class="shiki"><code class="language-plaintext">Check this folder and read its CLAUDE.md</code></pre></div><p><span>&#8230;loads only the part you asked for.</span></p><p><span>You save time &amp; tokens with the same move.</span></p><p><span>Two more layers work alongside your files:</span></p><p><span>Claude keeps its own notes with Auto Memory, saving what it learned from your corrections per project, and </span><code>/memory</code><span> opens them for editing. So the memory system runs both ways: </span><em><span>you write instructions, and Claude records what it learns.</span></em></p><p><span>And for larger projects, &#8220;rules&#8221; split the load further&#8230;</span></p><p><span>A rule is a small MD file under </span><code>.claude/rules/</code><span>, one topic per file, and it could carry a &#8220;paths&#8221; field limiting it to matching files. So your API conventions load only when Claude touches the API folder, and they cost nothing on the frontend days.</span></p><h4 style="text-align: center;"><strong>&#167;</strong></h4><h2><span>Compound engineering</span></h2><p><span>Every correction you make is worth money, but only if you write it down.</span></p><p><span>Claude makes a mistake, you fix it once, and then you turn the fix into a rule in your </span><code>CLAUDE.md</code><span> or a skill.</span></p><p><span>The next session starts smarter, and the mistake never comes back.</span></p><p><span>I run my entire setup this way. When a correction repeats twice, it becomes a rule.</span></p><p><span>When a workflow repeats twice, it becomes a skill.</span></p><p><span>So the system compounds, and the setup I have today is the sum of every mistake Claude made this year.</span></p><h4 style="text-align: center;"><strong>&#167;</strong></h4><div class="callout-block" data-callout="true"><p style="text-align: center;"><em><strong><span data-color="#3a7bf9" style="color: rgb(58, 123, 249);">Reminder: this is a teaser of the subscriber-only newsletter, exclusive to my golden members.</span></strong></em></p><p>When you upgrade, you&#8217;ll get:</p><ul><li><p><strong>A complete Claude Code setup, from your first session to a fully automated one.</strong></p></li><li><p>Deep dives into how MCP, subagents, agent orchestration &amp; hooks actually work inside Claude Code.</p></li><li><p><strong>How real Claude Code projects handle context, scale, and running without you.</strong></p></li></ul><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/subscribe&quot;,&quot;text&quot;:&quot;Unlock Full Access&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://newsletter.systemdesign.one/subscribe"><span>Unlock Full Access</span></a></p><p>(If this newsletter has helped you become a better software engineer, consider subscribing to support my work.)</p></div><h4 style="text-align: center;"><strong>&#167;</strong></h4><h2><span>Skills that reduce tokens</span></h2>
      <p>
          <a href="https://newsletter.systemdesign.one/p/claude-code-token-usage">
              Read more
          </a>
      </p>
   ]]></content:encoded></item><item><title><![CDATA[Everyone is a scalability expert till traffic spikes]]></title><description><![CDATA[#197: Part 1 - Break into scalability]]></description><link>https://newsletter.systemdesign.one/p/scalability-patterns-system-design</link><guid isPermaLink="false">https://newsletter.systemdesign.one/p/scalability-patterns-system-design</guid><dc:creator><![CDATA[Neo Kim]]></dc:creator><pubDate>Thu, 29 Oct 2026 15:22:13 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/743de268-886c-4762-9fff-171235a92fc0_1280x720.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p></p><p></p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption"><span data-color="#3a7bf9" style="color: rgb(58, 123, 249);">Subscribe today &amp; never miss a letter from me again!</span></p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><ul><li><p><em><a href="https://newsletter.systemdesign.one/p/scalability-patterns-system-design/?action=share">Share this letter</a> &amp; I&#8217;ll send you some rewards for the referrals.</em></p></li></ul><div><hr></div><p>Once upon a time, there lived a junior software engineer named Michiko.</p><p>She worked for a tech company named Hooli.</p><p>Although she was bright, she never got promoted.</p><p>So she was sad &amp; frustrated&#8230;</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!CkeI!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6bc8de4b-01cc-4bd7-9be6-7e7a19f88cdf_1200x630.gif 424w, https://substackcdn.com/image/fetch/$s_!CkeI!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6bc8de4b-01cc-4bd7-9be6-7e7a19f88cdf_1200x630.gif 848w, https://substackcdn.com/image/fetch/$s_!CkeI!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6bc8de4b-01cc-4bd7-9be6-7e7a19f88cdf_1200x630.gif 1272w, https://substackcdn.com/image/fetch/$s_!CkeI!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6bc8de4b-01cc-4bd7-9be6-7e7a19f88cdf_1200x630.gif 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!CkeI!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6bc8de4b-01cc-4bd7-9be6-7e7a19f88cdf_1200x630.gif" width="420" height="220.5" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/6bc8de4b-01cc-4bd7-9be6-7e7a19f88cdf_1200x630.gif&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:630,&quot;width&quot;:1200,&quot;resizeWidth&quot;:420,&quot;bytes&quot;:213872,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/gif&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://newsletter.systemdesign.one/i/222503287?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6bc8de4b-01cc-4bd7-9be6-7e7a19f88cdf_1200x630.gif&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!CkeI!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6bc8de4b-01cc-4bd7-9be6-7e7a19f88cdf_1200x630.gif 424w, https://substackcdn.com/image/fetch/$s_!CkeI!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6bc8de4b-01cc-4bd7-9be6-7e7a19f88cdf_1200x630.gif 848w, https://substackcdn.com/image/fetch/$s_!CkeI!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6bc8de4b-01cc-4bd7-9be6-7e7a19f88cdf_1200x630.gif 1272w, https://substackcdn.com/image/fetch/$s_!CkeI!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6bc8de4b-01cc-4bd7-9be6-7e7a19f88cdf_1200x630.gif 1456w" sizes="100vw" fetchpriority="high"></picture><div></div></div></div></figure></div><p>Until one day, she had the idea to apply for a new job.</p><p>And the interviewer asked her 1 simple question:</p><p><em>&#8220;How to effectively scale a system?&#8221;</em></p><p>But she couldn&#8217;t answer it &amp; failed the interview in 5 minutes.</p><p>So she went home &amp; studied it to fill the knowledge gap.</p><p>Onward.</p><div><hr></div><div class="callout-block" data-callout="true"><h2><a href="https://prlens.dev/?utm_source=neokim&amp;utm_medium=newsletter&amp;utm_campaign=ad1">Mermaid diagrams are the floor</a></h2><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://prlens.dev/?utm_source=neokim&amp;utm_medium=newsletter&amp;utm_campaign=ad1" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!8piF!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbed88e8-d685-4a85-8795-4805c8781f1a_1200x628.gif 424w, https://substackcdn.com/image/fetch/$s_!8piF!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbed88e8-d685-4a85-8795-4805c8781f1a_1200x628.gif 848w, https://substackcdn.com/image/fetch/$s_!8piF!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbed88e8-d685-4a85-8795-4805c8781f1a_1200x628.gif 1272w, https://substackcdn.com/image/fetch/$s_!8piF!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbed88e8-d685-4a85-8795-4805c8781f1a_1200x628.gif 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!8piF!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbed88e8-d685-4a85-8795-4805c8781f1a_1200x628.gif" width="1200" height="628" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/bbed88e8-d685-4a85-8795-4805c8781f1a_1200x628.gif&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:628,&quot;width&quot;:1200,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:1202568,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/gif&quot;,&quot;href&quot;:&quot;https://prlens.dev/?utm_source=neokim&amp;utm_medium=newsletter&amp;utm_campaign=ad1&quot;,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://newsletter.systemdesign.one/i/222466719?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbed88e8-d685-4a85-8795-4805c8781f1a_1200x628.gif&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!8piF!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbed88e8-d685-4a85-8795-4805c8781f1a_1200x628.gif 424w, https://substackcdn.com/image/fetch/$s_!8piF!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbed88e8-d685-4a85-8795-4805c8781f1a_1200x628.gif 848w, https://substackcdn.com/image/fetch/$s_!8piF!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbed88e8-d685-4a85-8795-4805c8781f1a_1200x628.gif 1272w, https://substackcdn.com/image/fetch/$s_!8piF!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbbed88e8-d685-4a85-8795-4805c8781f1a_1200x628.gif 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Your coding agent can already paste a Mermaid chart into a PR description.</p><p>You still read every file to learn what it does to your system.</p><p><strong><a href="https://prlens.dev/?utm_source=neokim&amp;utm_medium=newsletter&amp;utm_campaign=ad1">PR Lens</a></strong> draws each pull request as a live architecture canvas, right in your pull request. Press play and it walks you through the change, one step at a time.</p><p>On every PR, you see:</p><ul><li><p><strong>What changed</strong>, marked on the parts of the system it touches</p></li><li><p><strong>What else it reaches</strong>, including services the diff never names</p></li><li><p><strong>How a request moves</strong> through the system now</p></li><li><p><strong>What could break</strong>, flagged before you merge</p></li></ul><p>2,200+ GitHub stars and 45,000+ pull requests drawn.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://prlens.dev/?utm_source=neokim&amp;utm_medium=newsletter&amp;utm_campaign=ad1&quot;,&quot;text&quot;:&quot;Add it to your repo&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://prlens.dev/?utm_source=neokim&amp;utm_medium=newsletter&amp;utm_campaign=ad1"><span>Add it to your repo</span></a></p><p>(Thanks to <a href="https://prlens.dev/?utm_source=neokim&amp;utm_medium=newsletter&amp;utm_campaign=ad1">Coldtea</a> for partnering on this newsletter.)</p></div><div><hr></div><h1><strong>Scalability Patterns in System Design</strong></h1><p>Michiko failed the interview, so you don&#8217;t have to.</p><p>And here are 11 must-know scalability techniques for you to ace job interviews:</p><h2><strong>1. Vertical scaling</strong></h2><blockquote><p>Vertical scaling gives one machine more power by adding CPU, memory, and/or faster disks.</p></blockquote><p>The app stays the same, but the machine it runs on gets more powerful... Or you move the app to a larger instance.</p><p>i.e., you do NOT need to rewrite any code. And vertical scaling is probably the quickest way to handle extra load.</p><p>It works well for monoliths &amp; older systems built to run on a <em>single</em> machine.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!52IU!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd14a851c-a3f6-4686-bdc6-fcf1451f0cf7_1536x1024.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!52IU!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd14a851c-a3f6-4686-bdc6-fcf1451f0cf7_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!52IU!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd14a851c-a3f6-4686-bdc6-fcf1451f0cf7_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!52IU!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd14a851c-a3f6-4686-bdc6-fcf1451f0cf7_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!52IU!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd14a851c-a3f6-4686-bdc6-fcf1451f0cf7_1536x1024.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!52IU!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd14a851c-a3f6-4686-bdc6-fcf1451f0cf7_1536x1024.png" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/d14a851c-a3f6-4686-bdc6-fcf1451f0cf7_1536x1024.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!52IU!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd14a851c-a3f6-4686-bdc6-fcf1451f0cf7_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!52IU!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd14a851c-a3f6-4686-bdc6-fcf1451f0cf7_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!52IU!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd14a851c-a3f6-4686-bdc6-fcf1451f0cf7_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!52IU!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd14a851c-a3f6-4686-bdc6-fcf1451f0cf7_1536x1024.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Yet larger machines cost more &amp; all machines have limits.</p><p>Plus, if your app relies on a single server, a failure could take the entire app down.</p><p>So scale up when you need more capacity quickly. Then consider horizontal scaling before one machine becomes your system&#8217;s biggest bottleneck.</p><h2><strong>2. Horizontal scaling</strong></h2><blockquote><p>Horizontal scaling (scaling out) adds more machines to share the load.</p></blockquote><p>You run several copies of the app &amp; distribute traffic across them using a load balancer.</p><p>i.e., you add machines as traffic grows &amp; remove them when demand drops.</p><p>It works well for applications with high/unpredictable traffic.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!9CMu!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff58eb42b-b594-4ff3-8608-2b9c5b5cd6af_1536x1024.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!9CMu!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff58eb42b-b594-4ff3-8608-2b9c5b5cd6af_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!9CMu!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff58eb42b-b594-4ff3-8608-2b9c5b5cd6af_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!9CMu!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff58eb42b-b594-4ff3-8608-2b9c5b5cd6af_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!9CMu!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff58eb42b-b594-4ff3-8608-2b9c5b5cd6af_1536x1024.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!9CMu!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff58eb42b-b594-4ff3-8608-2b9c5b5cd6af_1536x1024.png" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/f58eb42b-b594-4ff3-8608-2b9c5b5cd6af_1536x1024.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!9CMu!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff58eb42b-b594-4ff3-8608-2b9c5b5cd6af_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!9CMu!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff58eb42b-b594-4ff3-8608-2b9c5b5cd6af_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!9CMu!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff58eb42b-b594-4ff3-8608-2b9c5b5cd6af_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!9CMu!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff58eb42b-b594-4ff3-8608-2b9c5b5cd6af_1536x1024.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Yet adding machines creates new challenges:</p><p>You need load balancing, coordination between instances &amp; keep shared data consistent.</p><p>Horizontal scaling improves &#8220;availability&#8221; with redundant instances. But if your app stores user sessions locally, requests routed to different instances could lose access to session data.</p><p>So before you scale out, you need to decide where your application stores its state.</p><p>And that&#8217;s why you&#8217;d need stateless services...</p><h2><strong>3. Stateless services</strong></h2><blockquote><p>A stateless service does NOT store client session state between requests.</p></blockquote><p>Every request provides information the server needs - auth token or session ID.</p><p>Plus, you don&#8217;t store session data on <em>one</em> server&#8230; instead, you keep it in a shared database, a cache like Redis, or use a signed token like a JSON Web Token (<strong>JWT</strong>).</p><p>i.e., any server could handle a request without relying on the previous one.</p><p>This makes it easier to add, remove, or replace servers as traffic patterns change.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!EqH6!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc03129c5-06ad-44c8-80f5-f8eedb60fc81_1536x1024.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!EqH6!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc03129c5-06ad-44c8-80f5-f8eedb60fc81_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!EqH6!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc03129c5-06ad-44c8-80f5-f8eedb60fc81_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!EqH6!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc03129c5-06ad-44c8-80f5-f8eedb60fc81_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!EqH6!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc03129c5-06ad-44c8-80f5-f8eedb60fc81_1536x1024.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!EqH6!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc03129c5-06ad-44c8-80f5-f8eedb60fc81_1536x1024.png" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/c03129c5-06ad-44c8-80f5-f8eedb60fc81_1536x1024.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!EqH6!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc03129c5-06ad-44c8-80f5-f8eedb60fc81_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!EqH6!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc03129c5-06ad-44c8-80f5-f8eedb60fc81_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!EqH6!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc03129c5-06ad-44c8-80f5-f8eedb60fc81_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!EqH6!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc03129c5-06ad-44c8-80f5-f8eedb60fc81_1536x1024.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Yet the state still needs to live somewhere.</p><p>And an external database/cache adds infrastructure, network latency &amp; an extra dependency to manage. Plus, relying on sticky sessions to keep users tied to one server reduces flexibility when servers fail or traffic changes.</p><p>So keep session state outside individual app instances when possible. Also, design operations to be &#8220;idempotent&#8221;.</p><p>With stateless services, you could scale individual instances more easily.</p><p>But what if those services depend heavily on each other&#8230;</p><p>That&#8217;s where you&#8217;d need loose coupling...</p><h2><strong>4. Loose coupling</strong></h2><blockquote><p>Loose coupling = each component depends on others as little as possible.</p></blockquote><p>A change in one component could force you to update several others in a &#8220;tightly coupled&#8221; system.</p><p>Plus, if one service becomes slow/fails, the problem could spread to other services that depend on it.</p><p>Loose coupling reduces these dependencies by letting components communicate through well-defined interfaces - APIs/messages. So each component knows what to expect from others without understanding their internal implementation.</p><p>This makes it easier to update, deploy, and/or scale components independently.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!IAId!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F24938952-55ba-4353-b9ae-83a8c2b49515_1536x1024.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!IAId!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F24938952-55ba-4353-b9ae-83a8c2b49515_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!IAId!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F24938952-55ba-4353-b9ae-83a8c2b49515_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!IAId!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F24938952-55ba-4353-b9ae-83a8c2b49515_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!IAId!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F24938952-55ba-4353-b9ae-83a8c2b49515_1536x1024.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!IAId!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F24938952-55ba-4353-b9ae-83a8c2b49515_1536x1024.png" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/24938952-55ba-4353-b9ae-83a8c2b49515_1536x1024.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!IAId!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F24938952-55ba-4353-b9ae-83a8c2b49515_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!IAId!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F24938952-55ba-4353-b9ae-83a8c2b49515_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!IAId!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F24938952-55ba-4353-b9ae-83a8c2b49515_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!IAId!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F24938952-55ba-4353-b9ae-83a8c2b49515_1536x1024.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Yet loose coupling introduces extra complexity:</p><p>Requests could travel across many services,,, thus making it HARD to trace &amp; debug failures. Also,splitting a system into many tiny services increases communication overhead &amp; coordination costs.</p><p>So use loose coupling when components need to evolve independently - microservices &amp; big apps.</p><p>And to organize these components into clear, manageable parts, you need modular architecture&#8230;</p><h2><strong>5. Modular architecture</strong></h2><blockquote><p>Modular architecture divides a system into separate modules, with each responsible for a specific function.</p></blockquote><p>You divide the app into modules for authentication, orders, payments, notifications, and so on.</p><p>i.e., each module has a clear responsibility &amp; communicates with others through defined interfaces.</p><p>The goal is to group functionality that belongs together &amp; keep dependencies between modules <em>minimal</em>. This follows the principle of <em>separation of concerns</em>: each module focuses on one part of the system.</p><p>So you update/test payment module without changing the authentication module.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!1jF0!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F07fe0343-6d45-4cc3-8c61-7cc018bbe55d_1672x941.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!1jF0!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F07fe0343-6d45-4cc3-8c61-7cc018bbe55d_1672x941.png 424w, https://substackcdn.com/image/fetch/$s_!1jF0!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F07fe0343-6d45-4cc3-8c61-7cc018bbe55d_1672x941.png 848w, https://substackcdn.com/image/fetch/$s_!1jF0!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F07fe0343-6d45-4cc3-8c61-7cc018bbe55d_1672x941.png 1272w, https://substackcdn.com/image/fetch/$s_!1jF0!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F07fe0343-6d45-4cc3-8c61-7cc018bbe55d_1672x941.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!1jF0!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F07fe0343-6d45-4cc3-8c61-7cc018bbe55d_1672x941.png" width="1456" height="819" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/07fe0343-6d45-4cc3-8c61-7cc018bbe55d_1672x941.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:819,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!1jF0!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F07fe0343-6d45-4cc3-8c61-7cc018bbe55d_1672x941.png 424w, https://substackcdn.com/image/fetch/$s_!1jF0!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F07fe0343-6d45-4cc3-8c61-7cc018bbe55d_1672x941.png 848w, https://substackcdn.com/image/fetch/$s_!1jF0!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F07fe0343-6d45-4cc3-8c61-7cc018bbe55d_1672x941.png 1272w, https://substackcdn.com/image/fetch/$s_!1jF0!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F07fe0343-6d45-4cc3-8c61-7cc018bbe55d_1672x941.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Yet choosing the &#8220;wrong module boundaries&#8221; creates unnecessary dependencies.</p><p>Plus, splitting an app into many modules makes the codebase hard to manage. Also, modules within the same app still share resources &amp; failure risks.</p><p>So modularity alone does NOT guarantee independent scaling or failure isolation.</p><p>It works well for large applications, modular monoliths &amp; microservice architectures!</p><p>But if these modules need to communicate without waiting for each other&#8230;you&#8217;d need event-driven architecture&#8230;</p><div><hr></div><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption"><span data-color="#3a7bf9" style="color: rgb(58, 123, 249);">Subscribe now and be the first to know about my new letters &amp; updates:</span></p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><div><hr></div><h2><strong>6. Event-driven architecture</strong></h2><blockquote><p>Event-driven architecture allows services to communicate by publishing &amp; reacting to events.</p></blockquote><p>A service publishes an event to a message broker instead of calling other services directly.</p><p>Imagine a user places an order on an e-commerce website:</p><p>The <code>order</code> service publishes an <code>OrderPlaced</code> event&#8230; Then other services consume it independently:</p><ul><li><p><code>Inventory</code> service reserves ordered items,</p></li><li><p><code>Payment</code> service initiates payment processing,</p></li><li><p><code>Notification</code> service sends an order confirmation when required steps succeed.</p></li></ul><p><code>Order</code> service does NOT wait for each consumer to finish processing the event.</p><p>i.e., a slow consumer does NOT block others.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!CCw1!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb151ec27-bd16-43d9-8beb-0634f5f6c723_1693x929.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!CCw1!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb151ec27-bd16-43d9-8beb-0634f5f6c723_1693x929.png 424w, https://substackcdn.com/image/fetch/$s_!CCw1!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb151ec27-bd16-43d9-8beb-0634f5f6c723_1693x929.png 848w, https://substackcdn.com/image/fetch/$s_!CCw1!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb151ec27-bd16-43d9-8beb-0634f5f6c723_1693x929.png 1272w, https://substackcdn.com/image/fetch/$s_!CCw1!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb151ec27-bd16-43d9-8beb-0634f5f6c723_1693x929.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!CCw1!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb151ec27-bd16-43d9-8beb-0634f5f6c723_1693x929.png" width="1456" height="799" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/b151ec27-bd16-43d9-8beb-0634f5f6c723_1693x929.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:799,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!CCw1!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb151ec27-bd16-43d9-8beb-0634f5f6c723_1693x929.png 424w, https://substackcdn.com/image/fetch/$s_!CCw1!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb151ec27-bd16-43d9-8beb-0634f5f6c723_1693x929.png 848w, https://substackcdn.com/image/fetch/$s_!CCw1!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb151ec27-bd16-43d9-8beb-0634f5f6c723_1693x929.png 1272w, https://substackcdn.com/image/fetch/$s_!CCw1!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb151ec27-bd16-43d9-8beb-0634f5f6c723_1693x929.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>This approach works well for workflows when many services need to react to the same event.</p><p>Yet event-driven systems create new problems:</p><p>Events could arrive late, out of order, or more than once. Plus, different services could temporarily hold different versions of the same data. This is called <em>eventual consistency</em>.</p><p>So you need idempotent consumers, reliable event delivery &amp; a way to handle failed operations.</p><p>Event-driven architecture helps services scale independently &amp; reduces synchronous dependencies.</p><p>But when you need to organize these services without creating a central bottleneck, you&#8217;ve got to &#8220;avoid&#8221; <em>centralized resources</em>&#8230;</p><div><hr></div><div class="captioned-button-wrap" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/p/scalability-patterns-system-design?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;}" data-component-name="CaptionedButtonToDOM"><div class="preamble"><p class="cta-caption"><em><span data-color="#3a7bf9" style="color: rgb(58, 123, 249);">Like what you read? Share it with your friends &amp; leave a comment!</span></em></p></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/p/scalability-patterns-system-design?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://newsletter.systemdesign.one/p/scalability-patterns-system-design?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p></div><div><hr></div><h2><strong>7. Shared-nothing architecture</strong></h2><blockquote><p>Shared-nothing architecture gives each node its own compute, memory &amp; storage <em>without</em> relying on shared resources for processing.</p></blockquote><p>There are several ways to achieve this:</p>
      <p>
          <a href="https://newsletter.systemdesign.one/p/scalability-patterns-system-design">
              Read more
          </a>
      </p>
   ]]></content:encoded></item><item><title><![CDATA[GitOps Deep Dive: How ArgoCD Works]]></title><description><![CDATA[#196: Part 5 - DevOps Mastermind]]></description><link>https://newsletter.systemdesign.one/p/how-argocd-works-gitops-kubernetes</link><guid isPermaLink="false">https://newsletter.systemdesign.one/p/how-argocd-works-gitops-kubernetes</guid><dc:creator><![CDATA[Neo Kim]]></dc:creator><pubDate>Mon, 26 Oct 2026 11:46:34 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/c4e4bfbb-e1aa-4e6c-8da2-21b587328d42_1280x720.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p></p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption"><span data-color="#3a7bf9" style="color: rgb(58, 123, 249);">Get my AI engineering playbook for FREE on newsletter signup:</span></p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><ul><li><p><em><a href="https://newsletter.systemdesign.one/p/how-argocd-works-gitops-kubernetes/?action=share">Share this letter</a> &amp; I&#8217;ll send you some rewards for the referrals.</em></p></li></ul><div><hr></div><p>The CI/CD pipeline changed how we deploy software.</p><p>It made deployments faster, repeatable, and automated. But once a deployment finishes, the pipeline has no idea what happens inside the cluster afterward.</p><p>Someone could edit a Deployment with <code>kubectl</code>, apply a hotfix during an incident, or update a resource manually. Those changes bypass Git and the pipeline.</p><p>As manual changes increase, the cluster slowly drifts from what gets stored in Git.</p><p>At some point, production no longer matches the configuration everyone believes is running. When an issue appears, it becomes difficult to see what changed... Also it becomes difficult to know who changed it or whether the running state still matches the last reviewed configuration.</p><p>GitOps solves this by making Git the source of truth for the cluster&#8230;not just another step in the deployment process&#8230;</p><p>An agent continuously compares the live cluster with the desired state stored in Git and reconciles any differences. If someone changes a resource outside Git, ArgoCD detects the drift and brings the cluster back to the expected state.</p><p>The CI pipeline still builds, tests, and publishes the application<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-1" href="#footnote-1" target="_self">1</a>.</p><p>But instead of deploying directly to Kubernetes, it updates Git. ArgoCD then handles the deployment and keeps the cluster aligned with the repository.</p><h4 style="text-align: center;"><strong>&#167;</strong></h4><div class="callout-block" data-callout="true"><h3><strong><a href="https://getunblocked.com/events/how-to-stop-rewriting-your-agent-s-code-oct-28/?utm_source=systemdesign&amp;utm_medium=email&amp;utm_campaign=primary">[Webinar] How to stop rewriting your agent&#8217;s code</a></strong></h3><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://getunblocked.com/events/how-to-stop-rewriting-your-agent-s-code-oct-28/?utm_source=systemdesign&amp;utm_medium=email&amp;utm_campaign=primary" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!wgp1!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c47510d-48c8-4f8b-9093-e2c256c980f3_1600x900.png 424w, https://substackcdn.com/image/fetch/$s_!wgp1!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c47510d-48c8-4f8b-9093-e2c256c980f3_1600x900.png 848w, https://substackcdn.com/image/fetch/$s_!wgp1!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c47510d-48c8-4f8b-9093-e2c256c980f3_1600x900.png 1272w, https://substackcdn.com/image/fetch/$s_!wgp1!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c47510d-48c8-4f8b-9093-e2c256c980f3_1600x900.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!wgp1!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c47510d-48c8-4f8b-9093-e2c256c980f3_1600x900.png" width="1456" height="819" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/7c47510d-48c8-4f8b-9093-e2c256c980f3_1600x900.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:819,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:560005,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:&quot;https://getunblocked.com/events/how-to-stop-rewriting-your-agent-s-code-oct-28/?utm_source=systemdesign&amp;utm_medium=email&amp;utm_campaign=primary&quot;,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://newsletter.systemdesign.one/i/221150475?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c47510d-48c8-4f8b-9093-e2c256c980f3_1600x900.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!wgp1!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c47510d-48c8-4f8b-9093-e2c256c980f3_1600x900.png 424w, https://substackcdn.com/image/fetch/$s_!wgp1!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c47510d-48c8-4f8b-9093-e2c256c980f3_1600x900.png 848w, https://substackcdn.com/image/fetch/$s_!wgp1!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c47510d-48c8-4f8b-9093-e2c256c980f3_1600x900.png 1272w, https://substackcdn.com/image/fetch/$s_!wgp1!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c47510d-48c8-4f8b-9093-e2c256c980f3_1600x900.png 1456w" sizes="auto, 100vw" loading="lazy" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Right now, you&#8217;re your agent&#8217;s context engine. You write the rules files, maintain the skills, and paste in the Slack thread it missed.</p><p>But hand-curated context goes stale. And when there are conflicting answers, your agent can&#8217;t tell which is right. You pay for it in higher token bills and longer reviews.</p><p><a href="https://getunblocked.com/events/how-to-stop-rewriting-your-agent-s-code-oct-28/?utm_source=systemdesign&amp;utm_medium=email&amp;utm_campaign=primary">Join us live on Oct 28 (FREE)</a> to see:</p><ul><li><p>The 8-levels of AI maturity and where teams get stuck</p></li><li><p>How a context engine curates context and resolves conflicts before your agent writes a line of code</p></li><li><p>Live demo: token savings in the same coding task, and use cases beyond code gen</p></li></ul><p>Bonus: 2 open source tools you can use today.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://getunblocked.com/events/how-to-stop-rewriting-your-agent-s-code-oct-28/?utm_source=systemdesign&amp;utm_medium=email&amp;utm_campaign=primary&quot;,&quot;text&quot;:&quot;Register now&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://getunblocked.com/events/how-to-stop-rewriting-your-agent-s-code-oct-28/?utm_source=systemdesign&amp;utm_medium=email&amp;utm_campaign=primary"><span>Register now</span></a></p><p><em>(Thanks to <a href="https://getunblocked.com/events/how-to-stop-rewriting-your-agent-s-code-oct-28/?utm_source=systemdesign&amp;utm_medium=email&amp;utm_campaign=primary">Unblocked</a> for partnering on this newsletter.)</em></p></div><h4 style="text-align: center;"><strong>&#167;</strong></h4><p>I want to introduce <strong><a href="https://x.com/twtayaan">Ayaan</a></strong> as a guest author.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://x.com/twtayaan" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!GCV8!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d442f35-0ac7-4552-bd87-66962939694d_2048x1024.png 424w, https://substackcdn.com/image/fetch/$s_!GCV8!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d442f35-0ac7-4552-bd87-66962939694d_2048x1024.png 848w, https://substackcdn.com/image/fetch/$s_!GCV8!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d442f35-0ac7-4552-bd87-66962939694d_2048x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!GCV8!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d442f35-0ac7-4552-bd87-66962939694d_2048x1024.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!GCV8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d442f35-0ac7-4552-bd87-66962939694d_2048x1024.png" width="1456" height="728" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/5d442f35-0ac7-4552-bd87-66962939694d_2048x1024.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:728,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:&quot;&quot;,&quot;type&quot;:null,&quot;href&quot;:&quot;https://x.com/twtayaan&quot;,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!GCV8!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d442f35-0ac7-4552-bd87-66962939694d_2048x1024.png 424w, https://substackcdn.com/image/fetch/$s_!GCV8!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d442f35-0ac7-4552-bd87-66962939694d_2048x1024.png 848w, https://substackcdn.com/image/fetch/$s_!GCV8!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d442f35-0ac7-4552-bd87-66962939694d_2048x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!GCV8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d442f35-0ac7-4552-bd87-66962939694d_2048x1024.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>He&#8217;s a <strong>Certified Kubernetes Administrator (CKA)</strong> who writes about Kubernetes, cloud infrastructure, and cloud-native systems. He builds in public and shares practical tips, projects, and lessons learned along the way.</p><p>Check out his work &amp; socials:</p><ul><li><p><a href="https://x.com/twtayaan">Twitter</a></p></li><li><p><a href="https://github.com/Ayaan49">Github</a></p></li></ul><h4 style="text-align: center;"><strong>&#167;</strong></h4><p><em>Here&#8217;s what&#8217;s inside:</em></p><ul><li><p><strong>Your deployment succeeds. Then production starts drifting.</strong> How manual Kubernetes changes create configuration drift, why traditional CI/CD pipelines miss them, and how GitOps keeps your cluster aligned with Git.</p></li><li><p><strong>Your CI pipeline no longer needs the keys to Kubernetes.</strong> How ArgoCD&#8217;s pull-based model changes deployments, where credentials live, and what happens when a system gets compromised.</p></li><li><p><strong>ArgoCD runs a reconciliation loop behind the scenes.</strong> How its core components detect changes, compare desired and live states, and use sync policies, self-healing, and hooks to control deployments.</p></li><li><p><strong>One template, hundreds of applications.</strong> How ApplicationSets manage deployments across clusters, why repository structure matters, and how Git commits promote releases from development to production.</p></li><li><p><strong>GitOps gets harder when you reach production.</strong> How to manage secrets, access control, monitoring, progressive delivery, and disaster recovery, plus the tradeoffs you need to understand before adopting it.</p></li></ul><p></p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption"><span data-color="#3a7bf9" style="color: rgb(58, 123, 249);">Golden members get all letters like these!&#8230;</span></p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><h4 style="text-align: center;"><strong>&#167;</strong></h4><h2>What Is GitOps</h2><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!qnwa!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3e32c002-ec05-4f5c-93fc-77123fcb091f_1536x1024.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!qnwa!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3e32c002-ec05-4f5c-93fc-77123fcb091f_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!qnwa!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3e32c002-ec05-4f5c-93fc-77123fcb091f_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!qnwa!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3e32c002-ec05-4f5c-93fc-77123fcb091f_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!qnwa!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3e32c002-ec05-4f5c-93fc-77123fcb091f_1536x1024.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!qnwa!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3e32c002-ec05-4f5c-93fc-77123fcb091f_1536x1024.png" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/3e32c002-ec05-4f5c-93fc-77123fcb091f_1536x1024.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!qnwa!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3e32c002-ec05-4f5c-93fc-77123fcb091f_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!qnwa!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3e32c002-ec05-4f5c-93fc-77123fcb091f_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!qnwa!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3e32c002-ec05-4f5c-93fc-77123fcb091f_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!qnwa!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3e32c002-ec05-4f5c-93fc-77123fcb091f_1536x1024.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>GitOps<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-2" href="#footnote-2" target="_self">2</a> is an operating model where a Git repository is the single source of truth for everything running in your infrastructure.</p><p>Every change to your application or cluster configuration happens through a Git commit. It gets reviewed, approved, and merged through the same process as application code.</p><p>i.e., changes are not applied directly to the cluster outside Git.</p><h4 style="text-align: center;"><strong>&#167;</strong></h4><h2>GitOps Principles</h2><p>The OpenGitOps<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-3" href="#footnote-3" target="_self">3</a> working group at CNCF<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-4" href="#footnote-4" target="_self">4</a> defined four principles to distinguish GitOps from a traditional deployment pipeline.</p><p>Let&#8217;s dive in!</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!UJ_X!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F41493a3b-e759-4caa-8a0c-245baeb564d0_1254x1254.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!UJ_X!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F41493a3b-e759-4caa-8a0c-245baeb564d0_1254x1254.png 424w, https://substackcdn.com/image/fetch/$s_!UJ_X!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F41493a3b-e759-4caa-8a0c-245baeb564d0_1254x1254.png 848w, https://substackcdn.com/image/fetch/$s_!UJ_X!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F41493a3b-e759-4caa-8a0c-245baeb564d0_1254x1254.png 1272w, https://substackcdn.com/image/fetch/$s_!UJ_X!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F41493a3b-e759-4caa-8a0c-245baeb564d0_1254x1254.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!UJ_X!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F41493a3b-e759-4caa-8a0c-245baeb564d0_1254x1254.png" width="1254" height="1254" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/41493a3b-e759-4caa-8a0c-245baeb564d0_1254x1254.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1254,&quot;width&quot;:1254,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!UJ_X!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F41493a3b-e759-4caa-8a0c-245baeb564d0_1254x1254.png 424w, https://substackcdn.com/image/fetch/$s_!UJ_X!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F41493a3b-e759-4caa-8a0c-245baeb564d0_1254x1254.png 848w, https://substackcdn.com/image/fetch/$s_!UJ_X!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F41493a3b-e759-4caa-8a0c-245baeb564d0_1254x1254.png 1272w, https://substackcdn.com/image/fetch/$s_!UJ_X!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F41493a3b-e759-4caa-8a0c-245baeb564d0_1254x1254.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3>1. Declarative</h3><p>In a declarative system, you describe what you want, not how to get there.</p><p>Instead of writing a script with steps like &#8220;<em>create this deployment, then scale it to three replicas, then expose it on port 80,&#8221; </em>you write a YAML file describing the end state. The system figures out how to make it happen.</p><p>Kubernetes is declarative by nature.</p><p>Helm charts<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-5" href="#footnote-5" target="_self">5</a>, Kustomize<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-6" href="#footnote-6" target="_self">6</a> overlays, and raw YAML manifests all follow the same model.</p><p>This matters for GitOps because a declarative system supports storage, versioning, and comparison. Unlike a sequence of commands, two YAML files allow you to see exactly what changed.</p><h3>2. Versioned and Immutable</h3><p>Every change to the desired state is a Git commit.</p><p>And Git commits do not change&#8230;</p><p>This gives you a complete, tamper-evident history of everything your cluster has ever been asked to run. Who requested the change, who approved it, when it got merged, and exactly what it modified. The audit trail becomes part of the deployment workflow, not a separate system.</p><p>Rollback is also a natural consequence of this principle. Reverting to a previous state means reverting a commit.</p><h3>3. Pulled Automatically</h3><p>This principle changes who is responsible for applying changes.</p><p>Instead of an external CI/CD pipeline pushing updates into Kubernetes, an agent inside the cluster pulls the desired state from Git&#8230; It then applies the changes.</p><p>The security implications of this shift are significant, and they get their own section shortly.</p><h3>4. Continuously Reconciled</h3><p>The previous three principles describe how state gets defined and delivered.</p><p>This one describes what happens after:</p><p>Continuous reconciliation means repeatedly comparing the live cluster state with the desired state in Git. These comparisons detect changes introduced by a manual edit, an autoscaler, or a partial update. The configured sync policy determines whether ArgoCD reports the difference or corrects it automatically.</p><p>To understand why GitOps works differently, we first need to compare it with the deployment model most people already use&#8230;</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/p/how-argocd-works-gitops-kubernetes?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://newsletter.systemdesign.one/p/how-argocd-works-gitops-kubernetes?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p><h4 style="text-align: center;"><strong>&#167;</strong></h4><h2>CD Without GitOps vs CD With GitOps</h2><p>With the four principles in place, the difference between traditional CD &amp; GitOps is easier to see.</p><p>It is not just a philosophical shift. The two models produce different security properties and failure modes.</p><p>They also provide different levels of visibility into what runs in production&#8230;</p><h3>How CD Worked Before GitOps</h3><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!dkQL!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F02669c3b-e3fd-49bf-8f76-cf1d6273f95d_1536x1024.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!dkQL!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F02669c3b-e3fd-49bf-8f76-cf1d6273f95d_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!dkQL!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F02669c3b-e3fd-49bf-8f76-cf1d6273f95d_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!dkQL!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F02669c3b-e3fd-49bf-8f76-cf1d6273f95d_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!dkQL!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F02669c3b-e3fd-49bf-8f76-cf1d6273f95d_1536x1024.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!dkQL!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F02669c3b-e3fd-49bf-8f76-cf1d6273f95d_1536x1024.png" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/02669c3b-e3fd-49bf-8f76-cf1d6273f95d_1536x1024.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!dkQL!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F02669c3b-e3fd-49bf-8f76-cf1d6273f95d_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!dkQL!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F02669c3b-e3fd-49bf-8f76-cf1d6273f95d_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!dkQL!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F02669c3b-e3fd-49bf-8f76-cf1d6273f95d_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!dkQL!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F02669c3b-e3fd-49bf-8f76-cf1d6273f95d_1536x1024.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>In a traditional push-based setup<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-7" href="#footnote-7" target="_self">7</a>, the CI/CD pipeline carries a release from source code to the Kubernetes cluster.</p><p>It runs tests, builds a container image, and pushes the image to a registry. The deployment stage then updates Kubernetes using a command such as <code>kubectl apply</code> or <code>helm upgrade</code>.</p><p>For example, releasing version 1.5 means the pipeline updates the Deployment to use the new image. Kubernetes then replaces the existing pods according to the configured rollout strategy.</p><p>The pipeline directly initiates the deployment. The deployment stage delivers the new configuration to the cluster.</p><h3>How CD Works With ArgoCD</h3><p>With ArgoCD<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-8" href="#footnote-8" target="_self">8</a>, the pipeline still tests the application, builds the image, and publishes it to a registry. It then updates the image reference in the configuration repository.</p><p>For the same version 1.5 release, the pipeline changes the image tag in a Kubernetes manifest, Helm values file, or Kustomize configuration. This change becomes a Git commit rather than a deployment command executed against the cluster.</p><p>ArgoCD detects the updated configuration and compares it with the live resources. With automated sync enabled, it applies the update. With manual sync, it waits for an operator to trigger synchronization. Kubernetes then performs the rollout.</p><p>i.e., Git becomes the handoff between the two systems.</p><p>CI produces the image and records the intended version. ArgoCD handles applying the configuration to Kubernetes.</p><p>This separation also changes where deployment credentials are needed, which brings us to the security boundary between push and pull.</p><h4 style="text-align: center;"><strong>&#167;</strong></h4><h2>Push vs Pull: Security Boundary</h2><p>The shift from push to pull is not just an architectural preference.</p><p>It changes where credentials live, how far a compromised system can reach, and the damage a supply chain attack can cause.</p><p>Now let&#8217;s compare the two approaches:</p><h3>Push-Based: Pipeline Holds the Keys</h3><p>In a push-based model, the CI pipeline needs direct access to the cluster to deploy anything.</p><p>i.e., cluster credentials live inside the CI system.</p><p>Anyone with sufficient access to the CI system or pipeline configuration could also gain access to those credentials. Every third-party action, plugin, or integration running inside the pipeline runs adjacent to them. And because pipelines are triggered by code changes, a compromised dependency runs in the pipeline. It runs in the same environment as your cluster credentials.</p><p>A compromised pipeline in this model is effectively a compromised cluster.</p><p>The attacker does not need to find a vulnerability in Kubernetes&#8230; They just need to get malicious code into the build process &amp; let the pipeline do the rest.</p><p>The blast radius is as wide as the credentials allow.</p><p>For most engineers, those credentials have broad permissions. Restricting them precisely is operationally painful. A compromised push-based pipeline often means cluster-admin access is in the wrong hands.</p><h3>Pull-Based: Cluster Holds the Keys</h3><p>In a pull-based model, the cluster reaches out to Git.</p><p>The CI system never touches the cluster.</p><p>ArgoCD runs inside the cluster with scoped RBAC<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-9" href="#footnote-9" target="_self">9</a> (Role-Based Access Control) permissions. It pulls the desired state from Git and applies it. The CI pipeline updates the Git repository when a build succeeds, and its job ends there. No cluster credentials ever leave the cluster environment.</p><p>A compromised CI system in this model could modify Git.</p><p>But modifying Git only changes the desired state. It does not directly execute anything in the cluster. ArgoCD still has to pull the change and reconcile it with the cluster. Because every change goes through Git first, you get a reviewable, auditable record before it reaches production.</p><p>And the blast radius<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-10" href="#footnote-10" target="_self">10</a> shrinks dramatically. A compromised pipeline becomes a much less dangerous event.</p><h4 style="text-align: center;"><strong>&#167;</strong></h4><div class="captioned-button-wrap" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/p/how-argocd-works-gitops-kubernetes?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;}" data-component-name="CaptionedButtonToDOM"><div class="preamble"><p class="cta-caption"><em><span data-color="#3a7bf9" style="color: rgb(58, 123, 249);">If something resonated with you today, share this letter. Because one idea, one action, can change everything.</span></em></p></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/p/how-argocd-works-gitops-kubernetes?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://newsletter.systemdesign.one/p/how-argocd-works-gitops-kubernetes?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p></div><h4 style="text-align: center;"><strong>&#167;</strong></h4><h2>ArgoCD Architecture</h2><p>Let&#8217;s look at how ArgoCD is built:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!8FYf!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7509f69a-b288-4cc9-bf97-1fabdba1eb01_1692x929.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!8FYf!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7509f69a-b288-4cc9-bf97-1fabdba1eb01_1692x929.png 424w, https://substackcdn.com/image/fetch/$s_!8FYf!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7509f69a-b288-4cc9-bf97-1fabdba1eb01_1692x929.png 848w, https://substackcdn.com/image/fetch/$s_!8FYf!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7509f69a-b288-4cc9-bf97-1fabdba1eb01_1692x929.png 1272w, https://substackcdn.com/image/fetch/$s_!8FYf!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7509f69a-b288-4cc9-bf97-1fabdba1eb01_1692x929.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!8FYf!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7509f69a-b288-4cc9-bf97-1fabdba1eb01_1692x929.png" width="1456" height="799" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/7509f69a-b288-4cc9-bf97-1fabdba1eb01_1692x929.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:799,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!8FYf!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7509f69a-b288-4cc9-bf97-1fabdba1eb01_1692x929.png 424w, https://substackcdn.com/image/fetch/$s_!8FYf!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7509f69a-b288-4cc9-bf97-1fabdba1eb01_1692x929.png 848w, https://substackcdn.com/image/fetch/$s_!8FYf!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7509f69a-b288-4cc9-bf97-1fabdba1eb01_1692x929.png 1272w, https://substackcdn.com/image/fetch/$s_!8FYf!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7509f69a-b288-4cc9-bf97-1fabdba1eb01_1692x929.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>ArgoCD is not an external service you call into.</p><p>It runs as Kubernetes workloads inside the cluster it manages, in the argocd namespace. This is <em>intentional</em>. The agent needs direct access to the Kubernetes API to reconcile state. Running inside the cluster provides the access &amp; keeps cluster credentials inside the cluster.</p><p>Here are 5 core components that work together to implement the GitOps reconciliation loop:</p><h3>1. API Server</h3><p>API Server is the entry point for everything human-facing in ArgoCD.</p><p>It exposes both gRPC<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-11" href="#footnote-11" target="_self">11</a> &amp; REST APIs consumed by the ArgoCD UI, argocd CLI, and any external tools integrating with ArgoCD. Every user request, whether it comes from the UI, CLI, or an external integration, passes through the API Server.</p><p>Also it handles all authentication &amp; authorization.</p><p>Before any request reaches the rest of the system, the API Server verifies who is making it and whether they have permission.</p><h3>2. Repo Server</h3><p>The Repo Server does one thing: <em>render Kubernetes manifests from a Git repository</em>.</p><p>It clones the repository, reads the source, and generates the manifests ArgoCD will apply to the cluster. It supports plain YAML, Helm charts, Kustomize overlays, and Jsonnet<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-12" href="#footnote-12" target="_self">12</a>. Whatever format your configuration lives in, the Repo Server handles the rendering.</p><p>It&#8217;s deliberately isolated from the Application Controller.</p><p>Rendering manifests requires running Helm, Kustomize &amp; Jsonnet. Those processes are often CPU-intensive and occasionally unstable. Keeping the Repo Server separate isolates render failures from the controller&#8217;s reconciliation loop.</p><h3>3. Application Controller</h3><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!Ku33!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdccd3d21-822d-41a3-8aaf-eff0d0cdbe92_1254x1254.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Ku33!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdccd3d21-822d-41a3-8aaf-eff0d0cdbe92_1254x1254.png 424w, https://substackcdn.com/image/fetch/$s_!Ku33!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdccd3d21-822d-41a3-8aaf-eff0d0cdbe92_1254x1254.png 848w, https://substackcdn.com/image/fetch/$s_!Ku33!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdccd3d21-822d-41a3-8aaf-eff0d0cdbe92_1254x1254.png 1272w, https://substackcdn.com/image/fetch/$s_!Ku33!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdccd3d21-822d-41a3-8aaf-eff0d0cdbe92_1254x1254.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!Ku33!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdccd3d21-822d-41a3-8aaf-eff0d0cdbe92_1254x1254.png" width="1254" height="1254" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/dccd3d21-822d-41a3-8aaf-eff0d0cdbe92_1254x1254.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1254,&quot;width&quot;:1254,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!Ku33!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdccd3d21-822d-41a3-8aaf-eff0d0cdbe92_1254x1254.png 424w, https://substackcdn.com/image/fetch/$s_!Ku33!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdccd3d21-822d-41a3-8aaf-eff0d0cdbe92_1254x1254.png 848w, https://substackcdn.com/image/fetch/$s_!Ku33!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdccd3d21-822d-41a3-8aaf-eff0d0cdbe92_1254x1254.png 1272w, https://substackcdn.com/image/fetch/$s_!Ku33!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdccd3d21-822d-41a3-8aaf-eff0d0cdbe92_1254x1254.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>The Application Controller is the heart of ArgoCD&#8230;It runs the reconciliation loop.</p><p>On every cycle, it gets the desired manifests from the Repo Server and the live cluster state from the Kubernetes API. Then it computes the diff between the two. Based on the application&#8217;s sync policy, it either alerts you to the difference or corrects it automatically.</p><p>The Application Controller runs as a StatefulSet.</p><p>(Scaling and sharding are covered later.)</p><h3>4. Redis</h3><p>Redis serves as a cache layer between Application Controller and Kubernetes API.</p><p>Without it, every UI load and every reconciliation cycle would need to query the Kubernetes API directly. At scale, this would generate unnecessary load on the Kubernetes API. Redis stores application state and cluster data so repeated reads hit the cache instead of the API.</p><p>It is worth being clear about what Redis is and is not in this context.</p><p>It is a cache, not a primary data store. ArgoCD does not depend on Redis for correctness. If Redis goes down, ArgoCD degrades gracefully. Performance suffers, but reconciliation continues.</p><h3>5. Notifications Controller</h3><p>Notifications Controller watches ArgoCD events &amp; route alerts to external systems.</p><p>When an application goes OutOfSync, a sync fails, or a health check degrades, the Notifications Controller sends an alert. It uses the destination you configured, such as Slack, email, PagerDuty, or a webhook. You define triggers and templates, and the controller handles the delivery.</p><p>In most production deployments, it gets installed alongside the rest of ArgoCD. It connects ArgoCD&#8217;s internal events with the people and systems responsible for responding to them.</p><p>Together, these components form ArgoCD's control plane...</p><p>Each component has a specific responsibility:</p><ul><li><p>API Server handles user requests.</p></li><li><p>Repo Server renders manifests.</p></li><li><p>Application Controller reconciles applications.</p></li><li><p>Redis caches data to reduce repeated API calls.</p></li><li><p>Notifications Controller reports important events.</p></li></ul><p>Together, these components keep the cluster aligned with the desired state in Git.</p><p>With the architecture covered, let&#8217;s see how these components work together during reconciliation&#8230;</p><h4 style="text-align: center;"><strong>&#167;</strong></h4><div class="callout-block" data-callout="true"><p style="text-align: center;"><em><strong><span data-color="#3a7bf9" style="color: rgb(58, 123, 249);">Reminder: this is a teaser of the subscriber-only newsletter, exclusive to my golden members.</span></strong></em></p><p>When you upgrade, you&#8217;ll get:</p><ul><li><p><strong>High-level architecture of real-world systems.</strong></p></li><li><p>Deep dive into how popular real-world systems work.</p></li><li><p><strong>How real-world systems handle scale, reliability &amp; performance.</strong></p></li></ul><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/subscribe&quot;,&quot;text&quot;:&quot;Unlock Full Access&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://newsletter.systemdesign.one/subscribe"><span>Unlock Full Access</span></a></p><p>(If this newsletter has helped you become a better software engineer, consider subscribing to support my work.)</p></div><h4 style="text-align: center;"><strong>&#167;</strong></h4><h2>How ArgoCD Keeps Your Cluster in Sync</h2>
      <p>
          <a href="https://newsletter.systemdesign.one/p/how-argocd-works-gitops-kubernetes">
              Read more
          </a>
      </p>
   ]]></content:encoded></item><item><title><![CDATA[11 Networking Concepts Explained in 11 Minutes]]></title><description><![CDATA[#195: Part 2 - Break Into Computer Networking]]></description><link>https://newsletter.systemdesign.one/p/computer-networking-fundamentals-for-software-engineers</link><guid isPermaLink="false">https://newsletter.systemdesign.one/p/computer-networking-fundamentals-for-software-engineers</guid><dc:creator><![CDATA[Neo Kim]]></dc:creator><pubDate>Fri, 23 Oct 2026 16:06:14 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/067daa95-a480-446b-b8f2-39b0005d6f6d_1280x720.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p></p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption"><span data-color="#3a7bf9" style="color: rgb(58, 123, 249);">Get my AI engineering playbook for FREE on newsletter signup:</span></p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><ul><li><p><em><a href="https://newsletter.systemdesign.one/p/computer-networking-fundamentals-for-software-engineers/?action=share">Share this letter</a> &amp; I&#8217;ll send you some rewards for the referrals.</em></p></li></ul><div><hr></div><p>Computer networking is an important part of ALL systems you&#8217;ll work on in your engineering career.</p><p>While junior developers may get by with writing code, senior engineers are valued for their ability to understand how systems communicate &amp; troubleshoot network problems.</p><blockquote><p><em>What happens when a network connection fails?</em></p><p><em>How does data travel between devices &amp; servers?</em></p><p><em>How do we diagnose slow connections, packet loss &amp; routing problems?</em></p></blockquote><p>These questions form the foundation of every reliable, high-performing system.</p><p>Onward.</p><div><hr></div><div class="callout-block" data-callout="true"><h3><strong><a href="https://getunblocked.com/events/how-to-stop-rewriting-your-agent-s-code-oct-28/?utm_source=systemdesign&amp;utm_medium=email&amp;utm_campaign=primary">[Webinar] How to stop rewriting your agent&#8217;s code</a></strong></h3><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://getunblocked.com/events/how-to-stop-rewriting-your-agent-s-code-oct-28/?utm_source=systemdesign&amp;utm_medium=email&amp;utm_campaign=primary" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!wgp1!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c47510d-48c8-4f8b-9093-e2c256c980f3_1600x900.png 424w, https://substackcdn.com/image/fetch/$s_!wgp1!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c47510d-48c8-4f8b-9093-e2c256c980f3_1600x900.png 848w, https://substackcdn.com/image/fetch/$s_!wgp1!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c47510d-48c8-4f8b-9093-e2c256c980f3_1600x900.png 1272w, https://substackcdn.com/image/fetch/$s_!wgp1!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c47510d-48c8-4f8b-9093-e2c256c980f3_1600x900.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!wgp1!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c47510d-48c8-4f8b-9093-e2c256c980f3_1600x900.png" width="1456" height="819" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/7c47510d-48c8-4f8b-9093-e2c256c980f3_1600x900.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:819,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:560005,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:&quot;https://getunblocked.com/events/how-to-stop-rewriting-your-agent-s-code-oct-28/?utm_source=systemdesign&amp;utm_medium=email&amp;utm_campaign=primary&quot;,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://newsletter.systemdesign.one/i/221150475?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c47510d-48c8-4f8b-9093-e2c256c980f3_1600x900.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!wgp1!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c47510d-48c8-4f8b-9093-e2c256c980f3_1600x900.png 424w, https://substackcdn.com/image/fetch/$s_!wgp1!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c47510d-48c8-4f8b-9093-e2c256c980f3_1600x900.png 848w, https://substackcdn.com/image/fetch/$s_!wgp1!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c47510d-48c8-4f8b-9093-e2c256c980f3_1600x900.png 1272w, https://substackcdn.com/image/fetch/$s_!wgp1!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c47510d-48c8-4f8b-9093-e2c256c980f3_1600x900.png 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Right now, you&#8217;re your agent&#8217;s context engine. You write the rules files, maintain the skills, and paste in the Slack thread it missed.</p><p>But hand-curated context goes stale. And when there are conflicting answers, your agent can&#8217;t tell which is right. You pay for it in higher token bills and longer reviews.</p><p><a href="https://getunblocked.com/events/how-to-stop-rewriting-your-agent-s-code-oct-28/?utm_source=systemdesign&amp;utm_medium=email&amp;utm_campaign=primary">Join us live on Oct 28 (FREE)</a> to see:</p><ul><li><p>The 8-levels of AI maturity and where teams get stuck</p></li><li><p>How a context engine curates context and resolves conflicts before your agent writes a line of code</p></li><li><p>Live demo: token savings in the same coding task, and use cases beyond code gen</p></li></ul><p>Bonus: 2 open source tools you can use today.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://getunblocked.com/events/how-to-stop-rewriting-your-agent-s-code-oct-28/?utm_source=systemdesign&amp;utm_medium=email&amp;utm_campaign=primary&quot;,&quot;text&quot;:&quot;Register now&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://getunblocked.com/events/how-to-stop-rewriting-your-agent-s-code-oct-28/?utm_source=systemdesign&amp;utm_medium=email&amp;utm_campaign=primary"><span>Register now</span></a></p><p><em>(Thanks to <a href="https://getunblocked.com/events/how-to-stop-rewriting-your-agent-s-code-oct-28/?utm_source=systemdesign&amp;utm_medium=email&amp;utm_campaign=primary">Unblocked</a> for partnering on this newsletter.)</em></p></div><div><hr></div><h1>Computer Networking Fundamentals</h1><p>Here are all the key concepts that you need to understand computer networking &amp; become a better engineer:</p><h2>1. Ethernet</h2><blockquote><p>Ethernet is a networking technology that connects devices within a Local Area Network (<strong>LAN</strong><a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-1" href="#footnote-1" target="_self">1</a>).</p></blockquote><p><em>How it works</em><a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-2" href="#footnote-2" target="_self">2</a><em>:</em></p><ol><li><p><strong>Framing:</strong> The sender wraps an IP packet inside an Ethernet frame<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-3" href="#footnote-3" target="_self">3</a>.</p></li><li><p><strong>Addressing:</strong> Frame contains source &amp; destination Media Access Control (<strong>MAC</strong>) addresses.</p></li><li><p><strong>Transmission:</strong> The network interface sends the frame over a physical link.</p></li><li><p><strong>Switching:</strong> Network switches forward frames toward their destinations using MAC addresses.</p></li><li><p><strong>Error detection:</strong> The receiver checks the frame&#8217;s Frame Check Sequence (<strong>FCS</strong>) to detect transmission errors.</p></li></ol><p>Typical Ethernet speeds include 1 Gbps for desktops, 10 or 25 Gbps for servers, and 100+ Gbps for data center networks.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!T4i1!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fca8beb0f-bcc2-4e24-a7d3-b15c13e7edaf_1536x1024.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!T4i1!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fca8beb0f-bcc2-4e24-a7d3-b15c13e7edaf_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!T4i1!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fca8beb0f-bcc2-4e24-a7d3-b15c13e7edaf_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!T4i1!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fca8beb0f-bcc2-4e24-a7d3-b15c13e7edaf_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!T4i1!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fca8beb0f-bcc2-4e24-a7d3-b15c13e7edaf_1536x1024.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!T4i1!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fca8beb0f-bcc2-4e24-a7d3-b15c13e7edaf_1536x1024.png" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/ca8beb0f-bcc2-4e24-a7d3-b15c13e7edaf_1536x1024.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!T4i1!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fca8beb0f-bcc2-4e24-a7d3-b15c13e7edaf_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!T4i1!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fca8beb0f-bcc2-4e24-a7d3-b15c13e7edaf_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!T4i1!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fca8beb0f-bcc2-4e24-a7d3-b15c13e7edaf_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!T4i1!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fca8beb0f-bcc2-4e24-a7d3-b15c13e7edaf_1536x1024.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><div class="callout-block" data-callout="true"><p>Imagine delivering packages between offices in the same building.</p><p>Each package carries the sender&#8217;s &amp; recipient&#8217;s room numbers. A mailroom reads the destination and forwards the package to the correct office.</p></div><h2>2. MAC address</h2><blockquote><p>A media access control (<strong>MAC</strong>) address<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-4" href="#footnote-4" target="_self">4</a> identifies a network card. They help devices communicate within a &#8220;local&#8221; network.</p></blockquote><p>Every card ships with a 48-bit address, written as six pairs of hexadecimal digits. The first half identifies the manufacturer, while the second half identifies the card.</p><p>IP addresses move packets between networks, and MAC addresses move frames within one network. A device learns its neighbor&#8217;s MAC address by broadcasting a question through the address resolution protocol<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-5" href="#footnote-5" target="_self">5</a> (<strong>ARP</strong>), and then caches that answer.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!fznl!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F93f7620a-d62a-418b-9c38-6551f1b8a45e_1536x1024.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!fznl!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F93f7620a-d62a-418b-9c38-6551f1b8a45e_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!fznl!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F93f7620a-d62a-418b-9c38-6551f1b8a45e_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!fznl!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F93f7620a-d62a-418b-9c38-6551f1b8a45e_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!fznl!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F93f7620a-d62a-418b-9c38-6551f1b8a45e_1536x1024.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!fznl!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F93f7620a-d62a-418b-9c38-6551f1b8a45e_1536x1024.png" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/93f7620a-d62a-418b-9c38-6551f1b8a45e_1536x1024.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!fznl!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F93f7620a-d62a-418b-9c38-6551f1b8a45e_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!fznl!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F93f7620a-d62a-418b-9c38-6551f1b8a45e_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!fznl!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F93f7620a-d62a-418b-9c38-6551f1b8a45e_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!fznl!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F93f7620a-d62a-418b-9c38-6551f1b8a45e_1536x1024.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><div class="callout-block" data-callout="true"><p>Think of an IP address as a building&#8217;s street address and a MAC address as an apartment number.</p></div><h2>3. Network switch</h2><blockquote><p>A switch<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-6" href="#footnote-6" target="_self">6</a> forwards frames between devices on the &#8220;same&#8221; network, using their MAC addresses.</p></blockquote><p>A switch learns devices that are reachable through each port by reading the source MAC address of incoming frames. Then it stores these MAC address-to-port mappings in a <em>forwarding</em> <em>table.</em><a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-7" href="#footnote-7" target="_self">7</a></p><p><em>How it works:</em></p><ul><li><p><strong>Known destination:</strong> The switch forwards the frame through the port associated with the destination MAC address.</p></li><li><p><strong>Unknown destination:</strong> The switch floods the frame through all other ports in the same VLAN until it learns where the destination is located.</p></li></ul><p>A switch forwards Ethernet frames within a local network using MAC addresses, while a router forwards IP packets between networks using IP addresses.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!Obzz!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe68cf883-3fd3-40ee-a21e-e15c621f6a89_1536x1024.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Obzz!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe68cf883-3fd3-40ee-a21e-e15c621f6a89_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!Obzz!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe68cf883-3fd3-40ee-a21e-e15c621f6a89_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!Obzz!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe68cf883-3fd3-40ee-a21e-e15c621f6a89_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!Obzz!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe68cf883-3fd3-40ee-a21e-e15c621f6a89_1536x1024.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!Obzz!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe68cf883-3fd3-40ee-a21e-e15c621f6a89_1536x1024.png" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/e68cf883-3fd3-40ee-a21e-e15c621f6a89_1536x1024.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!Obzz!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe68cf883-3fd3-40ee-a21e-e15c621f6a89_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!Obzz!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe68cf883-3fd3-40ee-a21e-e15c621f6a89_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!Obzz!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe68cf883-3fd3-40ee-a21e-e15c621f6a89_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!Obzz!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe68cf883-3fd3-40ee-a21e-e15c621f6a89_1536x1024.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><div class="callout-block" data-callout="true"><p>Imagine a mailroom in an office building:</p><p>The mailroom learns which office belongs to each employee &amp; delivers letters directly to the correct room. If it doesn&#8217;t know the destination, it sends copies to every office until it learns where the recipient is.</p></div><h2>4. Wi-Fi</h2><blockquote><p>Wi-Fi<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-8" href="#footnote-8" target="_self">8</a> is a wireless networking technology that uses radio waves to transmit data between devices.</p></blockquote><p><em>How it works:</em></p><ul><li><p><strong>Wireless transmission:</strong> Devices send &amp; receive data using radio signals.</p></li><li><p><strong>Channel sharing:</strong> Devices use mechanisms such as Carrier Sense Multiple Access with Collision Avoidance<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-9" href="#footnote-9" target="_self">9</a> (<strong>CSMA/CA</strong>) to reduce transmission collisions.</p></li><li><p><strong>Frequency bands:</strong> Wi-Fi operates on 2.4 GHz, 5 GHz, and 6 GHz bands.</p></li><li><p><strong>Signal strength:</strong> Devices adjust transmission rates based on signal quality &amp; interference.</p></li><li><p><strong>Retransmissions:</strong> Lost/corrupted frames require retransmission; thus reducing throughput.</p></li></ul><p>The 2.4 GHz band typically provides longer range but has less available capacity &amp; more interference.</p><p>The 5 GHz &amp; 6 GHz bands offer wider channels and potentially higher speeds, but their signals have shorter range &amp; weaker penetration through obstacles.</p><p>Also devices with poor signal quality transmit at lower data rates and consume more airtime. This reduces the capacity available to other devices sharing the channel.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!nyeT!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff00e8d83-6a58-4322-b3d6-09dd0e854268_1536x1024.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!nyeT!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff00e8d83-6a58-4322-b3d6-09dd0e854268_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!nyeT!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff00e8d83-6a58-4322-b3d6-09dd0e854268_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!nyeT!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff00e8d83-6a58-4322-b3d6-09dd0e854268_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!nyeT!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff00e8d83-6a58-4322-b3d6-09dd0e854268_1536x1024.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!nyeT!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff00e8d83-6a58-4322-b3d6-09dd0e854268_1536x1024.png" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/f00e8d83-6a58-4322-b3d6-09dd0e854268_1536x1024.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!nyeT!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff00e8d83-6a58-4322-b3d6-09dd0e854268_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!nyeT!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff00e8d83-6a58-4322-b3d6-09dd0e854268_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!nyeT!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff00e8d83-6a58-4322-b3d6-09dd0e854268_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!nyeT!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff00e8d83-6a58-4322-b3d6-09dd0e854268_1536x1024.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><div class="callout-block" data-callout="true"><p>Imagine a room where people take turns speaking.</p><p>Everyone listens before talking to avoid interruptions. If someone far away speaks slowly, others have to wait longer to finish.</p></div><div><hr></div><div class="captioned-button-wrap" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/p/computer-networking-fundamentals-for-software-engineers?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;}" data-component-name="CaptionedButtonToDOM"><div class="preamble"><p class="cta-caption"><em><span data-color="#3a7bf9" style="color: rgb(58, 123, 249);">If something resonated with you today, share this letter. Because one idea, one action, can change everything.</span></em></p></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/p/computer-networking-fundamentals-for-software-engineers?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://newsletter.systemdesign.one/p/computer-networking-fundamentals-for-software-engineers?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p></div><div><hr></div><h2>5. Static IP address</h2><blockquote><p>A static IP address is a &#8220;fixed&#8221; IP address assigned to a device or network interface.</p></blockquote><p>You configure it manually or reserve an address through Dynamic Host Configuration Protocol (<strong>DHCP</strong>).</p><p>A static IP<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-10" href="#footnote-10" target="_self">10</a> typically remains the same across device reboots as long as its assignment remains valid. Servers, printers, routers &amp; other devices that need predictable network addresses use static IPs.</p><p>DNS records, firewall rules &amp; network configurations reference these stable addresses.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!cIYo!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6057ac7c-953e-49a3-85cf-308eda0a6203_1536x1024.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!cIYo!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6057ac7c-953e-49a3-85cf-308eda0a6203_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!cIYo!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6057ac7c-953e-49a3-85cf-308eda0a6203_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!cIYo!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6057ac7c-953e-49a3-85cf-308eda0a6203_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!cIYo!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6057ac7c-953e-49a3-85cf-308eda0a6203_1536x1024.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!cIYo!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6057ac7c-953e-49a3-85cf-308eda0a6203_1536x1024.png" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/6057ac7c-953e-49a3-85cf-308eda0a6203_1536x1024.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!cIYo!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6057ac7c-953e-49a3-85cf-308eda0a6203_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!cIYo!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6057ac7c-953e-49a3-85cf-308eda0a6203_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!cIYo!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6057ac7c-953e-49a3-85cf-308eda0a6203_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!cIYo!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6057ac7c-953e-49a3-85cf-308eda0a6203_1536x1024.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><div class="callout-block" data-callout="true"><p>Think of it as a reserved parking space with your name on the sign.</p><p>The space stays assigned to you, whether you manage the reservation yourself or someone else manages it.</p></div><h2>6. DHCP</h2><blockquote><p>Dynamic Host Configuration Protocol (<strong>DHCP</strong>) automatically assigns IP addresses &amp; other network settings to devices when they join a network.</p></blockquote><p><em>How it works:</em></p><ol><li><p><strong>Discover:</strong> The device broadcasts a message to find available DHCP servers.</p></li><li><p><strong>Offer:</strong> A DHCP server offers an IP address &amp; network configuration.</p></li><li><p><strong>Request:</strong> The device requests the offered address.</p></li><li><p><strong>Acknowledge:</strong> Server confirms the assignment &amp; grants an IP address <em>lease</em>.</p></li></ol><p>(This process is called DORA.)</p><p>Plus, DHCP<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-11" href="#footnote-11" target="_self">11</a> server provides a subnet mask, default gateway, and Domain Name System (<strong>DNS</strong>) servers.</p><p>Yet each IP address lease has a lifetime.</p><p>Devices typically renew leases before they expire. If a lease expires without renewal, the server reassigns the address to another device.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!Z8fp!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc7d8ab01-f9ba-4387-988a-345d603cab48_1536x1024.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Z8fp!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc7d8ab01-f9ba-4387-988a-345d603cab48_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!Z8fp!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc7d8ab01-f9ba-4387-988a-345d603cab48_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!Z8fp!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc7d8ab01-f9ba-4387-988a-345d603cab48_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!Z8fp!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc7d8ab01-f9ba-4387-988a-345d603cab48_1536x1024.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!Z8fp!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc7d8ab01-f9ba-4387-988a-345d603cab48_1536x1024.png" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/c7d8ab01-f9ba-4387-988a-345d603cab48_1536x1024.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!Z8fp!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc7d8ab01-f9ba-4387-988a-345d603cab48_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!Z8fp!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc7d8ab01-f9ba-4387-988a-345d603cab48_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!Z8fp!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc7d8ab01-f9ba-4387-988a-345d603cab48_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!Z8fp!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc7d8ab01-f9ba-4387-988a-345d603cab48_1536x1024.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><div class="callout-block" data-callout="true"><p>Imagine checking into a hotel:</p><p>The receptionist assigns you a room number for your stay. When you check out, the room becomes available for another guest.</p></div><h2>7. Default gateway</h2><blockquote><p>A default gateway is a router that forwards traffic to destinations outside a device&#8217;s local network when &#8220;no&#8221; specific route exists.</p></blockquote><p><em>How it works:</em></p><ul><li><p><strong>Local destination:</strong> The device sends traffic directly to another device on the same subnet.</p></li><li><p><strong>Remote destination:</strong> The device forwards traffic to its default gateway.</p></li><li><p><strong>Packet forwarding:</strong> Gateway uses its routing table to forward the packet toward its destination.</p></li></ul><p>A laptop with IP address <code>192.168.1.10/24</code> might use <code>192.168.1.1</code> as its default gateway.</p><p>The gateway<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-12" href="#footnote-12" target="_self">12</a> typically has an IP address on the same subnet,,, thus allowing the laptop to reach it directly.</p><p>An incorrect default gateway prevents a device from reaching &#8220;external&#8221; networks, even if local communication works.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!QN5s!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faa8257ee-c1b4-4716-9bcd-10b78b7d5683_1536x1024.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!QN5s!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faa8257ee-c1b4-4716-9bcd-10b78b7d5683_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!QN5s!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faa8257ee-c1b4-4716-9bcd-10b78b7d5683_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!QN5s!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faa8257ee-c1b4-4716-9bcd-10b78b7d5683_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!QN5s!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faa8257ee-c1b4-4716-9bcd-10b78b7d5683_1536x1024.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!QN5s!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faa8257ee-c1b4-4716-9bcd-10b78b7d5683_1536x1024.png" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/aa8257ee-c1b4-4716-9bcd-10b78b7d5683_1536x1024.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!QN5s!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faa8257ee-c1b4-4716-9bcd-10b78b7d5683_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!QN5s!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faa8257ee-c1b4-4716-9bcd-10b78b7d5683_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!QN5s!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faa8257ee-c1b4-4716-9bcd-10b78b7d5683_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!QN5s!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faa8257ee-c1b4-4716-9bcd-10b78b7d5683_1536x1024.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><div class="callout-block" data-callout="true"><p>Think of a building with one main exit.</p><p>You move between rooms inside the building without leaving. To reach another building, you must use the main exit.</p></div><h2>8. Static route</h2><blockquote><p>A static route is a (manually) configured entry in a routing table that tells a router where to forward packets.</p></blockquote><p>You specify a destination IP range &amp; an outgoing network interface.</p><p><em>How it works:</em></p><ul><li><p><strong>Configure a route:</strong> Define a destination network, such as <code>10.0.2.0/24</code>.</p></li><li><p><strong>Specify the next hop:</strong> Point the route to another router, such as <code>192.168.1.1</code>.</p></li><li><p><strong>Forward packets:</strong> The router uses the route to forward matching traffic toward its destination.</p></li></ul><p>Yet static routes<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-13" href="#footnote-13" target="_self">13</a> do not automatically learn new network paths.</p><p>If a network path fails, the static route still remains configured. Some implementations detect failures &amp; deactivate/replace the route.</p><p>Static routes work well for cloud networking configurations.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!Sbj4!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8fa1eba5-be8c-4ace-99f0-d71726c2d0d3_1536x1024.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Sbj4!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8fa1eba5-be8c-4ace-99f0-d71726c2d0d3_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!Sbj4!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8fa1eba5-be8c-4ace-99f0-d71726c2d0d3_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!Sbj4!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8fa1eba5-be8c-4ace-99f0-d71726c2d0d3_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!Sbj4!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8fa1eba5-be8c-4ace-99f0-d71726c2d0d3_1536x1024.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!Sbj4!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8fa1eba5-be8c-4ace-99f0-d71726c2d0d3_1536x1024.png" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/8fa1eba5-be8c-4ace-99f0-d71726c2d0d3_1536x1024.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!Sbj4!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8fa1eba5-be8c-4ace-99f0-d71726c2d0d3_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!Sbj4!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8fa1eba5-be8c-4ace-99f0-d71726c2d0d3_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!Sbj4!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8fa1eba5-be8c-4ace-99f0-d71726c2d0d3_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!Sbj4!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8fa1eba5-be8c-4ace-99f0-d71726c2d0d3_1536x1024.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><div class="callout-block" data-callout="true"><p>Imagine a road sign directing drivers toward a specific town.</p><p>The sign stays in place even when the road closes, unless someone updates it or a traffic management system redirects drivers.</p></div><div><hr></div><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption"><em><span data-color="#3a7bf9" style="color: rgb(58, 123, 249);">Enter your email to join the inner circle and receive exclusive insights &amp; tips:</span></em></p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><div><hr></div><h2>9. OSPF</h2><blockquote><p>Open Shortest Path First (<strong>OSPF</strong>) is a dynamic routing protocol that lets routers automatically discover &amp; update routes within an Autonomous System (<strong>AS</strong>).</p></blockquote><p><em>How it works:</em></p><ol><li><p><strong>Discover neighbors:</strong> Routers exchange Hello packets to discover &amp; maintain neighboring routers.</p></li><li><p><strong>Share topology:</strong> Routers flood Link-State Advertisements<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-14" href="#footnote-14" target="_self">14</a> (<strong>LSA</strong>s) to share network information within an OSPF area.</p></li><li><p><strong>Build a network map:</strong> Routers in the same area maintain a synchronized link-state database.</p></li><li><p><strong>Calculate routes:</strong> Each router uses Dijkstra&#8217;s Shortest Path First<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-15" href="#footnote-15" target="_self">15</a> algorithm to calculate the lowest-cost paths.</p></li><li><p><strong>Adapt to failures:</strong> When a link fails, routers propagate updated information and recalculate affected routes.</p></li></ol><p>OSPF<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-16" href="#footnote-16" target="_self">16</a> selects paths based on configurable interface costs.</p><p>Bandwidth influences these costs, but administrators could adjust them manually.</p><p>Plus, OSPF reroutes traffic after network failures. Convergence time depends on failure detection, topology size &amp; router configuration.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!sbTG!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fccb4e72a-e860-4583-b9bc-ee7bc6851442_1536x1024.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!sbTG!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fccb4e72a-e860-4583-b9bc-ee7bc6851442_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!sbTG!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fccb4e72a-e860-4583-b9bc-ee7bc6851442_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!sbTG!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fccb4e72a-e860-4583-b9bc-ee7bc6851442_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!sbTG!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fccb4e72a-e860-4583-b9bc-ee7bc6851442_1536x1024.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!sbTG!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fccb4e72a-e860-4583-b9bc-ee7bc6851442_1536x1024.png" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/ccb4e72a-e860-4583-b9bc-ee7bc6851442_1536x1024.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!sbTG!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fccb4e72a-e860-4583-b9bc-ee7bc6851442_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!sbTG!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fccb4e72a-e860-4583-b9bc-ee7bc6851442_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!sbTG!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fccb4e72a-e860-4583-b9bc-ee7bc6851442_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!sbTG!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fccb4e72a-e860-4583-b9bc-ee7bc6851442_1536x1024.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><div class="callout-block" data-callout="true"><p>Imagine every driver in a city using a shared road map.</p><p>When a road closes, the map updates, and each driver calculates a new route based on the available roads.</p></div><h2>10. ICMP</h2><blockquote><p>Internet Control Message Protocol (<strong>ICMP</strong><a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-17" href="#footnote-17" target="_self">17</a>) carries network error messages &amp; diagnostic information. It helps devices identify problems such as unreachable destinations, expired packet lifetimes, and packets too large for a network link.</p></blockquote><p><em>How it works:</em></p><ul><li><p><strong>Destination unreachable:</strong> A router or destination host sends an ICMP message when it cannot deliver a packet.</p></li><li><p><strong>Time exceeded:</strong> A router sends an ICMP message when a packet&#8217;s Time to Live (<strong>TTL</strong>) or Hop Limit reaches zero.</p></li><li><p><strong>Packet too big:</strong> ICMP reports when a packet exceeds the supported Maximum Transmission Unit (<strong>MTU</strong>).</p></li><li><p><strong>Echo request and reply:</strong> Tools such as <code>ping</code> use ICMP to test whether a destination responds.</p></li></ul><p>Yet ICMP messages are NOT guaranteed to arrive&#8230;</p><p>Firewalls, rate limiting &amp; network conditions could block/drop them.</p><p>ICMP helps diagnose network problems but does not directly retransmit lost application data or repair failed connections.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!BTLD!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F67f9616c-41c5-42c8-a674-aadc2a594352_1536x1024.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!BTLD!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F67f9616c-41c5-42c8-a674-aadc2a594352_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!BTLD!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F67f9616c-41c5-42c8-a674-aadc2a594352_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!BTLD!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F67f9616c-41c5-42c8-a674-aadc2a594352_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!BTLD!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F67f9616c-41c5-42c8-a674-aadc2a594352_1536x1024.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!BTLD!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F67f9616c-41c5-42c8-a674-aadc2a594352_1536x1024.png" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/67f9616c-41c5-42c8-a674-aadc2a594352_1536x1024.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!BTLD!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F67f9616c-41c5-42c8-a674-aadc2a594352_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!BTLD!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F67f9616c-41c5-42c8-a674-aadc2a594352_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!BTLD!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F67f9616c-41c5-42c8-a674-aadc2a594352_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!BTLD!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F67f9616c-41c5-42c8-a674-aadc2a594352_1536x1024.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><div class="callout-block" data-callout="true"><p>Imagine sending a letter that cannot reach its destination&#8230;</p><p>The post office returns a notice explaining why delivery failed, such as an invalid address or an unavailable route.</p><p>The notice helps you identify the problem, but it does not deliver the original letter.</p></div><h2>11. Ping</h2><blockquote><p>Ping<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-18" href="#footnote-18" target="_self">18</a> is a network diagnostic tool that tests whether a destination host responds &amp; measures round-trip time (<strong>RTT</strong>). It uses ICMP Echo Request &amp; Echo Reply messages.</p></blockquote><p><em>How it works:</em></p><ol><li><p><strong>Send request:</strong> Ping sends an ICMP Echo Request to the destination.</p></li><li><p><strong>Receive reply:</strong> If the destination responds, it sends an ICMP Echo Reply.</p></li><li><p><strong>Measure latency:</strong> Ping calculates time between sending request &amp; receiving reply.</p></li><li><p><strong>Track responses:</strong> Ping reports round-trip times &amp; percentage of requests without replies.</p></li></ol><p>Repeated tests help identify latency spikes, intermittent connectivity &amp; potential packet loss.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!ZYSY!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa530971d-9c2c-4966-b7f4-2f8bc2d3cc0d_1536x1024.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!ZYSY!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa530971d-9c2c-4966-b7f4-2f8bc2d3cc0d_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!ZYSY!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa530971d-9c2c-4966-b7f4-2f8bc2d3cc0d_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!ZYSY!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa530971d-9c2c-4966-b7f4-2f8bc2d3cc0d_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!ZYSY!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa530971d-9c2c-4966-b7f4-2f8bc2d3cc0d_1536x1024.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!ZYSY!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa530971d-9c2c-4966-b7f4-2f8bc2d3cc0d_1536x1024.png" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/a530971d-9c2c-4966-b7f4-2f8bc2d3cc0d_1536x1024.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!ZYSY!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa530971d-9c2c-4966-b7f4-2f8bc2d3cc0d_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!ZYSY!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa530971d-9c2c-4966-b7f4-2f8bc2d3cc0d_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!ZYSY!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa530971d-9c2c-4966-b7f4-2f8bc2d3cc0d_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!ZYSY!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa530971d-9c2c-4966-b7f4-2f8bc2d3cc0d_1536x1024.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><div class="callout-block" data-callout="true"><p>Imagine knocking on a door &amp; timing how long someone takes to answer.</p><p>A quick response suggests someone is available. A delayed response takes longer, while silence only means nobody answered.</p></div><div><hr></div><h2>Final Words</h2><p>Now you understand the basic networking concepts that help devices communicate, it&#8217;s time to explore how they work in real systems.</p><p>Look at the apps &amp; networks you use every day and think about what happens behind the scenes.</p><blockquote><p><em>What happens when a network link fails?</em></p><p><em>How does your laptop connect to the internet?</em></p><p><em>How do routers find the right path for packets?</em></p><p><em>How do engineers diagnose slow or failed connections?</em></p></blockquote><p>After understanding these concepts, start applying them in your own work.</p><p>Use tools like <code>ping</code>, <code>mtr</code>, and <code>tcpdump</code> to explore network behavior. Experiment with routing configurations and troubleshoot connectivity problems.</p><p>Remember, every experienced engineer started with the fundamentals. The more you practice, the easier it becomes to understand and troubleshoot complex networks.</p><div><hr></div><p>If you find this newsletter valuable, share it with a friend, and subscribe if you haven&#8217;t already. There are <a href="http://newsletter.systemdesign.one/subscribe?group=true">group discounts</a>, <a href="http://newsletter.systemdesign.one/subscribe?gift=true">gift options</a>, and <a href="https://newsletter.systemdesign.one/leaderboard">referral rewards</a> available.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe now&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://newsletter.systemdesign.one/subscribe?"><span>Subscribe now</span></a></p><div><hr></div><div class="captioned-image-container"><figure><a class="image-link image2" target="_blank" href="https://www.linkedin.com/in/nk-systemdesign-one/" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!68aQ!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe4eab3a0-fda8-4df8-b651-0f976b223e9a_320x320.png 424w, https://substackcdn.com/image/fetch/$s_!68aQ!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe4eab3a0-fda8-4df8-b651-0f976b223e9a_320x320.png 848w, https://substackcdn.com/image/fetch/$s_!68aQ!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe4eab3a0-fda8-4df8-b651-0f976b223e9a_320x320.png 1272w, https://substackcdn.com/image/fetch/$s_!68aQ!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe4eab3a0-fda8-4df8-b651-0f976b223e9a_320x320.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!68aQ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe4eab3a0-fda8-4df8-b651-0f976b223e9a_320x320.png" width="160" height="160" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/e4eab3a0-fda8-4df8-b651-0f976b223e9a_320x320.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:320,&quot;width&quot;:320,&quot;resizeWidth&quot;:160,&quot;bytes&quot;:121414,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:&quot;https://www.linkedin.com/in/nk-systemdesign-one/&quot;,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://newsletter.systemdesign.one/i/136161496?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe4eab3a0-fda8-4df8-b651-0f976b223e9a_320x320.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!68aQ!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe4eab3a0-fda8-4df8-b651-0f976b223e9a_320x320.png 424w, https://substackcdn.com/image/fetch/$s_!68aQ!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe4eab3a0-fda8-4df8-b651-0f976b223e9a_320x320.png 848w, https://substackcdn.com/image/fetch/$s_!68aQ!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe4eab3a0-fda8-4df8-b651-0f976b223e9a_320x320.png 1272w, https://substackcdn.com/image/fetch/$s_!68aQ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe4eab3a0-fda8-4df8-b651-0f976b223e9a_320x320.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div></div></div></a><figcaption class="image-caption"><strong>Find me elsewhere: <a href="https://www.linkedin.com/in/nk-systemdesign-one/">LinkedIn</a> | <a href="https://x.com/intent/follow?screen_name=systemdesignone">Twitter</a> | <a href="https://www.instagram.com/systemdesignone/">Instagram</a></strong></figcaption></figure></div><div><hr></div><p><strong>Want to reach 250K+ tech professionals at scale? </strong>&#128240;</p><p>If your company wants to reach 250K+ tech professionals, <a href="https://newsletter.systemdesign.one/p/sponsorship">advertise with me</a>.</p><div><hr></div><p>Thank you for supporting this newsletter.</p><p>You are now 250,001+ readers strong, very close to 251k. Let&#8217;s try to get 251k readers by 27 October. Consider sharing this letter with your friends and get rewards.</p><p>Y&#8217;all are the best.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!6oWl!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e739087-a910-4643-be36-997b6dd5b4af_800x500.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!6oWl!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e739087-a910-4643-be36-997b6dd5b4af_800x500.png 424w, https://substackcdn.com/image/fetch/$s_!6oWl!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e739087-a910-4643-be36-997b6dd5b4af_800x500.png 848w, https://substackcdn.com/image/fetch/$s_!6oWl!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e739087-a910-4643-be36-997b6dd5b4af_800x500.png 1272w, https://substackcdn.com/image/fetch/$s_!6oWl!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e739087-a910-4643-be36-997b6dd5b4af_800x500.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!6oWl!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e739087-a910-4643-be36-997b6dd5b4af_800x500.png" width="590" height="368.75" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/2e739087-a910-4643-be36-997b6dd5b4af_800x500.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:500,&quot;width&quot;:800,&quot;resizeWidth&quot;:590,&quot;bytes&quot;:87878,&quot;alt&quot;:&quot;system design newsletter&quot;,&quot;title&quot;:&quot;system design newsletter&quot;,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://newsletter.systemdesign.one/i/163380418?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e739087-a910-4643-be36-997b6dd5b4af_800x500.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="system design newsletter" title="system design newsletter" srcset="https://substackcdn.com/image/fetch/$s_!6oWl!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e739087-a910-4643-be36-997b6dd5b4af_800x500.png 424w, https://substackcdn.com/image/fetch/$s_!6oWl!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e739087-a910-4643-be36-997b6dd5b4af_800x500.png 848w, https://substackcdn.com/image/fetch/$s_!6oWl!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e739087-a910-4643-be36-997b6dd5b4af_800x500.png 1272w, https://substackcdn.com/image/fetch/$s_!6oWl!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e739087-a910-4643-be36-997b6dd5b4af_800x500.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/p/computer-networking-fundamentals-for-software-engineers?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://newsletter.systemdesign.one/p/computer-networking-fundamentals-for-software-engineers?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p><div><hr></div><h3>References</h3><ul><li><p>Google, <em>IPv6 Statistics</em>: <a href="https://www.google.com/intl/en/ipv6/statistics.html">https://www.google.com/intl/en/ipv6/statistics.html</a></p></li><li><p>Meta Engineering, <em>More details about the October 4 outage</em>: <a href="https://engineering.fb.com/2021/10/05/networking-traffic/outage-details/">https://engineering.fb.com/2021/10/05/networking-traffic/outage-details/</a></p></li><li><p>RFC 9293, <em>Transmission Control Protocol</em>: <a href="https://www.rfc-editor.org/rfc/rfc9293.html">https://www.rfc-editor.org/rfc/rfc9293.html</a></p></li><li><p>RFC 9000, <em>QUIC: A UDP-Based Multiplexed and Secure Transport</em>: <a href="https://www.rfc-editor.org/rfc/rfc9000.html">https://www.rfc-editor.org/rfc/rfc9000.html</a></p></li><li><p>RFC 9110, <em>HTTP Semantics</em>: <a href="https://www.rfc-editor.org/rfc/rfc9110.html">https://www.rfc-editor.org/rfc/rfc9110.html</a></p></li><li><p>RFC 9111, <em>HTTP Caching</em>: <a href="https://www.rfc-editor.org/rfc/rfc9111.html">https://www.rfc-editor.org/rfc/rfc9111.html</a></p></li><li><p>RFC 8446, <em>The Transport Layer Security Protocol Version 1.3</em>: <a href="https://www.rfc-editor.org/rfc/rfc8446.html">https://www.rfc-editor.org/rfc/rfc8446.html</a></p></li><li><p>RFC 1035, <em>Domain Names: Implementation and Specification</em>: <a href="https://www.rfc-editor.org/rfc/rfc1035.html">https://www.rfc-editor.org/rfc/rfc1035.html</a></p></li><li><p>RFC 4632, <em>Classless Inter-domain Routing</em>: <a href="https://www.rfc-editor.org/rfc/rfc4632.html">https://www.rfc-editor.org/rfc/rfc4632.html</a></p></li><li><p>RFC 4271, <em>A Border Gateway Protocol 4</em>: <a href="https://www.rfc-editor.org/rfc/rfc4271.html">https://www.rfc-editor.org/rfc/rfc4271.html</a></p></li><li><p>HTTP Archive, <em>Web Almanac 2025</em>: <a href="https://almanac.httparchive.org/en/2025/">https://almanac.httparchive.org/en/2025/</a></p></li></ul><div class="footnote" data-component-name="FootnoteToDOM"><a id="footnote-1" href="#footnote-anchor-1" class="footnote-number" contenteditable="false" target="_self">1</a><div class="footnote-content"><p>A network that connects devices within a limited geographic area, such as a home, office, or building, allowing them to communicate and share resources.</p></div></div><div class="footnote" data-component-name="FootnoteToDOM"><a id="footnote-2" href="#footnote-anchor-2" class="footnote-number" contenteditable="false" target="_self">2</a><div class="footnote-content"><p>Use <code>ethtool</code> to inspect Ethernet link speed &amp; status.</p></div></div><div class="footnote" data-component-name="FootnoteToDOM"><a id="footnote-3" href="#footnote-anchor-3" class="footnote-number" contenteditable="false" target="_self">3</a><div class="footnote-content"><p>A data unit at the data link layer that carries a payload, such as an IP packet, along with source and destination MAC addresses and error-detection information across an Ethernet network.</p></div></div><div class="footnote" data-component-name="FootnoteToDOM"><a id="footnote-4" href="#footnote-anchor-4" class="footnote-number" contenteditable="false" target="_self">4</a><div class="footnote-content"><p>Network switches use MAC addresses to forward Ethernet frames to the correct destination. Use <code>ip neigh</code> to inspect neighboring IP-to-MAC mappings and Wireshark to analyze ARP traffic.</p></div></div><div class="footnote" data-component-name="FootnoteToDOM"><a id="footnote-5" href="#footnote-anchor-5" class="footnote-number" contenteditable="false" target="_self">5</a><div class="footnote-content"><p>A network protocol that discovers the MAC address associated with a known IPv4 address on a local network.</p></div></div><div class="footnote" data-component-name="FootnoteToDOM"><a id="footnote-6" href="#footnote-anchor-6" class="footnote-number" contenteditable="false" target="_self">6</a><div class="footnote-content"><p>Network switches connect devices within local networks &amp; data centers.</p><p>They use MAC address tables to forward Ethernet frames efficiently, thus reducing unnecessary traffic. Use <code>bridge fdb</code> to inspect forwarding entries &amp; GNS3 to simulate switched networks.</p></div></div><div class="footnote" data-component-name="FootnoteToDOM"><a id="footnote-7" href="#footnote-anchor-7" class="footnote-number" contenteditable="false" target="_self">7</a><div class="footnote-content"><p>Imagine a switch connecting 3 devices:</p><ol><li><p>Laptop A sends a frame through Port 1.</p></li><li><p>The switch reads the frame&#8217;s source MAC address (MAC A).</p></li><li><p>It records that MAC A is reachable through Port 1.</p></li><li><p>When Laptop B sends a frame through Port 2, the switch learns that MAC B is reachable through Port 2.</p></li><li><p>Later, when a frame arrives for MAC B, the switch forwards it through Port 2.</p></li></ol><p>The switch stores this information in a MAC address table, called the forwarding table.</p></div></div><div class="footnote" data-component-name="FootnoteToDOM"><a id="footnote-8" href="#footnote-anchor-8" class="footnote-number" contenteditable="false" target="_self">8</a><div class="footnote-content"><p>Wi-Fi performance affects application latency, throughput, and reliability, especially for mobile devices &amp; laptops.</p><p>Use <code>iperf3</code> to measure throughput and Wireshark in monitor mode to inspect wireless traffic.</p></div></div><div class="footnote" data-component-name="FootnoteToDOM"><a id="footnote-9" href="#footnote-anchor-9" class="footnote-number" contenteditable="false" target="_self">9</a><div class="footnote-content"><p>Carrier Sense Multiple Access with Collision Avoidance (CSMA/CA): A network access method used by Wi-Fi devices to listen for channel activity and reduce collisions by waiting before transmitting data.</p></div></div><div class="footnote" data-component-name="FootnoteToDOM"><a id="footnote-10" href="#footnote-anchor-10" class="footnote-number" contenteditable="false" target="_self">10</a><div class="footnote-content"><p>They help you:</p><ul><li><p>Access devices without tracking changing IP addresses.</p></li><li><p>Configure stable DNS records &amp; firewall allowlists.</p></li><li><p>Simplify network monitoring &amp; troubleshooting.</p></li></ul></div></div><div class="footnote" data-component-name="FootnoteToDOM"><a id="footnote-11" href="#footnote-anchor-11" class="footnote-number" contenteditable="false" target="_self">11</a><div class="footnote-content"><p>DHCP simplifies IP address management &amp; reduces manual configuration errors.</p><p>Office networks use DHCP to auto-configure laptops, phones, and other devices. Cloud providers use DHCP-based mechanisms to deliver network settings to virtual machines.</p><p>Use ISC Kea to manage DHCP servers or your cloud provider&#8217;s built-in networking services.</p></div></div><div class="footnote" data-component-name="FootnoteToDOM"><a id="footnote-12" href="#footnote-anchor-12" class="footnote-number" contenteditable="false" target="_self">12</a><div class="footnote-content"><p>Default gateways allow servers, laptops &amp; other devices to communicate with external networks. An incorrect gateway configuration breaks internet access or communication between subnets.</p><p>Use <code>ip route</code> to inspect the default gateway and routing table.</p><p>For high availability, technologies such as Virtual Router Redundancy Protocol (<strong>VRRP</strong>) can provide a redundant default gateway.</p></div></div><div class="footnote" data-component-name="FootnoteToDOM"><a id="footnote-13" href="#footnote-anchor-13" class="footnote-number" contenteditable="false" target="_self">13</a><div class="footnote-content"><p>Static routes help control how traffic moves between subnets, cloud networks, and on-premises infrastructure.</p><p>Cloud route tables typically use manually configured routes to direct traffic through internet gateways, NAT gateways, or VPN connections.</p><p>Use <code>ip route add</code> or Terraform to configure routes.</p></div></div><div class="footnote" data-component-name="FootnoteToDOM"><a id="footnote-14" href="#footnote-anchor-14" class="footnote-number" contenteditable="false" target="_self">14</a><div class="footnote-content"><p>Link-State Advertisements (LSAs): Messages used by OSPF routers to share network topology information, including links and their costs, so routers can calculate the shortest paths.</p></div></div><div class="footnote" data-component-name="FootnoteToDOM"><a id="footnote-15" href="#footnote-anchor-15" class="footnote-number" contenteditable="false" target="_self">15</a><div class="footnote-content"><p>Dijkstra&#8217;s Shortest Path First (SPF) algorithm: A graph algorithm that calculates the lowest-cost paths from a source node to all other reachable nodes in a network.</p></div></div><div class="footnote" data-component-name="FootnoteToDOM"><a id="footnote-16" href="#footnote-anchor-16" class="footnote-number" contenteditable="false" target="_self">16</a><div class="footnote-content"><p>OSPF lets enterprise networks automatically adapt to topology changes without manually updating every router.</p></div></div><div class="footnote" data-component-name="FootnoteToDOM"><a id="footnote-17" href="#footnote-anchor-17" class="footnote-number" contenteditable="false" target="_self">17</a><div class="footnote-content"><p>ICMP helps engineers diagnose connectivity issues, routing failures, and MTU problems.</p><p>Use <code>ping</code> to test reachability, <code>traceroute</code> or <code>mtr</code> to investigate network paths, and <code>tcpdump</code> to inspect ICMP messages.</p></div></div><div class="footnote" data-component-name="FootnoteToDOM"><a id="footnote-18" href="#footnote-anchor-18" class="footnote-number" contenteditable="false" target="_self">18</a><div class="footnote-content"><p>Ping helps you investigate network reachability &amp; latency problems.</p><p>Use <code>ping</code> to check whether a destination responds and <code>mtr</code> to investigate latency and packet loss across network paths.</p><p>A successful ping does not guarantee that an application is working. A server could respond to ICMP while its HTTP service or database port remains unavailable.</p></div></div>]]></content:encoded></item><item><title><![CDATA[The 80/20 of AI Concepts]]></title><description><![CDATA[#194: A Practical Guide to 14 Advanced AI Concepts]]></description><link>https://newsletter.systemdesign.one/p/advanced-ai-concepts-software-engineers</link><guid isPermaLink="false">https://newsletter.systemdesign.one/p/advanced-ai-concepts-software-engineers</guid><dc:creator><![CDATA[Neo Kim]]></dc:creator><pubDate>Thu, 22 Oct 2026 14:03:02 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/47b08c4b-e403-4640-b3c7-cef2a116010e_1280x720.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p></p><div class="callout-block" data-callout="true"><h2><a href="https://inngest.link/jtH41MP">Build for the long run</a></h2><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://inngest.link/jtH41MP" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!OCoh!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9dfec45c-1e26-4436-9cea-7fa5dcf7a0c3_2190x1061.png 424w, https://substackcdn.com/image/fetch/$s_!OCoh!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9dfec45c-1e26-4436-9cea-7fa5dcf7a0c3_2190x1061.png 848w, https://substackcdn.com/image/fetch/$s_!OCoh!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9dfec45c-1e26-4436-9cea-7fa5dcf7a0c3_2190x1061.png 1272w, https://substackcdn.com/image/fetch/$s_!OCoh!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9dfec45c-1e26-4436-9cea-7fa5dcf7a0c3_2190x1061.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!OCoh!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9dfec45c-1e26-4436-9cea-7fa5dcf7a0c3_2190x1061.png" width="1456" height="705" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/9dfec45c-1e26-4436-9cea-7fa5dcf7a0c3_2190x1061.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:705,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:5778022,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:&quot;https://inngest.link/jtH41MP&quot;,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://newsletter.systemdesign.one/i/214903444?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9dfec45c-1e26-4436-9cea-7fa5dcf7a0c3_2190x1061.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!OCoh!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9dfec45c-1e26-4436-9cea-7fa5dcf7a0c3_2190x1061.png 424w, https://substackcdn.com/image/fetch/$s_!OCoh!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9dfec45c-1e26-4436-9cea-7fa5dcf7a0c3_2190x1061.png 848w, https://substackcdn.com/image/fetch/$s_!OCoh!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9dfec45c-1e26-4436-9cea-7fa5dcf7a0c3_2190x1061.png 1272w, https://substackcdn.com/image/fetch/$s_!OCoh!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9dfec45c-1e26-4436-9cea-7fa5dcf7a0c3_2190x1061.png 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Long-running agents need endurance.</p><p>They must endure waiting for users and weather tool failures. They must take each new jump in model intelligence in stride.</p><p>Inngest makes long-running agents durable, observable and improvable over time.</p><p><strong><a href="https://inngest.link/jtH41MP"><mark data-color="rgb(255, 255, 0)" style="background-color: rgb(255, 255, 0); color: rgb(0, 0, 0);">Build for the long run with Inngest &#187;</mark></a></strong></p><p><em>(Thanks to <a href="https://inngest.link/jtH41MP">Inngest</a> for partnering on this newsletter.)</em></p></div><h4 style="text-align: center;"><strong>&#167;</strong></h4><p>The 80/20 rule states that 80% of results come from 20% of actions.</p><p>It is the ONE rule you must obey &amp; this is the most important rule when it comes to learning something new.</p><p>Let me explain...</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!TY-V!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F82835cca-bc51-44d5-b354-c916f0689441_1200x630.gif" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!TY-V!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F82835cca-bc51-44d5-b354-c916f0689441_1200x630.gif 424w, https://substackcdn.com/image/fetch/$s_!TY-V!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F82835cca-bc51-44d5-b354-c916f0689441_1200x630.gif 848w, https://substackcdn.com/image/fetch/$s_!TY-V!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F82835cca-bc51-44d5-b354-c916f0689441_1200x630.gif 1272w, https://substackcdn.com/image/fetch/$s_!TY-V!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F82835cca-bc51-44d5-b354-c916f0689441_1200x630.gif 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!TY-V!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F82835cca-bc51-44d5-b354-c916f0689441_1200x630.gif" width="728" height="382.2" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/82835cca-bc51-44d5-b354-c916f0689441_1200x630.gif&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:630,&quot;width&quot;:1200,&quot;resizeWidth&quot;:728,&quot;bytes&quot;:362861,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/gif&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://newsletter.systemdesign.one/i/221435962?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F82835cca-bc51-44d5-b354-c916f0689441_1200x630.gif&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!TY-V!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F82835cca-bc51-44d5-b354-c916f0689441_1200x630.gif 424w, https://substackcdn.com/image/fetch/$s_!TY-V!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F82835cca-bc51-44d5-b354-c916f0689441_1200x630.gif 848w, https://substackcdn.com/image/fetch/$s_!TY-V!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F82835cca-bc51-44d5-b354-c916f0689441_1200x630.gif 1272w, https://substackcdn.com/image/fetch/$s_!TY-V!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F82835cca-bc51-44d5-b354-c916f0689441_1200x630.gif 1456w" sizes="100vw"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Imagine you want to fill a bottle with pebbles, sand &amp; water.</p><p>If you pour in the sand &amp; water first, you&#8217;ll struggle to fit the big pebbles.</p><p>But if you start with the pebbles, the sand <em>automagically</em> fills the gaps between them, and the water fills the remaining space.</p><p>i.e., do the 20% that makes the real difference (80%) first.</p><p>And this is why I&#8217;m writing these letters<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-1" href="#footnote-1" target="_self">1</a> on AI engineering concepts for you.</p><p>I want you to win in the AI era by understanding the &#8220;fundamental&#8221; AI concepts.</p><p><em>Inside this newsletter, you&#8217;ll find:</em></p><ol start="35"><li><p>Mixture-of-Experts,</p></li><li><p>Large Language Model Serving &amp; Inference Engines,</p></li><li><p>Synthetic Data &amp; Self-Instruction,</p></li><li><p>LLM-as-a-Judge,</p></li><li><p>Direct Preference Optimization,</p></li><li><p>Knowledge Distillation,</p></li><li><p>Temperature &amp; Sampling,</p></li><li><p>Gradient Descent &amp; Optimization,</p></li><li><p>Backpropagation,</p></li><li><p>Loss Functions,</p></li><li><p>Generalization,</p></li><li><p>Scaling Laws,</p></li><li><p>AI Alignment,</p></li><li><p>AI Safety &amp; Guardrails.</p></li></ol><p>For each, we&#8217;ll cover:</p><ul><li><p>What it means &amp; how it works,</p></li><li><p>A real-world analogy where useful,</p></li><li><p>A common mistake to avoid,</p></li><li><p>Why it matters.</p></li></ul><p>Let&#8217;s go!</p><h4 style="text-align: center;"><strong>&#167;</strong></h4><p></p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption"><span data-color="#3a7bf9" style="color: rgb(58, 123, 249);">Get my AI engineering playbook for FREE on newsletter signup:</span></p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><ul><li><p><em><a href="https://newsletter.systemdesign.one/p/advanced-ai-concepts-software-engineers/?action=share">Share this letter</a> &amp; I&#8217;ll send you some rewards for the referrals.</em></p></li></ul><h4 style="text-align: center;"><strong>&#167;</strong></h4><h2><strong>35. Mixture-of-Experts</strong></h2><blockquote><p>Mixture-of-Experts (<strong>MoE</strong><a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-2" href="#footnote-2" target="_self">2</a>) is a model design that uses a router to direct information to neural network components<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-3" href="#footnote-3" target="_self">3</a> called <em>experts</em>. Common sparse designs activate only some experts for each token.</p></blockquote><p>Let&#8217;s look inside a model as it processes a question:</p><p>Imagine one layer contains eight experts. A router selects two experts to process a token&#8217;s numerical representation. Then the layer combines their results &amp; passes the information onward.</p><p>The next token could use a different pair of experts.</p><p>The router learns how to assign work during training. Each expert performs part of the model&#8217;s calculations, while other components also contribute to the response.</p><p>(These experts work within the same model.)</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!WF-T!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7f3ef785-f04d-487e-8c3b-494bdd99d112_1448x1086.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!WF-T!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7f3ef785-f04d-487e-8c3b-494bdd99d112_1448x1086.png 424w, https://substackcdn.com/image/fetch/$s_!WF-T!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7f3ef785-f04d-487e-8c3b-494bdd99d112_1448x1086.png 848w, https://substackcdn.com/image/fetch/$s_!WF-T!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7f3ef785-f04d-487e-8c3b-494bdd99d112_1448x1086.png 1272w, https://substackcdn.com/image/fetch/$s_!WF-T!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7f3ef785-f04d-487e-8c3b-494bdd99d112_1448x1086.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!WF-T!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7f3ef785-f04d-487e-8c3b-494bdd99d112_1448x1086.png" width="1448" height="1086" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/7f3ef785-f04d-487e-8c3b-494bdd99d112_1448x1086.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1086,&quot;width&quot;:1448,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!WF-T!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7f3ef785-f04d-487e-8c3b-494bdd99d112_1448x1086.png 424w, https://substackcdn.com/image/fetch/$s_!WF-T!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7f3ef785-f04d-487e-8c3b-494bdd99d112_1448x1086.png 848w, https://substackcdn.com/image/fetch/$s_!WF-T!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7f3ef785-f04d-487e-8c3b-494bdd99d112_1448x1086.png 1272w, https://substackcdn.com/image/fetch/$s_!WF-T!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7f3ef785-f04d-487e-8c3b-494bdd99d112_1448x1086.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><strong>Analogy</strong></h3><p>Think of a workshop with several workers &amp; a coordinator.</p><p>The coordinator assigns each task to a few workers. Those workers contribute their results, and the work continues through the workshop.</p><p>Not every worker needs to handle every piece.</p><h3><strong>Common mistake</strong></h3><p>Assuming each expert has a fixed subject - mathematics, coding, or history.</p><p>The word &#8220;expert&#8221; describes a neural network component. Training shapes its behavior &amp; router&#8217;s choices, but those choices don&#8217;t necessarily follow recognizable subject categories.</p><h3><strong>Why it matters</strong></h3><p>Sparse MoE allows a model to have more parameters <em>without</em> using all its experts for every token. Only selected experts perform calculations, thus reducing computation while giving the model more capacity to learn patterns.</p><p>Yet all expert weights still need to be stored in memory/storage.</p><p>So using fewer experts per token reduces computation, but doesn&#8217;t reduce the model&#8217;s memory requirements.</p><h2><strong>36. LLM Serving and Inference Engines</strong></h2><blockquote><p>LLM serving makes a trained model available to receive requests &amp; return responses. An inference engine is the software that runs the model&#8217;s calculations.</p></blockquote><p>You submit <em>&#8220;Explain this error&#8221;</em> through a chatbot:</p><ol><li><p>The serving system accepts your request &amp; passes it to the software that runs the model.</p></li><li><p>Inference engine uses the model&#8217;s weights &amp; your input to generate tokens.</p></li><li><p>The serving system sends those tokens back to your screen.</p></li></ol><p>Your answer arrives through several coordinated steps.</p><p>Now imagine hundreds of users submitting questions at once...</p><p>The system schedules work within the available memory &amp; computing capacity. Some requests need longer responses than others. Requests that exceed available capacity must wait or get rejected.</p><p>The model needs software around it to serve real users.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!HuhG!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d81cba9-e78a-46bb-9097-1fff22ba6052_1448x1086.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!HuhG!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d81cba9-e78a-46bb-9097-1fff22ba6052_1448x1086.png 424w, https://substackcdn.com/image/fetch/$s_!HuhG!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d81cba9-e78a-46bb-9097-1fff22ba6052_1448x1086.png 848w, https://substackcdn.com/image/fetch/$s_!HuhG!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d81cba9-e78a-46bb-9097-1fff22ba6052_1448x1086.png 1272w, https://substackcdn.com/image/fetch/$s_!HuhG!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d81cba9-e78a-46bb-9097-1fff22ba6052_1448x1086.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!HuhG!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d81cba9-e78a-46bb-9097-1fff22ba6052_1448x1086.png" width="1448" height="1086" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/5d81cba9-e78a-46bb-9097-1fff22ba6052_1448x1086.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1086,&quot;width&quot;:1448,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!HuhG!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d81cba9-e78a-46bb-9097-1fff22ba6052_1448x1086.png 424w, https://substackcdn.com/image/fetch/$s_!HuhG!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d81cba9-e78a-46bb-9097-1fff22ba6052_1448x1086.png 848w, https://substackcdn.com/image/fetch/$s_!HuhG!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d81cba9-e78a-46bb-9097-1fff22ba6052_1448x1086.png 1272w, https://substackcdn.com/image/fetch/$s_!HuhG!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d81cba9-e78a-46bb-9097-1fff22ba6052_1448x1086.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><strong>Analogy</strong></h3><p>Think of a restaurant&#8217;s kitchen &amp; service team<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-4" href="#footnote-4" target="_self">4</a>:</p><p>The kitchen prepares the meals&#8230; The wider restaurant operation accepts orders, coordinates workload, and delivers food to customers.</p><p>Preparing a meal is one part of serving a restaurant full of people.</p><h3><strong>Common mistake</strong></h3><p>Testing one request &amp; assuming the system is ready for traffic.</p><p>A setup that responds quickly to one user could slow down when many requests arrive at once<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-5" href="#footnote-5" target="_self">5</a>. Test realistic request volumes &amp; response lengths, then check waiting times, failures &amp; memory use.</p><h3><strong>Why it matters</strong></h3><p>LLM serving helps applications handle requests from many users.</p><p>While inference engines run the model&#8217;s calculations to generate responses. Together, they affect response time, hardware efficiency, and how reliably an application serves users.</p><p>Choosing a suitable engine &amp; testing the complete serving system helps turn a trained model into a dependable service.</p><h2><strong>37. Synthetic Data and Self-Instruction</strong></h2><blockquote><p>Synthetic data is &#8220;artificially created&#8221; data used for purposes such as training/testing. Self-instruction<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-6" href="#footnote-6" target="_self">6</a> uses a language model to generate tasks &amp; corresponding responses for further training.</p></blockquote><p>Imagine you need more training examples for a model that classifies &#8220;customer support&#8221; messages.</p><p>You ask a language model to generate fictional messages about billing &amp; technical problems.</p><p>With <em>self-instruction</em>, the model also creates instructions &amp; expected answers for those messages.</p><p>Each example contains a task &amp; its correct label:</p><ul><li><p>Instruction: Classify this customer message.</p></li><li><p>Message: <em>&#8220;My card got charged twice.&#8221;</em></p></li><li><p>Expected answer: Billing.</p></li></ul><p>The system then checks generated examples &amp; removes incorrect, repetitive, and/or unsuitable ones.</p><p>The remaining examples could be used to <em>fine-tune</em> the model&#8230;</p><p>Also generating examples alone doesn&#8217;t change the model&#8217;s weights&#8230; A separate training step uses those examples to update the weights, thus helping the model learn to classify similar messages.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!HnO9!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e1c6e62-f2b4-4484-b907-d77928e559b3_1622x969.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!HnO9!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e1c6e62-f2b4-4484-b907-d77928e559b3_1622x969.png 424w, https://substackcdn.com/image/fetch/$s_!HnO9!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e1c6e62-f2b4-4484-b907-d77928e559b3_1622x969.png 848w, https://substackcdn.com/image/fetch/$s_!HnO9!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e1c6e62-f2b4-4484-b907-d77928e559b3_1622x969.png 1272w, https://substackcdn.com/image/fetch/$s_!HnO9!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e1c6e62-f2b4-4484-b907-d77928e559b3_1622x969.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!HnO9!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e1c6e62-f2b4-4484-b907-d77928e559b3_1622x969.png" width="1456" height="870" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/2e1c6e62-f2b4-4484-b907-d77928e559b3_1622x969.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:870,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!HnO9!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e1c6e62-f2b4-4484-b907-d77928e559b3_1622x969.png 424w, https://substackcdn.com/image/fetch/$s_!HnO9!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e1c6e62-f2b4-4484-b907-d77928e559b3_1622x969.png 848w, https://substackcdn.com/image/fetch/$s_!HnO9!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e1c6e62-f2b4-4484-b907-d77928e559b3_1622x969.png 1272w, https://substackcdn.com/image/fetch/$s_!HnO9!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e1c6e62-f2b4-4484-b907-d77928e559b3_1622x969.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><strong>Analogy</strong></h3><p>Think of preparing practice questions with an answer key:</p><p>You create exercises to cover different situations. Before anyone studies from them, you check whether the questions make sense &amp; if the answers are correct.</p><p>Faulty practice material teaches faulty lessons.</p><h3><strong>Common mistake</strong></h3><p>Treating a large generated dataset as a high-quality dataset.</p><p>Generated examples could contain wrong answers, repeated patterns, or unrealistic situations. Check their accuracy &amp; variety, then evaluate the trained model on separate examples that reflect real use.</p><h3><strong>Why it matters</strong></h3><ul><li><p>Synthetic data helps expand training material when suitable examples are scarce.</p></li><li><p>Self-instruction provides one way to create examples for instruction-based tasks.</p></li></ul><p>The value comes from useful, checked examples rather than volume alone.</p><h4 style="text-align: center;"><strong>&#167;</strong></h4><div class="captioned-button-wrap" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/p/advanced-ai-concepts-software-engineers?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;}" data-component-name="CaptionedButtonToDOM"><div class="preamble"><p class="cta-caption"><em><span data-color="#3a7bf9" style="color: rgb(58, 123, 249);">If something resonated with you today, share this letter. Because one idea, one action, can change everything.</span></em></p></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/p/advanced-ai-concepts-software-engineers?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://newsletter.systemdesign.one/p/advanced-ai-concepts-software-engineers?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p></div><h4 style="text-align: center;"><strong>&#167;</strong></h4><h2><strong>38. LLM-as-a-Judge</strong></h2><blockquote><p>LLM-as-a-Judge uses a language model to evaluate another model&#8217;s responses against predefined criteria, such as factual accuracy, relevance, or completeness, and assigns scores/rankings based on those criteria.</p></blockquote><p>Ask an assistant to shorten a customer&#8217;s complaint.</p><p>The customer wrote, <em>&#8220;My order arrived three days late, and the box was damaged.&#8221;</em> But the assistant produces <em>&#8220;The order arrived late.&#8221;</em> You give a judge model both versions &amp; ask it to check whether the summary includes both problems.</p><p>The judge could flag the missing detail:<em> damaged box.</em></p><p>Your instructions define what the judge checks. It could score one response, explain a problem, or compare two answers &amp; select the better one.</p><p>Clear criteria give the review a specific purpose.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!KmNs!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcacff04e-2678-4009-a078-86e76961998d_1730x909.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!KmNs!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcacff04e-2678-4009-a078-86e76961998d_1730x909.png 424w, https://substackcdn.com/image/fetch/$s_!KmNs!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcacff04e-2678-4009-a078-86e76961998d_1730x909.png 848w, https://substackcdn.com/image/fetch/$s_!KmNs!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcacff04e-2678-4009-a078-86e76961998d_1730x909.png 1272w, https://substackcdn.com/image/fetch/$s_!KmNs!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcacff04e-2678-4009-a078-86e76961998d_1730x909.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!KmNs!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcacff04e-2678-4009-a078-86e76961998d_1730x909.png" width="1456" height="765" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/cacff04e-2678-4009-a078-86e76961998d_1730x909.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:765,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!KmNs!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcacff04e-2678-4009-a078-86e76961998d_1730x909.png 424w, https://substackcdn.com/image/fetch/$s_!KmNs!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcacff04e-2678-4009-a078-86e76961998d_1730x909.png 848w, https://substackcdn.com/image/fetch/$s_!KmNs!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcacff04e-2678-4009-a078-86e76961998d_1730x909.png 1272w, https://substackcdn.com/image/fetch/$s_!KmNs!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcacff04e-2678-4009-a078-86e76961998d_1730x909.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><strong>Analogy</strong></h3><p>Think of a teacher checking a student&#8217;s work:</p><p>The teacher compares the answer to the assignment requirements. A judge model performs a similar review using the instructions and reference material you supply.</p><h3><strong>Common mistake</strong></h3><p>Treating the judge&#8217;s score as proof of correctness.</p><p>A judge could overlook an error or favor a longer answer without good reason. Compare some of its reviews with careful human assessments. When comparing 2 answers, swap their order to check whether its preference changes.</p><h3><strong>Why it matters</strong></h3><p>Reading every response by hand takes time.</p><p>LLM judges help teams review large collections of answers &amp; identify responses that need closer inspection.</p><p>Their usefulness depends on how reliably their assessments match qualities you need.</p><h2><strong>39. Direct Preference Optimization</strong></h2><blockquote><p>Direct Preference Optimization (<strong>DPO</strong>) &#8220;trains&#8221; a language model to favor preferred responses over less preferred alternatives using pairs of answers to the same request.</p></blockquote><p>Imagine you want an assistant to reply politely to customers.</p><p>For <em>&#8220;My order is late,&#8221;</em> one response says, <em>&#8220;Just wait.&#8221;</em> Another says, <em>&#8220;I&#8217;m sorry about the delay. Please send your order number.&#8221;</em> A reviewer marks the second response as the better choice.</p><p>The pair then becomes a training example.</p><p>DPO uses many such pairs to adjust the model&#8217;s weights toward preferred responses during training. It uses recorded choices directly, without first training a separate reward model to score answers.</p><p>Choosing a response records feedback; the training step changes the model.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!4OKh!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff0536bc8-f6ab-4894-bf6a-268edeb7c9cb_1729x910.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!4OKh!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff0536bc8-f6ab-4894-bf6a-268edeb7c9cb_1729x910.png 424w, https://substackcdn.com/image/fetch/$s_!4OKh!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff0536bc8-f6ab-4894-bf6a-268edeb7c9cb_1729x910.png 848w, https://substackcdn.com/image/fetch/$s_!4OKh!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff0536bc8-f6ab-4894-bf6a-268edeb7c9cb_1729x910.png 1272w, https://substackcdn.com/image/fetch/$s_!4OKh!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff0536bc8-f6ab-4894-bf6a-268edeb7c9cb_1729x910.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!4OKh!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff0536bc8-f6ab-4894-bf6a-268edeb7c9cb_1729x910.png" width="1456" height="766" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/f0536bc8-f6ab-4894-bf6a-268edeb7c9cb_1729x910.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:766,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!4OKh!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff0536bc8-f6ab-4894-bf6a-268edeb7c9cb_1729x910.png 424w, https://substackcdn.com/image/fetch/$s_!4OKh!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff0536bc8-f6ab-4894-bf6a-268edeb7c9cb_1729x910.png 848w, https://substackcdn.com/image/fetch/$s_!4OKh!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff0536bc8-f6ab-4894-bf6a-268edeb7c9cb_1729x910.png 1272w, https://substackcdn.com/image/fetch/$s_!4OKh!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff0536bc8-f6ab-4894-bf6a-268edeb7c9cb_1729x910.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><strong>Analogy</strong></h3><p>Think of helping someone write customer replies:</p><p>You show two drafts &amp; point out which one better follows the company&#8217;s standards. Repeated practice guides how the person writes future replies.</p><p>DPO guides a model through similar comparisons.</p><h3><strong>Common mistake</strong></h3><p>Choosing preferred answers based only on how pleasant they sound.</p><p>A polite reply could promise a refund the customer doesn&#8217;t qualify for. Review accuracy &amp; company policy alongside tone, so the training examples reward useful behavior.</p><h3><strong>Why it matters</strong></h3><p>DPO provides a simpler way to train from preferences than approaches which require a separate reward model and reinforcement learning.</p><p>The quality of those preferences shapes the behavior the model learns.</p><h2><strong>40. Knowledge Distillation</strong></h2><blockquote><p>Knowledge distillation <em>trains</em> a student model<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-7" href="#footnote-7" target="_self">7</a> to learn from a teacher model&#8217;s outputs. A common goal is to preserve useful abilities in a smaller model.</p></blockquote><p>Imagine a large model recognizes cats &amp; dogs well, but your application needs something smaller.</p><p>You show photos to the teacher model &amp; collect its predictions. The student processes the same photos, and the training algorithm compares its predictions to the teacher&#8217;s. Then it adjusts student&#8217;s weights to bring their predictions closer together.</p><p>The teacher&#8217;s outputs guide the student&#8217;s training.</p><p>Some methods also use the teacher&#8217;s confidence across possible answers. This gives the student more information than a single label such as &#8220;cat.&#8221; The student learns through its own weight adjustments.</p><p>After training, the student handles new photos independently.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!-4yy!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4bc9ad3b-1bd4-4e80-8b20-db7716ba9d0a_1448x1086.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!-4yy!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4bc9ad3b-1bd4-4e80-8b20-db7716ba9d0a_1448x1086.png 424w, https://substackcdn.com/image/fetch/$s_!-4yy!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4bc9ad3b-1bd4-4e80-8b20-db7716ba9d0a_1448x1086.png 848w, https://substackcdn.com/image/fetch/$s_!-4yy!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4bc9ad3b-1bd4-4e80-8b20-db7716ba9d0a_1448x1086.png 1272w, https://substackcdn.com/image/fetch/$s_!-4yy!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4bc9ad3b-1bd4-4e80-8b20-db7716ba9d0a_1448x1086.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!-4yy!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4bc9ad3b-1bd4-4e80-8b20-db7716ba9d0a_1448x1086.png" width="1448" height="1086" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/4bc9ad3b-1bd4-4e80-8b20-db7716ba9d0a_1448x1086.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1086,&quot;width&quot;:1448,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!-4yy!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4bc9ad3b-1bd4-4e80-8b20-db7716ba9d0a_1448x1086.png 424w, https://substackcdn.com/image/fetch/$s_!-4yy!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4bc9ad3b-1bd4-4e80-8b20-db7716ba9d0a_1448x1086.png 848w, https://substackcdn.com/image/fetch/$s_!-4yy!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4bc9ad3b-1bd4-4e80-8b20-db7716ba9d0a_1448x1086.png 1272w, https://substackcdn.com/image/fetch/$s_!-4yy!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4bc9ad3b-1bd4-4e80-8b20-db7716ba9d0a_1448x1086.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><strong>Analogy</strong></h3><p>Think of a teacher helping a student practice.</p><p>The teacher demonstrates answers, and the student learns from those examples. The student attempts new questions later without the teacher beside them.</p><p>Practice doesn&#8217;t guarantee the student matches the teacher on every question.</p><h3><strong>Common mistake</strong></h3><p>Assuming the student inherits every ability the teacher has.</p><p>A student trained on cat &amp; dog photos hasn&#8217;t necessarily learned to recognize other animals. It could also repeat the teacher&#8217;s errors. Test it on separate examples which reflect the job you need it to perform.</p><h3><strong>Why it matters</strong></h3><p>Distillation helps teams develop models which need less memory and computation for a chosen task. A smaller student could serve requests faster or run on more limited hardware.</p><p>Check whether those savings preserve the quality your application needs.</p><h2><strong>41. Temperature and Sampling</strong></h2><blockquote><p>Sampling is how an AI model chooses the next token during inference. Tokens the model considers likely have a higher chance of being chosen.</p><p>Temperature controls how predictable that choice is by adjusting the probabilities to make likely options more dominant or give less likely options more chance.</p></blockquote><p>Ask an assistant to suggest a name for your bakery:</p><p>The model builds each suggestion one token at a time. At each step, it assigns probabilities to possible next tokens. Sampling chooses among those options according to their chances.</p><p>Different choices lead to different names.</p><p>Lower temperature favors the leading options more strongly, usually producing less variation across attempts. Higher temperature gives less likely options more chance, usually producing more varied suggestions.</p><p>Yet some results could be unusual/awkward&#8230; More variation doesn&#8217;t guarantee a better name.</p><p>These settings affect token selection <em>without</em> changing the model&#8217;s weights.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!TBXr!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F207edc93-1e2b-45f4-862a-cee4d39842ad_1729x910.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!TBXr!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F207edc93-1e2b-45f4-862a-cee4d39842ad_1729x910.png 424w, https://substackcdn.com/image/fetch/$s_!TBXr!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F207edc93-1e2b-45f4-862a-cee4d39842ad_1729x910.png 848w, https://substackcdn.com/image/fetch/$s_!TBXr!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F207edc93-1e2b-45f4-862a-cee4d39842ad_1729x910.png 1272w, https://substackcdn.com/image/fetch/$s_!TBXr!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F207edc93-1e2b-45f4-862a-cee4d39842ad_1729x910.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!TBXr!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F207edc93-1e2b-45f4-862a-cee4d39842ad_1729x910.png" width="1456" height="766" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/207edc93-1e2b-45f4-862a-cee4d39842ad_1729x910.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:766,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!TBXr!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F207edc93-1e2b-45f4-862a-cee4d39842ad_1729x910.png 424w, https://substackcdn.com/image/fetch/$s_!TBXr!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F207edc93-1e2b-45f4-862a-cee4d39842ad_1729x910.png 848w, https://substackcdn.com/image/fetch/$s_!TBXr!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F207edc93-1e2b-45f4-862a-cee4d39842ad_1729x910.png 1272w, https://substackcdn.com/image/fetch/$s_!TBXr!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F207edc93-1e2b-45f4-862a-cee4d39842ad_1729x910.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><strong>Analogy</strong></h3><p>Think of drawing a slip from a bowl.</p><p>Several slips carry the same word, while others carry different words. A word with more slips has a higher chance of being selected.</p><p>Temperature changes the balance of slips; sampling makes the draw.</p><h3><strong>Common mistake</strong></h3><p>Treating temperature as a quality control.</p><p>A higher setting doesn&#8217;t guarantee originality, and a lower setting doesn&#8217;t guarantee accuracy. Review results for readability &amp; relevance, and check factual answers against reliable sources.</p><h3><strong>Why it matters</strong></h3><p>Different tasks need different amounts of variation&#8230;</p><p>Name suggestions benefit from a range of options, while extracting details from a receipt benefits from consistent results.</p><p>Test the settings against the task rather than using one value for everything.</p><h4 style="text-align: center;"><strong>&#167;</strong></h4><div class="callout-block" data-callout="true"><p style="text-align: center;"><em><strong><span data-color="#3a7bf9" style="color: rgb(58, 123, 249);">Reminder: this is a teaser of the subscriber-only newsletter, exclusive to my golden members.</span></strong></em></p><p>When you upgrade, you&#8217;ll get:</p><ul><li><p><strong>High-level architecture of real-world systems.</strong></p></li><li><p>Deep dive into how popular real-world systems work.</p></li><li><p><strong>How real-world systems handle scale, reliability &amp; performance.</strong></p></li></ul><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/subscribe&quot;,&quot;text&quot;:&quot;Unlock Full Access&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://newsletter.systemdesign.one/subscribe"><span>Unlock Full Access</span></a></p><p>(If this newsletter has helped you become a better software engineer, consider subscribing to support my work.)</p></div><h4 style="text-align: center;"><strong>&#167;</strong></h4><h2><strong>42. Gradient Descent and Optimization</strong></h2>
      <p>
          <a href="https://newsletter.systemdesign.one/p/advanced-ai-concepts-software-engineers">
              Read more
          </a>
      </p>
   ]]></content:encoded></item><item><title><![CDATA[64 AI Engineering Concepts, Clearly Explained]]></title><description><![CDATA[#193: Part 2 - RLHF, Pretraining, Vector Search, and 14 others.]]></description><link>https://newsletter.systemdesign.one/p/advanced-ai-concepts-explained</link><guid isPermaLink="false">https://newsletter.systemdesign.one/p/advanced-ai-concepts-explained</guid><dc:creator><![CDATA[Neo Kim]]></dc:creator><pubDate>Tue, 20 Oct 2026 13:27:50 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/7fb0bd93-e36d-4683-8991-10bf77823357_1280x720.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>A reader asks:</p><p><em>&#8220;Hi Neo, do I really need to understand all these AI concepts? RLHF, vector search, quantization, diffusion models, MCP, hybrid retrieval... It feels like every time I learn one concept, five more appear. Lol&#8221;</em></p><p>All I will say is this:</p><p>You don&#8217;t need to master every AI concept to start building AI apps.</p><p>At least, not all at once.</p><p>But understanding how these concepts work makes a BIG difference.</p><p>Onward.</p><h4 style="text-align: center;"><strong>&#167;</strong></h4><div class="callout-block" data-callout="true"><h2><a href="https://agentfield.ai/github/codeaf/?utm_source=sys_design&amp;utm_medium=newsletter&amp;utm_campaign=sys_design-261020&amp;utm_id=sys_design-261020-codeaf&amp;utm_content=codeaf">Same model, 2x the fixes: the coding harness is part of the system</a></h2><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://agentfield.ai/github/codeaf/?utm_source=sys_design&amp;utm_medium=newsletter&amp;utm_campaign=sys_design-261020&amp;utm_id=sys_design-261020-codeaf&amp;utm_content=codeaf" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!fMh4!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fedf82a71-3ec7-457d-b65a-fee94213abbd_1600x840.png 424w, https://substackcdn.com/image/fetch/$s_!fMh4!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fedf82a71-3ec7-457d-b65a-fee94213abbd_1600x840.png 848w, https://substackcdn.com/image/fetch/$s_!fMh4!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fedf82a71-3ec7-457d-b65a-fee94213abbd_1600x840.png 1272w, https://substackcdn.com/image/fetch/$s_!fMh4!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fedf82a71-3ec7-457d-b65a-fee94213abbd_1600x840.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!fMh4!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fedf82a71-3ec7-457d-b65a-fee94213abbd_1600x840.png" width="1456" height="764" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/edf82a71-3ec7-457d-b65a-fee94213abbd_1600x840.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:764,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:424976,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:&quot;https://agentfield.ai/github/codeaf/?utm_source=sys_design&amp;utm_medium=newsletter&amp;utm_campaign=sys_design-261020&amp;utm_id=sys_design-261020-codeaf&amp;utm_content=codeaf&quot;,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://newsletter.systemdesign.one/i/179236490?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fedf82a71-3ec7-457d-b65a-fee94213abbd_1600x840.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!fMh4!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fedf82a71-3ec7-457d-b65a-fee94213abbd_1600x840.png 424w, https://substackcdn.com/image/fetch/$s_!fMh4!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fedf82a71-3ec7-457d-b65a-fee94213abbd_1600x840.png 848w, https://substackcdn.com/image/fetch/$s_!fMh4!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fedf82a71-3ec7-457d-b65a-fee94213abbd_1600x840.png 1272w, https://substackcdn.com/image/fetch/$s_!fMh4!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fedf82a71-3ec7-457d-b65a-fee94213abbd_1600x840.png 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Most teams choose a coding agent by its model. AgentField tested the other half: ten coding harnesses on one open model, run on the same 113 real GitHub issues from the DeepSWE benchmark. Results ran from 3 solved issues to 62. The system around the model decides as much as the model does.</p><p><strong><a href="https://agentfield.ai/github/codeaf/?utm_source=sys_design&amp;utm_medium=newsletter&amp;utm_campaign=sys_design-261020&amp;utm_id=sys_design-261020-codeaf&amp;utm_content=codeaf">CodeAF</a></strong>, a new open-source coding harness by AgentField AI, finished #1 of the ten (method and data are public on GitHub):</p><ul><li><p>2x OpenCode&#8217;s solves and nearly 4x Claude Code&#8217;s, on the same model</p></li><li><p>4.8x less per solved issue than OpenCode, 3.4x less than Claude Code</p></li><li><p>Built for open models like DeepSeek, Qwen, GLM and Kimi, and it works with closed models too</p></li><li><p>One small Go binary, open source</p></li></ul><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://agentfield.ai/github/codeaf/?utm_source=sys_design&amp;utm_medium=newsletter&amp;utm_campaign=sys_design-261020&amp;utm_id=sys_design-261020-codeaf&amp;utm_content=codeaf&quot;,&quot;text&quot;:&quot;Try CodeAF on GitHub&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://agentfield.ai/github/codeaf/?utm_source=sys_design&amp;utm_medium=newsletter&amp;utm_campaign=sys_design-261020&amp;utm_id=sys_design-261020-codeaf&amp;utm_content=codeaf"><span>Try CodeAF on GitHub</span></a></p><p>(Thanks to <a href="https://agentfield.ai/github/codeaf/?utm_source=sys_design&amp;utm_medium=newsletter&amp;utm_campaign=sys_design-261020&amp;utm_id=sys_design-261020-codeaf&amp;utm_content=codeaf">AgentField</a> for partnering on this newsletter.)</p></div><h4 style="text-align: center;"><strong>&#167;</strong></h4><p></p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption"><span data-color="#3a7bf9" style="color: rgb(58, 123, 249);">Subscribe now and be the first to know about my letters:</span></p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><ul><li><p><em><a href="https://newsletter.systemdesign.one/p/advanced-ai-concepts-explained/?action=share">Share this letter</a> &amp; I&#8217;ll send you some rewards for the referrals.</em></p></li></ul><h4 style="text-align: center;"><strong>&#167;</strong></h4><p>This is part 2 of a four-part<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-1" href="#footnote-1" target="_self">1</a> newsletter series:</p><ol start="18"><li><p>Reinforcement Learning from Human Feedback,</p></li><li><p>Pretraining,</p></li><li><p>Vector Search and Approximate Nearest Neighbors,</p></li><li><p>In-Context Learning,</p></li><li><p>Hallucination, Factuality and Grounding,</p></li><li><p>Quantization,</p></li><li><p>Context Windows and Long-Context Modeling,</p></li><li><p>Structured Outputs and Constrained Decoding,</p></li><li><p>Model Context Protocol,</p></li><li><p>Parameter-Efficient Fine-Tuning,</p></li><li><p>Diffusion Models,</p></li><li><p>Autoregressive Generation and Next-Token Prediction,</p></li><li><p>Self-Supervised Learning,</p></li><li><p>Inference,</p></li><li><p>Retrieval Reranking,</p></li><li><p>Hybrid Retrieval,</p></li><li><p>Chunking and Document Segmentation.</p></li></ol><p>( &#8230;and much more in Parts 3 &amp; 4!)</p><p>For each, I&#8217;ll share:</p><ul><li><p>What it means &amp; how it works,</p></li><li><p>A real-world analogy where useful,</p></li><li><p>A common mistake to avoid,</p></li><li><p>Why it matters.</p></li></ul><p>Let&#8217;s dive in!</p><h4 style="text-align: center;"><strong>&#167;</strong></h4><h2><strong>18. Reinforcement Learning from Human Feedback</strong></h2><blockquote><p>Reinforcement Learning from Human Feedback (<strong>RLHF</strong>) uses human feedback to guide further model training toward preferred responses.</p></blockquote><p>Give reviewers two answers to the same question.</p><p>The reviewers compare qualities such as usefulness &amp; how well each answer follows instructions. Those comparisons train a reward model<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-2" href="#footnote-2" target="_self">2</a> to score responses. The language model then receives further training to produce responses with higher scores.</p><p>Human preferences help shape what the training process rewards.</p><p>This improves behavior according to the chosen criteria, but still needs careful evaluation.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!A6JM!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F26cc0758-522f-4c54-bef8-5c5b3fe73b23_1448x1086.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!A6JM!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F26cc0758-522f-4c54-bef8-5c5b3fe73b23_1448x1086.png 424w, https://substackcdn.com/image/fetch/$s_!A6JM!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F26cc0758-522f-4c54-bef8-5c5b3fe73b23_1448x1086.png 848w, https://substackcdn.com/image/fetch/$s_!A6JM!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F26cc0758-522f-4c54-bef8-5c5b3fe73b23_1448x1086.png 1272w, https://substackcdn.com/image/fetch/$s_!A6JM!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F26cc0758-522f-4c54-bef8-5c5b3fe73b23_1448x1086.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!A6JM!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F26cc0758-522f-4c54-bef8-5c5b3fe73b23_1448x1086.png" width="1448" height="1086" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/26cc0758-522f-4c54-bef8-5c5b3fe73b23_1448x1086.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1086,&quot;width&quot;:1448,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!A6JM!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F26cc0758-522f-4c54-bef8-5c5b3fe73b23_1448x1086.png 424w, https://substackcdn.com/image/fetch/$s_!A6JM!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F26cc0758-522f-4c54-bef8-5c5b3fe73b23_1448x1086.png 848w, https://substackcdn.com/image/fetch/$s_!A6JM!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F26cc0758-522f-4c54-bef8-5c5b3fe73b23_1448x1086.png 1272w, https://substackcdn.com/image/fetch/$s_!A6JM!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F26cc0758-522f-4c54-bef8-5c5b3fe73b23_1448x1086.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><strong>Analogy</strong></h3><p>Think of a coach reviewing practice performances:</p><p>Feedback guides the next attempt. The improvement depends partly on whether the coach rewards the qualities the task actually requires.</p><h3><strong>Common mistake</strong></h3><p>Confusing a preferred answer with a &#8220;verified&#8221; answer.</p><p>Reviewers could favor a polished response containing a subtle factual error. Check accuracy separately from tone &amp; helpfulness.</p><h3><strong>Why it matters</strong></h3><p>RLHF helps shape a broadly trained model into a more useful assistant. The feedback criteria influence which behaviors the model learns to favor.</p><h2><strong>19. Pretraining</strong></h2><blockquote><p>Pretraining is an initial training stage in which a model learns broad patterns from a large dataset before more specialized training.</p></blockquote><p>Before a model follows your request, it needs a foundation.</p><p>For a typical generative language model, pretraining repeatedly tests predictions of the next token<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-3" href="#footnote-3" target="_self">3</a>. Prediction errors guide adjustments to the model&#8217;s weights. Across many examples, these adjustments help the model learn patterns in the text.</p><p>The text itself supplies the next-token answers.</p><p>Other models use different tasks, such as predicting missing pieces of an input.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!-795!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F60dad202-4eb7-4186-87b8-be4450a6ceae_1448x1086.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!-795!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F60dad202-4eb7-4186-87b8-be4450a6ceae_1448x1086.png 424w, https://substackcdn.com/image/fetch/$s_!-795!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F60dad202-4eb7-4186-87b8-be4450a6ceae_1448x1086.png 848w, https://substackcdn.com/image/fetch/$s_!-795!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F60dad202-4eb7-4186-87b8-be4450a6ceae_1448x1086.png 1272w, https://substackcdn.com/image/fetch/$s_!-795!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F60dad202-4eb7-4186-87b8-be4450a6ceae_1448x1086.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!-795!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F60dad202-4eb7-4186-87b8-be4450a6ceae_1448x1086.png" width="1448" height="1086" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/60dad202-4eb7-4186-87b8-be4450a6ceae_1448x1086.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1086,&quot;width&quot;:1448,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!-795!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F60dad202-4eb7-4186-87b8-be4450a6ceae_1448x1086.png 424w, https://substackcdn.com/image/fetch/$s_!-795!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F60dad202-4eb7-4186-87b8-be4450a6ceae_1448x1086.png 848w, https://substackcdn.com/image/fetch/$s_!-795!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F60dad202-4eb7-4186-87b8-be4450a6ceae_1448x1086.png 1272w, https://substackcdn.com/image/fetch/$s_!-795!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F60dad202-4eb7-4186-87b8-be4450a6ceae_1448x1086.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><strong>Analogy</strong></h3><p>Think of a broad education before a specialized course:</p><p>The foundation covers many subjects &amp; patterns. Later instruction directs those abilities toward a particular role.</p><h3><strong>Common mistake</strong></h3><p>Assuming pretraining produces a ready-to-use assistant.</p><p>A model trained to predict text doesn&#8217;t automatically respond in the style users expect. Further training &amp; application design help shape how it handles requests.</p><h3><strong>Why it matters</strong></h3><p>Pretraining provides capabilities reused across many tasks.</p><p>The data influences what the model learns well and where its knowledge has gaps.</p><h2><strong>20. Vector Search and Approximate Nearest Neighbors</strong></h2><blockquote><p>Vector search finds items with similar numerical representations. Approximate Nearest Neighbors (<strong>ANN</strong>) methods speed up search while accepting the possibility of <em>missing</em> some closest matches.</p></blockquote><p>Imagine searching millions of support documents:</p><p>The system converts your question into an embedding<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-4" href="#footnote-4" target="_self">4</a> &amp; compares it with document embeddings with <em>vector search</em>. An exact<em> search</em> finds the closest matches under a chosen comparison rule. An <em>approximate search</em> uses an index, an organized search structure, to reduce the work.</p><p>The shortcut helps results arrive faster.</p><p>How much accuracy it sacrifices depends on the method &amp; its settings&#8230;</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!BArz!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4e61a115-7a33-4789-93a4-93456c8ba917_1448x1086.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!BArz!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4e61a115-7a33-4789-93a4-93456c8ba917_1448x1086.png 424w, https://substackcdn.com/image/fetch/$s_!BArz!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4e61a115-7a33-4789-93a4-93456c8ba917_1448x1086.png 848w, https://substackcdn.com/image/fetch/$s_!BArz!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4e61a115-7a33-4789-93a4-93456c8ba917_1448x1086.png 1272w, https://substackcdn.com/image/fetch/$s_!BArz!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4e61a115-7a33-4789-93a4-93456c8ba917_1448x1086.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!BArz!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4e61a115-7a33-4789-93a4-93456c8ba917_1448x1086.png" width="1448" height="1086" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/4e61a115-7a33-4789-93a4-93456c8ba917_1448x1086.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1086,&quot;width&quot;:1448,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!BArz!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4e61a115-7a33-4789-93a4-93456c8ba917_1448x1086.png 424w, https://substackcdn.com/image/fetch/$s_!BArz!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4e61a115-7a33-4789-93a4-93456c8ba917_1448x1086.png 848w, https://substackcdn.com/image/fetch/$s_!BArz!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4e61a115-7a33-4789-93a4-93456c8ba917_1448x1086.png 1272w, https://substackcdn.com/image/fetch/$s_!BArz!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4e61a115-7a33-4789-93a4-93456c8ba917_1448x1086.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><strong>Analogy</strong></h3><p>Think of a librarian directing you to likely shelves&#8230;</p><p>You search a smaller area instead of checking every book. The shortcut saves time, but you could miss a useful book elsewhere.</p><h3><strong>Common mistake</strong></h3><p>Choosing the fastest setup without checking its results.</p><p>A quick search is unhelpful if it repeatedly misses the relevant document. Compare retrieval quality, response time &amp; memory use.</p><h3><strong>Why it matters</strong></h3><ul><li><p>Vector Search: Best for <em>smaller</em> datasets requiring the true nearest matches.</p></li><li><p>Approximate Nearest Neighbors: Best for <em>large</em> datasets where search speed &amp; scalability matter more than finding the exact nearest match.</p></li></ul><h4 style="text-align: center;"><strong>&#167;</strong></h4><div class="captioned-button-wrap" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/p/advanced-ai-concepts-explained?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;}" data-component-name="CaptionedButtonToDOM"><div class="preamble"><p class="cta-caption"><em><span data-color="#3a7bf9" style="color: rgb(58, 123, 249);">If something resonated with you today, share this letter. Because one idea, one action, can change everything.</span></em></p></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/p/advanced-ai-concepts-explained?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://newsletter.systemdesign.one/p/advanced-ai-concepts-explained?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p></div><h4 style="text-align: center;"><strong>&#167;</strong></h4><h2><strong>21. In-Context Learning</strong></h2><blockquote><p>In-context learning is a model&#8217;s ability to use instructions/examples in its current context without updating its weights.</p></blockquote><p>Show the model a few examples before asking it to label a new message:</p><div class="callout-block" data-callout="true"><p>&#8220;My card was charged twice&#8221; &#8594; Billing</p><p>&#8220;The page crashes when I sign in&#8221; &#8594; Technical</p></div><p>The examples demonstrate the categories &amp; response format. The model then uses the pattern to label the next message. No separate training run occurs!</p><p>The examples <em>guide</em> the response through the context.</p><p>(Remove them from a later request, and you shouldn&#8217;t assume the model still has access to them.)</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!h2Ta!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb63d4df7-c766-4a50-86ba-03c7a6f98de6_1448x1086.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!h2Ta!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb63d4df7-c766-4a50-86ba-03c7a6f98de6_1448x1086.png 424w, https://substackcdn.com/image/fetch/$s_!h2Ta!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb63d4df7-c766-4a50-86ba-03c7a6f98de6_1448x1086.png 848w, https://substackcdn.com/image/fetch/$s_!h2Ta!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb63d4df7-c766-4a50-86ba-03c7a6f98de6_1448x1086.png 1272w, https://substackcdn.com/image/fetch/$s_!h2Ta!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb63d4df7-c766-4a50-86ba-03c7a6f98de6_1448x1086.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!h2Ta!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb63d4df7-c766-4a50-86ba-03c7a6f98de6_1448x1086.png" width="1448" height="1086" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/b63d4df7-c766-4a50-86ba-03c7a6f98de6_1448x1086.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1086,&quot;width&quot;:1448,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!h2Ta!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb63d4df7-c766-4a50-86ba-03c7a6f98de6_1448x1086.png 424w, https://substackcdn.com/image/fetch/$s_!h2Ta!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb63d4df7-c766-4a50-86ba-03c7a6f98de6_1448x1086.png 848w, https://substackcdn.com/image/fetch/$s_!h2Ta!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb63d4df7-c766-4a50-86ba-03c7a6f98de6_1448x1086.png 1272w, https://substackcdn.com/image/fetch/$s_!h2Ta!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb63d4df7-c766-4a50-86ba-03c7a6f98de6_1448x1086.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><strong>Analogy</strong></h3><p>Think of filling out a form after examining a completed sample:</p><p>The sample shows how the fields work. You apply the pattern immediately without taking a new course.</p><h3><strong>Common mistake</strong></h3><p>Providing examples that cover only obvious cases.</p><p>If every billing example contains &#8220;invoice,&#8221; the model could rely too heavily on a specific word. Include varied wording &amp; difficult cases, then test on separate examples.</p><h3><strong>Why it matters</strong></h3><p>In-context learning lets you demonstrate a task quickly. It helps you test formats/categories before investing in fine-tuning.</p><h2><strong>22. Hallucination, Factuality, and Grounded Answers</strong></h2><blockquote><p>A hallucination is generated content presented as &#8220;true&#8221; despite being false/unsupported. Factuality concerns correctness; grounding connects an answer to evidence.</p></blockquote><p>An assistant gives you a confident answer &amp; 3 citations.</p><p>One source doesn&#8217;t exist. Another exists without supporting the claim. The answer looks well researched, but its appearance tells you little about its accuracy.</p><p>The evidence needs inspection too&#8230;</p><p>A <em>grounded</em> answer uses supplied or retrieved material. It could still repeat an error from an outdated source, so the material's quality matters.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!pdpF!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc516c5b9-59e2-4f95-9b72-7454dc88e100_1448x1086.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!pdpF!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc516c5b9-59e2-4f95-9b72-7454dc88e100_1448x1086.png 424w, https://substackcdn.com/image/fetch/$s_!pdpF!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc516c5b9-59e2-4f95-9b72-7454dc88e100_1448x1086.png 848w, https://substackcdn.com/image/fetch/$s_!pdpF!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc516c5b9-59e2-4f95-9b72-7454dc88e100_1448x1086.png 1272w, https://substackcdn.com/image/fetch/$s_!pdpF!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc516c5b9-59e2-4f95-9b72-7454dc88e100_1448x1086.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!pdpF!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc516c5b9-59e2-4f95-9b72-7454dc88e100_1448x1086.png" width="1448" height="1086" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/c516c5b9-59e2-4f95-9b72-7454dc88e100_1448x1086.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1086,&quot;width&quot;:1448,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!pdpF!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc516c5b9-59e2-4f95-9b72-7454dc88e100_1448x1086.png 424w, https://substackcdn.com/image/fetch/$s_!pdpF!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc516c5b9-59e2-4f95-9b72-7454dc88e100_1448x1086.png 848w, https://substackcdn.com/image/fetch/$s_!pdpF!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc516c5b9-59e2-4f95-9b72-7454dc88e100_1448x1086.png 1272w, https://substackcdn.com/image/fetch/$s_!pdpF!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc516c5b9-59e2-4f95-9b72-7454dc88e100_1448x1086.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><strong>Analogy</strong></h3><p>Think of a report with footnotes.</p><p>The footnotes help you trace the claims. You still need to check whether the sources exist &amp; support what the report says.</p><h3><strong>Common mistake</strong></h3><p>Treating a citation as a completed fact-check.</p><p>Check the source &amp; compare its content with the answer. Give the system a way to report insufficient evidence instead of filling gaps with guesses.</p><h3><strong>Why it matters</strong></h3><p>These distinctions help you find the source of an error. The application could need better documents, better retrieval, and/or stricter checks on the final answer.</p><h2><strong>23. Quantization</strong></h2><blockquote><p>Quantization stores model values using fewer bits, thus reducing memory needs while preserving less numerical details.</p></blockquote><p>A model is too large for your machine's memory.</p><p>Its weights are numbers, and storing those numbers takes space. Quantization converts selected values into <em>smaller</em> numerical representations. The model needs less memory, though the conversion introduces <em>approximations</em>.</p><p>You trade some precision for a smaller memory footprint.</p><p>The effect on output quality depends on the model &amp; the quantization method.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!ZvWk!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F40e57925-4633-49db-974a-ad5bf0700470_1448x1086.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!ZvWk!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F40e57925-4633-49db-974a-ad5bf0700470_1448x1086.png 424w, https://substackcdn.com/image/fetch/$s_!ZvWk!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F40e57925-4633-49db-974a-ad5bf0700470_1448x1086.png 848w, https://substackcdn.com/image/fetch/$s_!ZvWk!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F40e57925-4633-49db-974a-ad5bf0700470_1448x1086.png 1272w, https://substackcdn.com/image/fetch/$s_!ZvWk!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F40e57925-4633-49db-974a-ad5bf0700470_1448x1086.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!ZvWk!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F40e57925-4633-49db-974a-ad5bf0700470_1448x1086.png" width="1448" height="1086" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/40e57925-4633-49db-974a-ad5bf0700470_1448x1086.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1086,&quot;width&quot;:1448,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!ZvWk!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F40e57925-4633-49db-974a-ad5bf0700470_1448x1086.png 424w, https://substackcdn.com/image/fetch/$s_!ZvWk!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F40e57925-4633-49db-974a-ad5bf0700470_1448x1086.png 848w, https://substackcdn.com/image/fetch/$s_!ZvWk!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F40e57925-4633-49db-974a-ad5bf0700470_1448x1086.png 1272w, https://substackcdn.com/image/fetch/$s_!ZvWk!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F40e57925-4633-49db-974a-ad5bf0700470_1448x1086.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><strong>Analogy</strong></h3><p>Think of recording measurements to the nearest centimeter instead of millimeters.</p><p>You preserve less detail. Whether the difference matters depends on what you need the measurements for.</p><h3><strong>Common mistake</strong></h3><p>Assuming a smaller model representation always runs faster with identical results.</p><p>Speed depends on the hardware &amp; software used to run it. Test both quality and response time before choosing a quantized version.</p><h3><strong>Why it matters</strong></h3><p>Quantization helps larger models fit within smaller memory budgets. It creates more deployment options when the quality tradeoff is acceptable.</p><h2><strong>24. Context Windows and Long-Context Models</strong></h2><blockquote><p>A context window is the amount of information a model handles within a single context (measured in tokens). Long-context models support larger amounts of material.</p></blockquote><p>Upload a long manual &amp; ask about a detail halfway through it.</p><p>Your instructions, conversation history, document text, and tool results all take up space. In typical text-generation systems, the response also needs room within the overall limit. The application decides what to include &amp; how much space to reserve for the answer.</p><p>A larger window holds more material.</p><p>Yet it doesn&#8217;t guarantee the model uses every detail correctly.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!Xsjn!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe2c95e33-a727-4913-921c-9725b3c907f0_1448x1086.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Xsjn!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe2c95e33-a727-4913-921c-9725b3c907f0_1448x1086.png 424w, https://substackcdn.com/image/fetch/$s_!Xsjn!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe2c95e33-a727-4913-921c-9725b3c907f0_1448x1086.png 848w, https://substackcdn.com/image/fetch/$s_!Xsjn!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe2c95e33-a727-4913-921c-9725b3c907f0_1448x1086.png 1272w, https://substackcdn.com/image/fetch/$s_!Xsjn!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe2c95e33-a727-4913-921c-9725b3c907f0_1448x1086.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!Xsjn!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe2c95e33-a727-4913-921c-9725b3c907f0_1448x1086.png" width="1448" height="1086" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/e2c95e33-a727-4913-921c-9725b3c907f0_1448x1086.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1086,&quot;width&quot;:1448,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!Xsjn!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe2c95e33-a727-4913-921c-9725b3c907f0_1448x1086.png 424w, https://substackcdn.com/image/fetch/$s_!Xsjn!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe2c95e33-a727-4913-921c-9725b3c907f0_1448x1086.png 848w, https://substackcdn.com/image/fetch/$s_!Xsjn!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe2c95e33-a727-4913-921c-9725b3c907f0_1448x1086.png 1272w, https://substackcdn.com/image/fetch/$s_!Xsjn!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe2c95e33-a727-4913-921c-9725b3c907f0_1448x1086.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><strong>Analogy</strong></h3><p>Think of a desk covered with reference pages.</p><p>A larger desk holds more pages. It doesn&#8217;t guarantee you notice every detail or connect the right passages.</p><h3><strong>Common mistake</strong></h3><p>Assuming a document that fits is a document the model uses reliably.</p><p>Test questions whose answers appear in different parts of the document. Remove irrelevant material or retrieve targeted passages when full-document answers become unreliable.</p><h3><strong>Why it matters</strong></h3><p>Context limits affect document analysis, code review &amp; long conversations. Plus the amount of material processed influences cost &amp; response time.</p><h2><strong>25. Structured Outputs and Constrained Decoding</strong></h2><blockquote><p>Structured outputs follow a specified format. Constrained decoding<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-5" href="#footnote-5" target="_self">5</a> restricts what model generates, so the output follows supported format rules.</p></blockquote><p>Your application needs 3 fields from a support ticket.</p><p>A paragraph is inconvenient for code to process. A schema, or set of format rules, defines the field names, value types, and permitted choices. <em>Constrained decoding</em> limits the model&#8217;s next-token options to keep the response within those rules.</p><p>This makes the result easier for software to process.</p><p>For example, a <em>structured output</em> could follow a JSON schema like this</p><div class="highlighted_code_block" data-attrs="{&quot;language&quot;:&quot;json&quot;,&quot;nodeId&quot;:&quot;c8c97a27-4f26-49d9-b8b0-3c7015c63156&quot;}" data-component-name="HighlightedCodeBlockToDOM"><pre class="shiki"><code class="language-json">{
 &#8220;category&#8221;: &#8220;billing&#8221;,
 &#8220;priority&#8221;: &#8220;high&#8221;,
 &#8220;order_id&#8221;: null
}</code></pre></div><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!6j9m!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62c83733-4b36-46fd-9f8f-ed3fd311ba0a_1448x1086.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!6j9m!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62c83733-4b36-46fd-9f8f-ed3fd311ba0a_1448x1086.png 424w, https://substackcdn.com/image/fetch/$s_!6j9m!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62c83733-4b36-46fd-9f8f-ed3fd311ba0a_1448x1086.png 848w, https://substackcdn.com/image/fetch/$s_!6j9m!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62c83733-4b36-46fd-9f8f-ed3fd311ba0a_1448x1086.png 1272w, https://substackcdn.com/image/fetch/$s_!6j9m!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62c83733-4b36-46fd-9f8f-ed3fd311ba0a_1448x1086.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!6j9m!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62c83733-4b36-46fd-9f8f-ed3fd311ba0a_1448x1086.png" width="1448" height="1086" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/62c83733-4b36-46fd-9f8f-ed3fd311ba0a_1448x1086.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1086,&quot;width&quot;:1448,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!6j9m!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62c83733-4b36-46fd-9f8f-ed3fd311ba0a_1448x1086.png 424w, https://substackcdn.com/image/fetch/$s_!6j9m!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62c83733-4b36-46fd-9f8f-ed3fd311ba0a_1448x1086.png 848w, https://substackcdn.com/image/fetch/$s_!6j9m!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62c83733-4b36-46fd-9f8f-ed3fd311ba0a_1448x1086.png 1272w, https://substackcdn.com/image/fetch/$s_!6j9m!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F62c83733-4b36-46fd-9f8f-ed3fd311ba0a_1448x1086.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><strong>Analogy</strong></h3><p>Think of a form with fixed fields &amp; dropdown menus.</p><p>The form controls where information goes &amp; which options are allowed. It doesn&#8217;t establish whether the selected option is correct.</p><h3><strong>Common mistake</strong></h3><p>Confusing correct format with correct <em>information</em>.</p><p>A response could follow every format rule and still misclassify a ticket. Check the values and handle refusals or incomplete responses separately.</p><h3><strong>Why it matters</strong></h3><p>Structured outputs reduce work needed to turn model responses into usable application data. They solve a formatting problem, while factual checks remain necessary.</p><h2><strong>26. Model Context Protocol</strong></h2><blockquote><p>Model Context Protocol (<strong>MCP</strong>) is an open standard for connecting AI apps to external tools &amp; information sources.</p></blockquote><p>Your assistant needs access to a document store &amp; a ticket system.</p><p>An MCP server exposes capabilities, such as a document search or ticket lookup. The AI application connects through an MCP client, which discovers those capabilities and exchanges requests &amp; results. The shared protocol gives both sides a common way to communicate.</p><p>The connection follows a standard format.</p><p>(The application &amp; connected services still need to enforce access permissions.)</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!ewp6!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1ddee2d0-6089-43f7-909a-6df95286fd27_1448x1086.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!ewp6!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1ddee2d0-6089-43f7-909a-6df95286fd27_1448x1086.png 424w, https://substackcdn.com/image/fetch/$s_!ewp6!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1ddee2d0-6089-43f7-909a-6df95286fd27_1448x1086.png 848w, https://substackcdn.com/image/fetch/$s_!ewp6!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1ddee2d0-6089-43f7-909a-6df95286fd27_1448x1086.png 1272w, https://substackcdn.com/image/fetch/$s_!ewp6!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1ddee2d0-6089-43f7-909a-6df95286fd27_1448x1086.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!ewp6!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1ddee2d0-6089-43f7-909a-6df95286fd27_1448x1086.png" width="1448" height="1086" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/1ddee2d0-6089-43f7-909a-6df95286fd27_1448x1086.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1086,&quot;width&quot;:1448,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!ewp6!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1ddee2d0-6089-43f7-909a-6df95286fd27_1448x1086.png 424w, https://substackcdn.com/image/fetch/$s_!ewp6!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1ddee2d0-6089-43f7-909a-6df95286fd27_1448x1086.png 848w, https://substackcdn.com/image/fetch/$s_!ewp6!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1ddee2d0-6089-43f7-909a-6df95286fd27_1448x1086.png 1272w, https://substackcdn.com/image/fetch/$s_!ewp6!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1ddee2d0-6089-43f7-909a-6df95286fd27_1448x1086.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><strong>Analogy</strong></h3><p>Think of a <em>standard</em> connector between devices:</p><p>A shared connector reduces custom wiring. Each device still offers different capabilities &amp; plugging it in doesn&#8217;t grant permission to use everything.</p><h3><strong>Common mistake</strong></h3><p>Treating an MCP server as automatically trusted.</p><p>A server could expose sensitive data or powerful actions. Review its capabilities, limit permissions &amp; decide which actions need approval.</p><h3><strong>Why it matters</strong></h3><p>MCP reduces custom integration work across compatible AI applications.</p><p>A server&#8217;s capabilities become reusable without designing a different communication interface for every application.</p><h2><strong>27. Parameter-Efficient Fine-Tuning and LoRA</strong></h2><blockquote><p>Parameter-Efficient Fine-Tuning (<strong>PEFT</strong>) adapts a model by training only a small portion of its parameters or added parameters. Low-Rank Adaptation (<strong>LoRA</strong>) is one method that learns compact additions to selected model weights.</p></blockquote><p>You want a model to follow your support team&#8217;s response style.</p><p>Full fine-tuning updates all of the model&#8217;s weights. LoRA keeps the original weights fixed and trains smaller groups of numbers that represent changes to selected layers. These learned additions influence the model&#8217;s output.</p><p>The additions are then saved as an adapter<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-6" href="#footnote-6" target="_self">6</a>.</p><p>The adapter works alongside the base model, or its changes are merged into compatible weights.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!n-iV!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faeb49f05-a6ee-4ae3-861a-1a24fe760234_1448x1086.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!n-iV!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faeb49f05-a6ee-4ae3-861a-1a24fe760234_1448x1086.png 424w, https://substackcdn.com/image/fetch/$s_!n-iV!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faeb49f05-a6ee-4ae3-861a-1a24fe760234_1448x1086.png 848w, https://substackcdn.com/image/fetch/$s_!n-iV!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faeb49f05-a6ee-4ae3-861a-1a24fe760234_1448x1086.png 1272w, https://substackcdn.com/image/fetch/$s_!n-iV!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faeb49f05-a6ee-4ae3-861a-1a24fe760234_1448x1086.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!n-iV!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faeb49f05-a6ee-4ae3-861a-1a24fe760234_1448x1086.png" width="1448" height="1086" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/aeb49f05-a6ee-4ae3-861a-1a24fe760234_1448x1086.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1086,&quot;width&quot;:1448,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!n-iV!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faeb49f05-a6ee-4ae3-861a-1a24fe760234_1448x1086.png 424w, https://substackcdn.com/image/fetch/$s_!n-iV!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faeb49f05-a6ee-4ae3-861a-1a24fe760234_1448x1086.png 848w, https://substackcdn.com/image/fetch/$s_!n-iV!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faeb49f05-a6ee-4ae3-861a-1a24fe760234_1448x1086.png 1272w, https://substackcdn.com/image/fetch/$s_!n-iV!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Faeb49f05-a6ee-4ae3-861a-1a24fe760234_1448x1086.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><strong>Analogy</strong></h3><p>Think of a company handbook with a short <em>supplement</em> for each department:</p><p>The main handbook stays the same. Each supplement adapts to a specific role. A LoRA adapter serves a similar purpose through learned numerical changes.</p><h3><strong>Common mistake</strong></h3><p>Assuming a small adapter works with any model.</p><p>Its changes depend on the base model &amp; layers used during training. Keep the matching model version and configuration, then test the combined result on separate examples.</p><h3><strong>Why it matters</strong></h3><p>PEFT reduces the resources needed to train and store specialized adaptations.</p><p>The base model still needs memory and computation, but maintaining several task-specific versions becomes more practical.</p><h4 style="text-align: center;"><strong>&#167;</strong></h4><div class="callout-block" data-callout="true"><p style="text-align: center;"><em><strong><span data-color="#3a7bf9" style="color: rgb(58, 123, 249);">Reminder: this is a teaser of the subscriber-only newsletter, exclusive to my golden members.</span></strong></em></p><p>When you upgrade, you&#8217;ll get:</p><ul><li><p><strong>High-level architecture of real-world systems.</strong></p></li><li><p>Deep dive into how popular real-world systems work.</p></li><li><p><strong>How real-world systems handle scale, reliability &amp; performance.</strong></p></li></ul><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/subscribe&quot;,&quot;text&quot;:&quot;Unlock Full Access&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://newsletter.systemdesign.one/subscribe"><span>Unlock Full Access</span></a></p><p>(If this newsletter has helped you become a better software engineer, consider subscribing to support my work.)</p></div><h4 style="text-align: center;"><strong>&#167;</strong></h4><h2><strong>28. Diffusion Models</strong></h2>
      <p>
          <a href="https://newsletter.systemdesign.one/p/advanced-ai-concepts-explained">
              Read more
          </a>
      </p>
   ]]></content:encoded></item><item><title><![CDATA[Networking - A Deep Dive]]></title><description><![CDATA[#192: Part 1 - Must Know Networking Concepts Explained]]></description><link>https://newsletter.systemdesign.one/p/networking-fundamentals-software-engineers</link><guid isPermaLink="false">https://newsletter.systemdesign.one/p/networking-fundamentals-software-engineers</guid><dc:creator><![CDATA[Neo Kim]]></dc:creator><pubDate>Mon, 19 Oct 2026 14:46:45 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/eeecbfac-3f92-46f2-99d7-7e29eccfca1c_1280x720.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Once upon a time, there lived a full-stack engineer named Jung.</p><p>He was very good at building apps &amp; deploying them.</p><p>Until one day, when the DevOps engineer from his team fell ill.</p><p>And he got the task of updating the VPC for their production app.</p><p>Yet he had zero knowledge of networking fundamentals.</p><p>So his tiny config change took down the entire app&#8230;</p><p>Onward.</p><h4 style="text-align: center;"><strong>&#167;</strong></h4><div class="callout-block" data-callout="true"><h3><a href="https://coveron.com/neocoveron">The Missing Layer in Your Security Stack (Partner)</a></h3><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://coveron.com/neocoveron" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!1zWe!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbca843bf-a417-43b8-908a-c3c59f0535fa_1868x1034.png 424w, https://substackcdn.com/image/fetch/$s_!1zWe!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbca843bf-a417-43b8-908a-c3c59f0535fa_1868x1034.png 848w, https://substackcdn.com/image/fetch/$s_!1zWe!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbca843bf-a417-43b8-908a-c3c59f0535fa_1868x1034.png 1272w, https://substackcdn.com/image/fetch/$s_!1zWe!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbca843bf-a417-43b8-908a-c3c59f0535fa_1868x1034.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!1zWe!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbca843bf-a417-43b8-908a-c3c59f0535fa_1868x1034.png" width="1456" height="806" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/bca843bf-a417-43b8-908a-c3c59f0535fa_1868x1034.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:806,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:&quot;https://coveron.com/neocoveron&quot;,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!1zWe!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbca843bf-a417-43b8-908a-c3c59f0535fa_1868x1034.png 424w, https://substackcdn.com/image/fetch/$s_!1zWe!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbca843bf-a417-43b8-908a-c3c59f0535fa_1868x1034.png 848w, https://substackcdn.com/image/fetch/$s_!1zWe!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbca843bf-a417-43b8-908a-c3c59f0535fa_1868x1034.png 1272w, https://substackcdn.com/image/fetch/$s_!1zWe!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbca843bf-a417-43b8-908a-c3c59f0535fa_1868x1034.png 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Most security tools focus on preventing or detecting an attack.</p><p>Password managers protect credentials. Monitoring tools flag suspicious activity. But when a scam succeeds and money is lost, detection alone doesn&#8217;t help you recover it.</p><p><a href="https://coveron.com/neocoveron">Coveron adds a recovery layer to your security stack.</a></p><p>As someone who spends most of the day online, I already use tools to reduce security risks, but I like the idea of having a fallback if those protections fail.</p><p>Coveron monitors for suspicious activity and exposed personal data, then adds financial protection and hands-on recovery support when something goes wrong.</p><p>What you get:</p><ul><li><p>Up to $100K for eligible scam or online fraud losses</p></li><li><p>Up to $100K for cyber extortion or threats</p></li><li><p>Up to $2M for eligible identity theft recovery costs</p></li><li><p>Real people who guide you step-by-step through recovery</p></li></ul><p>Tools reduce risk. Insurance reduces loss. Coveron is built for what happens when prevention isn&#8217;t enough.</p><p style="text-align: center;"><em><strong>One scam can cost you everything &#8212; protect yourself now, first 100 users get 20% off with code neocoveron</strong></em></p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://coveron.com/neocoveron&quot;,&quot;text&quot;:&quot;Get Coveron &#8594;&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://coveron.com/neocoveron"><span>Get Coveron &#8594;</span></a></p><p><em>(Thanks to <a href="https://coveron.com/neocoveron">Coveron</a> for partnering on this newsletter.)</em></p></div><h4 style="text-align: center;"><strong>&#167;</strong></h4><p>I spent a few hours putting together this list of foundational networking concepts, so you don&#8217;t have to.</p><p>Curious to know how many are new to you:</p><ol><li><p>Network stack,</p></li><li><p>Encapsulation,</p></li><li><p>IP addressing,</p></li><li><p>Subnets &amp; CIDR,</p></li><li><p>VPC,</p></li><li><p>Routing &amp; forwarding,</p></li><li><p>NAT,</p></li><li><p>BGP,</p></li><li><p>Ports and sockets,</p></li><li><p>VPN.</p></li></ol><p>For each, I&#8217;ll share:</p><ul><li><p>What it is &amp; how it works in simple words,</p></li><li><p>A real-world analogy,</p></li><li><p>Why it matters.</p></li></ul><p>Let&#8217;s go!</p><h4 style="text-align: center;"><strong>&#167;</strong></h4><p></p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption"><span data-color="#3a7bf9" style="color: rgb(58, 123, 249);">Don't miss out on new updates! Enter your email to stay in the loop:</span></p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><ul><li><p><em><a href="https://newsletter.systemdesign.one/p/networking-fundamentals-software-engineers/?action=share">Share this letter</a> &amp; I&#8217;ll send you some rewards for the referrals.</em></p></li></ul><h4 style="text-align: center;"><strong>&#167;</strong></h4><h2><strong>1. Network stack</strong></h2><blockquote><p>The network stack divides communication into layers. Each layer handles a specific task &amp; passes data to the next layer.</p></blockquote><p>Two models explain how these layers work:</p><ul><li><p><strong>OSI Model:</strong> The Open Systems Interconnection (OSI<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-1" href="#footnote-1" target="_self">1</a>) model has 7 layers. It helps you understand and troubleshoot network communication.</p></li><li><p><strong>TCP/IP Model:</strong> The Transmission Control Protocol/Internet Protocol (TCP/IP) model has four layers: Application, Transport, Internet, and Link. It reflects how modern internet protocols work.</p></li></ul><p><em>Think about opening a website:</em></p><p>Your browser sends an HTTP request. The transport layer manages communication between applications, the internet layer routes packets toward their destination, and the link layer delivers data across the local network.</p><p>Each layer handles its own responsibility.</p><p>So your app does not have to manage packet routing or physical transmission.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!PsRr!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2f9802e7-80ab-4033-9b24-57572926e805_1536x1024.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!PsRr!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2f9802e7-80ab-4033-9b24-57572926e805_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!PsRr!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2f9802e7-80ab-4033-9b24-57572926e805_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!PsRr!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2f9802e7-80ab-4033-9b24-57572926e805_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!PsRr!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2f9802e7-80ab-4033-9b24-57572926e805_1536x1024.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!PsRr!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2f9802e7-80ab-4033-9b24-57572926e805_1536x1024.png" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/2f9802e7-80ab-4033-9b24-57572926e805_1536x1024.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!PsRr!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2f9802e7-80ab-4033-9b24-57572926e805_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!PsRr!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2f9802e7-80ab-4033-9b24-57572926e805_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!PsRr!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2f9802e7-80ab-4033-9b24-57572926e805_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!PsRr!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2f9802e7-80ab-4033-9b24-57572926e805_1536x1024.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><strong>Analogy</strong></h3><p>A letter passing through the postal system:</p><p>You write the words. An envelope carries the address; a sorting office picks the route; and a truck moves the physical mail. Each stage trusts the next to do its job.</p><h3><strong>Why it matters</strong></h3><p>Imagine your application suddenly loses its database connection&#8230;</p><p>The problem could occur in several places:</p><ul><li><p><strong>Application layer:</strong> DNS fails to resolve the database hostname.</p></li><li><p><strong>Transport layer:</strong> A firewall blocks the database&#8217;s TCP port.</p></li><li><p><strong>Transport/Application boundary:</strong> A TLS handshake fails.</p></li><li><p><strong>Internet layer:</strong> A routing issue prevents packets from reaching the database.</p></li></ul><p>Understanding the network stack helps you identify where communication fails &amp; narrow down the root cause.</p><p>Reach for <code>tcpdump</code> when you suspect the transport, and <code>curl</code> when you suspect the application. Pair it with encapsulation, which shows what each layer adds to your data.</p><h2><strong>2. Encapsulation</strong></h2><blockquote><p>Encapsulation is the process of wrapping data with protocol headers as it moves down the network stack.</p></blockquote><p>When you send an HTTP request, each layer adds information needed to deliver data:</p><ol><li><p><strong>Application Layer:</strong> Creates the HTTP request containing the payload.</p></li><li><p><strong>Transport Layer:</strong> Adds a TCP header with source &amp; destination ports, sequence numbers, and control information.</p></li><li><p><strong>Network Layer:</strong> Adds an IP header with source &amp; destination IP addresses.</p></li><li><p><strong>Data Link Layer:</strong> Wraps the IP packet in an Ethernet frame with a header &amp; trailer.</p></li></ol><p>The receiving machine reverses this process; it&#8217;s called <em>decapsulation</em>.</p><p>Each layer processes its protocol information &amp; passes remaining data to layer above.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!dJGi!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F751a1c05-5b8b-4886-a9f0-ab69fbd148c2_1536x1024.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!dJGi!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F751a1c05-5b8b-4886-a9f0-ab69fbd148c2_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!dJGi!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F751a1c05-5b8b-4886-a9f0-ab69fbd148c2_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!dJGi!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F751a1c05-5b8b-4886-a9f0-ab69fbd148c2_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!dJGi!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F751a1c05-5b8b-4886-a9f0-ab69fbd148c2_1536x1024.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!dJGi!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F751a1c05-5b8b-4886-a9f0-ab69fbd148c2_1536x1024.png" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/751a1c05-5b8b-4886-a9f0-ab69fbd148c2_1536x1024.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!dJGi!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F751a1c05-5b8b-4886-a9f0-ab69fbd148c2_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!dJGi!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F751a1c05-5b8b-4886-a9f0-ab69fbd148c2_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!dJGi!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F751a1c05-5b8b-4886-a9f0-ab69fbd148c2_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!dJGi!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F751a1c05-5b8b-4886-a9f0-ab69fbd148c2_1536x1024.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><strong>Analogy</strong></h3><p>A gift in a box, inside a padded parcel, inside a shipping crate.</p><p>Each wrapper carries the label its own handler needs, and nobody opens more than one layer.</p><h3><strong>Why it matters</strong></h3><p>Every header adds overhead to network communication.</p><p>Sending a 10-byte payload over TCP/IPv4 and Ethernet requires:</p><ul><li><p>TCP header: 20 bytes</p></li><li><p>IPv4 header: 20 bytes</p></li><li><p>Ethernet header and trailer: 18 bytes</p></li><li><p>Application payload: 10 bytes</p></li></ul><p>That&#8217;s 58 bytes of protocol overhead for just 10 bytes of payload, excluding Ethernet padding and other link-level overhead.</p><p>Small messages consume significantly more bandwidth than their payload size suggests.</p><p>Read frames with <code>Wireshark</code> and set path sizes in your virtual private network config. Read it alongside ports and sockets, which explain the transport header fields.</p><h2><strong>3. IP addressing</strong></h2><blockquote><p>An Internet Protocol (<strong>IP</strong>) address identifies a network interface &amp; helps routers deliver packets to the correct destination.</p></blockquote><p>There are two main versions:</p><ul><li><p><strong>IPv4:</strong> Uses 32-bit addresses, such as <code>192.168.1.1</code>. It supports 4.3 billion unique addresses.</p></li><li><p><strong>IPv6:</strong> Uses 128-bit addresses, such as <code>2001:db8::1</code>. It provides a vastly larger address space.</p></li></ul><p>IPv4 addresses became <em>scarce</em> as the Internet grew&#8230;</p><p>To conserve public addresses, networks adopted Network Address Translation (<strong>NAT</strong>). This allows many devices to share one public IPv4 address.</p><p>IPv6 addresses this limitation by providing enough addresses for an enormous number of devices &amp; networks.</p><p>Plus, many networks support both IPv4 &amp; IPv6, an approach called <strong>dual stack</strong>.</p><ul><li><p>Your laptop could have both an IPv4 and an IPv6 address.</p></li><li><p>A website could publish both IPv4 (<strong>A</strong>) and IPv6 (<strong>AAAA</strong>) DNS records.</p></li><li><p>Your device selects a reachable address &amp; connects using corresponding protocol.</p></li></ul><p>IPv6 adoption varies across networks&#8230;</p><p>Some mobile carriers and cloud providers support it extensively, while many corporate networks still rely on IPv4.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!xwz_!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e856f42-7683-43b8-84ac-192471cb95eb_1536x1024.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!xwz_!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e856f42-7683-43b8-84ac-192471cb95eb_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!xwz_!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e856f42-7683-43b8-84ac-192471cb95eb_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!xwz_!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e856f42-7683-43b8-84ac-192471cb95eb_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!xwz_!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e856f42-7683-43b8-84ac-192471cb95eb_1536x1024.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!xwz_!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e856f42-7683-43b8-84ac-192471cb95eb_1536x1024.png" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/2e856f42-7683-43b8-84ac-192471cb95eb_1536x1024.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!xwz_!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e856f42-7683-43b8-84ac-192471cb95eb_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!xwz_!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e856f42-7683-43b8-84ac-192471cb95eb_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!xwz_!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e856f42-7683-43b8-84ac-192471cb95eb_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!xwz_!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e856f42-7683-43b8-84ac-192471cb95eb_1536x1024.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><strong>Analogy</strong></h3><p>Think of IPv4 as a city running out of house addresses.</p><p>IPv6 expands the address space so dramatically that there is room for an enormous number of new buildings.</p><h3><strong>Why it matters</strong></h3><p>When designing a cloud network, you need to choose which IP versions to support.</p><ul><li><p><strong>AWS VPC:</strong> Configure IPv4-only, dual-stack, or IPv6-only subnets according to your requirements.</p></li><li><p><strong>Kubernetes:</strong> Configure IPv4, IPv6, or dual-stack networking based on your cluster and Container Network Interface (CNI) support.</p></li></ul><p>Also you need to understand <em>subnetting</em>.</p><p>It divides an IP address range into smaller networks for routing &amp; address management.</p><h2><strong>4. Subnets and CIDR</strong></h2>
      <p>
          <a href="https://newsletter.systemdesign.one/p/networking-fundamentals-software-engineers">
              Read more
          </a>
      </p>
   ]]></content:encoded></item><item><title><![CDATA[HTTP1 vs HTTP2 vs HTTP3]]></title><description><![CDATA[#191: Part 2 - 23 Concepts Every Engineer Must Know]]></description><link>https://newsletter.systemdesign.one/p/http2-vs-http3-explained</link><guid isPermaLink="false">https://newsletter.systemdesign.one/p/http2-vs-http3-explained</guid><dc:creator><![CDATA[Neo Kim]]></dc:creator><pubDate>Sat, 17 Oct 2026 13:34:54 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/3e68996e-75f2-4bb9-8a93-12c2a2b06a8e_1280x720.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p></p><div class="callout-block" data-callout="true"><h2><a href="https://nordvpn.com/neokim">Security at the Architecture Level (Partner)</a></h2><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://nordvpn.com/neokim" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!wN90!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5daa28bf-f309-480e-9b7e-7aed8a0c3300_1080x1080.png 424w, https://substackcdn.com/image/fetch/$s_!wN90!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5daa28bf-f309-480e-9b7e-7aed8a0c3300_1080x1080.png 848w, https://substackcdn.com/image/fetch/$s_!wN90!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5daa28bf-f309-480e-9b7e-7aed8a0c3300_1080x1080.png 1272w, https://substackcdn.com/image/fetch/$s_!wN90!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5daa28bf-f309-480e-9b7e-7aed8a0c3300_1080x1080.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!wN90!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5daa28bf-f309-480e-9b7e-7aed8a0c3300_1080x1080.png" width="1080" height="1080" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/5daa28bf-f309-480e-9b7e-7aed8a0c3300_1080x1080.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1080,&quot;width&quot;:1080,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:&quot;https://nordvpn.com/neokim&quot;,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!wN90!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5daa28bf-f309-480e-9b7e-7aed8a0c3300_1080x1080.png 424w, https://substackcdn.com/image/fetch/$s_!wN90!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5daa28bf-f309-480e-9b7e-7aed8a0c3300_1080x1080.png 848w, https://substackcdn.com/image/fetch/$s_!wN90!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5daa28bf-f309-480e-9b7e-7aed8a0c3300_1080x1080.png 1272w, https://substackcdn.com/image/fetch/$s_!wN90!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5daa28bf-f309-480e-9b7e-7aed8a0c3300_1080x1080.png 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Most online attacks don&#8217;t happen in one step.</p><p>A phishing link leads to a fake site, the site pushes a malicious download, and the download delivers the payload.</p><p><a href="https://nordvpn.com/neokim">NordVPN approaches this like a defense-in-depth problem.</a></p><p>As someone who works online every day, I already think about security in layers, so having phishing, scam, and malware protection alongside the VPN is useful to me.</p><p>NordVPN&#8217;s built-in next-gen antivirus aims to intercept threats earlier in the attack pipeline instead of waiting until a malicious file has already reached your device. NordVPN uses a proprietary machine learning model to block phishing sites before they cause harm, while its malware protection scans downloading files and removes them if they&#8217;re dangerous.</p><p>What you get:</p><ul><li><p>Anti-phishing protection powered by NordVPN&#8217;s proprietary machine-learning model</p></li><li><p>Protection against scam websites and fake online stores</p></li><li><p>Malware scanning for downloading files</p></li></ul><p>The VPN remains a core part of NordVPN, alongside these additional security features.</p><p>Depending on your plan and operating system, one subscription gives you access to multiple security tools.</p><h4 style="text-align: center;"><strong><a href="https://nordvpn.com/neokim"><mark data-color="#ffff00" style="background-color: rgb(255, 255, 0); color: rgb(0, 0, 0);">Get NordVPN 2Y plan with 4 extra months&#8594;</mark></a></strong></h4><p>Plus, it comes with a 30-day money-back guarantee.</p><p><em>(Thanks to <a href="https://nordvpn.com/neokim">NordVPN</a> for partnering on this newsletter.)</em></p></div><h4 style="text-align: center;"><strong>&#167;</strong></h4><p><code>HTTP/2</code> was supposed to make the web faster.</p><p>It introduced multiplexing, header compression &amp; better connection reuse.</p><p>But one lost TCP packet could still delay every stream on the connection.</p><p><code>HTTP/3</code> was built to solve this problem.</p><p>In this newsletter, we&#8217;ll break down 12 essential concepts behind <code>HTTP/2</code> &amp; <code>HTTP/3</code>:</p><ol start="12"><li><p>Multiplexing,</p></li><li><p>HPACK header compression,</p></li><li><p>Flow control,</p></li><li><p>Stream prioritization,</p></li><li><p>Server push &amp; early hints,</p></li><li><p>TCP head-of-line blocking,</p></li><li><p>QUIC over UDP,</p></li><li><p>QUIC handshake,</p></li><li><p>Independent streams &amp; loss recovery,</p></li><li><p>Connection IDs &amp; connection migration,</p></li><li><p>QPACK,</p></li><li><p>Alt-Svc &amp; HTTPS DNS records.</p></li></ol><p>For each, I&#8217;ll share:</p><ul><li><p>What it is &amp; how it works in simple words,</p></li><li><p>A real-world analogy,</p></li><li><p>Why it matters.</p></li></ul><p>Let&#8217;s go!</p><h4 style="text-align: center;"><strong>&#167;</strong></h4><p></p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption"><span data-color="#3a7bf9" style="color: rgb(58, 123, 249);">Get my system design playbook for FREE on newsletter signup:</span></p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><ul><li><p><em><a href="https://newsletter.systemdesign.one/p/http2-vs-http3-explained/?action=share">Share this letter</a> &amp; I&#8217;ll send you some rewards for the referrals.</em></p></li></ul><h4 style="text-align: center;"><strong>&#167;</strong></h4><h2><strong>12. Multiplexing</strong></h2><p>Multiplexing sends many requests &amp; responses at the same time over ONE connection by interleaving their frames.</p><p>The queue disappears&#8230;</p><p>Request one waits on a slow database while other requests return &amp; render. The server replies in whatever order the data becomes ready.</p><p>The result shows up plainly in connection counts:</p><p>A desktop page opens <em>fewer</em> connections on <code>HTTP/2</code> than on <code>HTTP/1.1</code>. The gap roughly doubles on lighter pages, where one connection does the work of two.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!04N5!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc5da7ae0-b04c-4973-8fff-5d388620d182_1536x1024.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!04N5!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc5da7ae0-b04c-4973-8fff-5d388620d182_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!04N5!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc5da7ae0-b04c-4973-8fff-5d388620d182_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!04N5!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc5da7ae0-b04c-4973-8fff-5d388620d182_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!04N5!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc5da7ae0-b04c-4973-8fff-5d388620d182_1536x1024.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!04N5!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc5da7ae0-b04c-4973-8fff-5d388620d182_1536x1024.png" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/c5da7ae0-b04c-4973-8fff-5d388620d182_1536x1024.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!04N5!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc5da7ae0-b04c-4973-8fff-5d388620d182_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!04N5!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc5da7ae0-b04c-4973-8fff-5d388620d182_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!04N5!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc5da7ae0-b04c-4973-8fff-5d388620d182_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!04N5!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc5da7ae0-b04c-4973-8fff-5d388620d182_1536x1024.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><strong>Analogy</strong></h3><p>One conversation channel where each sentence carries the speaker&#8217;s name.</p><p>Three people talk at once, and nobody loses the thread.</p><h3><strong>Why it matters</strong></h3><p>Multiplexing changes how you bundle because many tiny files beat a large one for cache reuse.</p><p>Confirm which protocol your assets use with the devtools protocol column and measure the bundling change in WebPageTest. Yet TCP head-of-line blocking<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-1" href="#footnote-1" target="_self">1</a> marks the limit multiplexing could never remove.</p><h2><strong>13. HPACK Header Compression</strong></h2><p>HPACK is the header compression format for <code>HTTP/2</code>.</p><p>It&#8217;s built from a static table, a dynamic table &amp; Huffman coding<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-2" href="#footnote-2" target="_self">2</a>.</p><p>Headers repeat almost perfectly. Your <code>user agent</code> string, <code>accept</code> headers, and <code>cookies</code> go out identically on every request, and on an API-heavy page they outweigh the payload.</p><p>HPACK indexes them:</p><p>A fixed table of common header names covers the obvious cases, so a method line becomes a single byte. Headers you have already sent go into a dynamic table shared by both ends.</p><p>The second request references those entries by number rather than repeating them, and the saving compounds across every request on the connection.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!d7E7!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb8cadb70-e81d-4a3b-af8d-740259fe43d9_1536x1024.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!d7E7!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb8cadb70-e81d-4a3b-af8d-740259fe43d9_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!d7E7!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb8cadb70-e81d-4a3b-af8d-740259fe43d9_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!d7E7!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb8cadb70-e81d-4a3b-af8d-740259fe43d9_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!d7E7!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb8cadb70-e81d-4a3b-af8d-740259fe43d9_1536x1024.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!d7E7!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb8cadb70-e81d-4a3b-af8d-740259fe43d9_1536x1024.png" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/b8cadb70-e81d-4a3b-af8d-740259fe43d9_1536x1024.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!d7E7!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb8cadb70-e81d-4a3b-af8d-740259fe43d9_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!d7E7!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb8cadb70-e81d-4a3b-af8d-740259fe43d9_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!d7E7!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb8cadb70-e81d-4a3b-af8d-740259fe43d9_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!d7E7!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb8cadb70-e81d-4a3b-af8d-740259fe43d9_1536x1024.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><strong>Analogy</strong></h3><p>Shorthand between two colleagues who talk daily.</p><p>Once you both agree a short name means one specific document, you stop saying the full title.</p><h3><strong>Why it matters</strong></h3><p>Header weight decides real latency on pages carrying large authentication tokens.</p><p>Inspect encoded header sizes in Wireshark and compare them against the devtools network panel. QPACK rebuilds this idea for out-of-order delivery.</p><h2><strong>14. Flow Control</strong></h2><p>Flow control limits how much data a sender may have outstanding, applied per stream &amp; across the whole connection.</p><p>The receiver sets a flow-control window. This limits how much DATA the sender can transmit before receiving additional flow-control credit. A <code>WINDOW_UPDATE</code> frame grants more room as the receiver consumes data, and the default initial window is 65,535 bytes.</p><p>This mechanism protects slow consumers<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-3" href="#footnote-3" target="_self">3</a>:</p><p>A browser decoding a large image should slow the pipe rather than buffer forever, and a proxy feeding a slow client behaves the same way.</p><p>Also it stops one greedy stream from monopolizing a connection everything else depends on.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!8jbF!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6e08a09d-25e4-45dd-bca5-188489a5b430_1536x1024.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!8jbF!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6e08a09d-25e4-45dd-bca5-188489a5b430_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!8jbF!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6e08a09d-25e4-45dd-bca5-188489a5b430_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!8jbF!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6e08a09d-25e4-45dd-bca5-188489a5b430_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!8jbF!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6e08a09d-25e4-45dd-bca5-188489a5b430_1536x1024.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!8jbF!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6e08a09d-25e4-45dd-bca5-188489a5b430_1536x1024.png" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/6e08a09d-25e4-45dd-bca5-188489a5b430_1536x1024.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!8jbF!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6e08a09d-25e4-45dd-bca5-188489a5b430_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!8jbF!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6e08a09d-25e4-45dd-bca5-188489a5b430_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!8jbF!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6e08a09d-25e4-45dd-bca5-188489a5b430_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!8jbF!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6e08a09d-25e4-45dd-bca5-188489a5b430_1536x1024.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><strong>Analogy</strong></h3><p>A coworker saying, &#8220;Send me five files at a time, and I will ask for more.&#8221;</p><p>They control the pace, so nothing piles up on their desk.</p><h3><strong>Why it matters</strong></h3><p>Large uploads, gRPC streams, and video segments all hit this ceiling before they hit bandwidth.</p><p>Watch window updates with <code>nghttp</code> and confirm socket-level throughput with <code>ss</code><a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-4" href="#footnote-4" target="_self">4</a>. Independent streams show how QUIC applies the same idea per stream.</p><h4 style="text-align: center;"><strong>&#167;</strong></h4><div class="captioned-button-wrap" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/p/http2-vs-http3-explained?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;}" data-component-name="CaptionedButtonToDOM"><div class="preamble"><p class="cta-caption"><em><span data-color="#3a7bf9" style="color: rgb(58, 123, 249);">If something resonated with you today, share this letter. Because one idea, one action, can change everything.</span></em></p></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/p/http2-vs-http3-explained?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://newsletter.systemdesign.one/p/http2-vs-http3-explained?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p></div><h4 style="text-align: center;"><strong>&#167;</strong></h4><h2><strong>15. Stream Prioritization</strong></h2><p>Stream prioritization lets a client tell the server which responses matter most, so the server can send that data first.</p><p>A browser needs CSS to render a webpage. Giving CSS higher priority than a large image helps the page become usable quicker, even if the total amount of data stays the same.</p><p><code>HTTP/2</code> used a complex system of stream dependencies &amp; weights to decide which data to send first. But browsers &amp; servers struggled to implement it consistently!</p><p>Then a simpler priority scheme got introduced... It uses 2 signals:</p><ul><li><p><strong>Urgency (0&#8211;7):</strong> Indicates how important a response is. Zero is the highest priority &amp; seven is the lowest.</p></li><li><p><strong>Incremental flag:</strong> Indicates whether a response can be delivered progressively, thus allowing the server to interleave its data with other responses.</p></li></ul><p>Also browsers provide the Fetch Priority API.</p><p>This lets developers influence the priority of resources using attributes such as <code>fetchpriority="high"</code>.</p><p>The browser uses these hints when scheduling requests and assigning HTTP priorities, although the Fetch Priority API does not directly expose HTTP&#8217;s urgency values.</p><p>Adoption of the Fetch Priority API grew rapidly after Chrome introduced support.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!cst2!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb79e2d65-7e89-46b3-b7f8-65c8d051d60a_1536x1024.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!cst2!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb79e2d65-7e89-46b3-b7f8-65c8d051d60a_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!cst2!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb79e2d65-7e89-46b3-b7f8-65c8d051d60a_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!cst2!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb79e2d65-7e89-46b3-b7f8-65c8d051d60a_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!cst2!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb79e2d65-7e89-46b3-b7f8-65c8d051d60a_1536x1024.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!cst2!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb79e2d65-7e89-46b3-b7f8-65c8d051d60a_1536x1024.png" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/b79e2d65-7e89-46b3-b7f8-65c8d051d60a_1536x1024.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!cst2!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb79e2d65-7e89-46b3-b7f8-65c8d051d60a_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!cst2!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb79e2d65-7e89-46b3-b7f8-65c8d051d60a_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!cst2!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb79e2d65-7e89-46b3-b7f8-65c8d051d60a_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!cst2!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb79e2d65-7e89-46b3-b7f8-65c8d051d60a_1536x1024.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><strong>Analogy</strong></h3><p>A triage nurse.</p><p>Everyone in the waiting room gets seen &amp; order reflects who needs help soonest.</p><h3><strong>Why it matters</strong></h3><p>One attribute on your largest visible image moves a headline performance metric.</p><p>Set fetchpriority in your markup and verify the assigned value in the Chrome devtools priority column. Early hints reach the same goal from the server side.</p><h2><strong>16. Server Push and Early Hints</strong></h2><p>Server Push lets a server send resources to a browser before browser requests them.</p><p>It was one of <code>HTTP/2</code>&#8217;s most promising features.</p><p>But it had its problems:</p><p>Servers pushed files already stored in the browser&#8217;s cache, thus wasting bandwidth. Also performance improvements were inconsistent, so adoption remained low.</p><p>(Chrome eventually disabled Server Push by default, and Firefox removed support.)</p><p>Early Hints emerged as a simpler alternative:</p><p>Instead of sending the resource itself, the server sends an HTTP <code>103 Early Hints</code> response containing <code>Link</code> headers while preparing the final response.</p><p>The browser then uses these hints to:</p><ul><li><p><strong>Preconnect:</strong> Establish connections to other servers before they&#8217;re needed.</p></li><li><p><strong>Preload:</strong> Start downloading crucial resources, such as CSS files, before the final HTML arrives.</p></li></ul><p>This helps the browser start useful work earlier &amp; potentially render the page faster.</p><p>Yet adoption remains limited&#8230; Only 2.9% of desktop pages used Early Hints, with Shopify accounting for roughly 90% of those deployments.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!quqC!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F71449a02-1ef0-4374-9733-769b7f81827d_1536x1024.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!quqC!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F71449a02-1ef0-4374-9733-769b7f81827d_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!quqC!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F71449a02-1ef0-4374-9733-769b7f81827d_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!quqC!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F71449a02-1ef0-4374-9733-769b7f81827d_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!quqC!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F71449a02-1ef0-4374-9733-769b7f81827d_1536x1024.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!quqC!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F71449a02-1ef0-4374-9733-769b7f81827d_1536x1024.png" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/71449a02-1ef0-4374-9733-769b7f81827d_1536x1024.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!quqC!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F71449a02-1ef0-4374-9733-769b7f81827d_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!quqC!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F71449a02-1ef0-4374-9733-769b7f81827d_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!quqC!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F71449a02-1ef0-4374-9733-769b7f81827d_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!quqC!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F71449a02-1ef0-4374-9733-769b7f81827d_1536x1024.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><strong>Analogy</strong></h3><p>Push is a waiter setting down a dish you never ordered &amp; may already have.</p><p>Early hints is a waiter handing you the menu while the kitchen warms up.</p><h3><strong>Why it matters</strong></h3><p>Dead push configuration survives in production long after browsers stopped honoring it.</p><p>Enable early hints at Cloudflare/Fastly and prove the gain with real user monitoring rather than a lab score. Chunked transfer encoding fills the same dead time once the response finally starts.</p><h2><strong>17. TCP Head-of-Line Blocking</strong></h2><p>TCP delivers data to applications in a strict order.</p><p>If a TCP packet gets lost, later data must wait until the missing packet gets retransmitted. Yet this creates a problem for <code>HTTP/2</code>&#8230;</p><p><code>HTTP/2</code> multiplexes multiple streams over one TCP connection. And TCP sees only a single ordered sequence of bytes, with no knowledge of individual <code>HTTP/2</code> streams. So one lost TCP packet delays data delivery across multiple streams, even when their data has already arrived.</p><p>This is called <em>TCP head-of-line (<strong>HOL</strong>) blocking.</em></p><p>TCP uses acknowledgments &amp; retransmissions to provide reliable, ordered delivery.</p><p>But this ordering can delay otherwise available data&#8230; On lossy networks, <code>HTTP/2</code> using one TCP connection sometimes performs worse than <code>HTTP/1.1</code> using six parallel connections.</p><p>Each TCP connection handles packet loss independently, so a lost packet on one connection does not directly block the others.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!20lI!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F48769818-b302-45c3-9c18-fceab11f4813_1536x1024.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!20lI!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F48769818-b302-45c3-9c18-fceab11f4813_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!20lI!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F48769818-b302-45c3-9c18-fceab11f4813_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!20lI!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F48769818-b302-45c3-9c18-fceab11f4813_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!20lI!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F48769818-b302-45c3-9c18-fceab11f4813_1536x1024.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!20lI!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F48769818-b302-45c3-9c18-fceab11f4813_1536x1024.png" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/48769818-b302-45c3-9c18-fceab11f4813_1536x1024.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!20lI!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F48769818-b302-45c3-9c18-fceab11f4813_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!20lI!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F48769818-b302-45c3-9c18-fceab11f4813_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!20lI!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F48769818-b302-45c3-9c18-fceab11f4813_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!20lI!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F48769818-b302-45c3-9c18-fceab11f4813_1536x1024.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><strong>Analogy</strong></h3><p>One conveyor belt at a sorting depot.</p><p>Drop a box &amp; everything behind it stops moving, even if those parcels have nothing to do with each other.</p><h3><strong>Why it matters</strong></h3><p>This single property is the entire reason <code>HTTP/3</code> exists.</p><p>Confirm retransmits with <code>ss -i</code> and capture the stall itself with <code>tcpdump</code><a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-5" href="#footnote-5" target="_self">5</a>. QUIC over UDP replaces the transport rather than working around it.</p><h4 style="text-align: center;"><strong>&#167;</strong></h4><div class="callout-block" data-callout="true"><p style="text-align: center;"><em><strong><span data-color="#3a7bf9" style="color: rgb(58, 123, 249);">Reminder: this is a teaser of the subscriber-only newsletter, exclusive to my golden members.</span></strong></em></p><p>When you upgrade, you&#8217;ll get:</p><ul><li><p><strong>High-level architecture of real-world systems.</strong></p></li><li><p>Deep dive into how popular real-world systems work.</p></li><li><p><strong>How real-world systems handle scale, reliability &amp; performance.</strong></p></li></ul><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/subscribe&quot;,&quot;text&quot;:&quot;Unlock Full Access&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://newsletter.systemdesign.one/subscribe"><span>Unlock Full Access</span></a></p><p>(If this newsletter has helped you become a better software engineer, consider subscribing to support my work.)</p></div><h4 style="text-align: center;"><strong>&#167;</strong></h4><h2><strong>18. QUIC over UDP</strong></h2>
      <p>
          <a href="https://newsletter.systemdesign.one/p/http2-vs-http3-explained">
              Read more
          </a>
      </p>
   ]]></content:encoded></item><item><title><![CDATA[HTTP1 vs HTTP2 vs HTTP3]]></title><description><![CDATA[#190: Part 1 - 23 Concepts Every Engineer Should Know]]></description><link>https://newsletter.systemdesign.one/p/http-concepts-explained</link><guid isPermaLink="false">https://newsletter.systemdesign.one/p/http-concepts-explained</guid><dc:creator><![CDATA[Neo Kim]]></dc:creator><pubDate>Thu, 15 Oct 2026 12:07:10 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/e3979e0c-96c2-4cea-bc82-998b1eb5e92e_1280x720.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p></p><div class="callout-block" data-callout="true"><h3><a href="https://www.vpdae.com/redirect/ap9y0q1160spq7um58llgjb4rq4">Live demo: give AI agents the access they need without credentials they can leak</a></h3><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://www.vpdae.com/redirect/ap9y0q1160spq7um58llgjb4rq4" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!CtdE!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F94f8bf40-15c7-47d7-acf7-37dbdafa15d2_1200x800.webp 424w, https://substackcdn.com/image/fetch/$s_!CtdE!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F94f8bf40-15c7-47d7-acf7-37dbdafa15d2_1200x800.webp 848w, https://substackcdn.com/image/fetch/$s_!CtdE!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F94f8bf40-15c7-47d7-acf7-37dbdafa15d2_1200x800.webp 1272w, https://substackcdn.com/image/fetch/$s_!CtdE!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F94f8bf40-15c7-47d7-acf7-37dbdafa15d2_1200x800.webp 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!CtdE!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F94f8bf40-15c7-47d7-acf7-37dbdafa15d2_1200x800.webp" width="1200" height="800" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/94f8bf40-15c7-47d7-acf7-37dbdafa15d2_1200x800.webp&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:800,&quot;width&quot;:1200,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:688818,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/webp&quot;,&quot;href&quot;:&quot;https://www.vpdae.com/redirect/ap9y0q1160spq7um58llgjb4rq4&quot;,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://newsletter.systemdesign.one/i/220234867?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F94f8bf40-15c7-47d7-acf7-37dbdafa15d2_1200x800.webp&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!CtdE!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F94f8bf40-15c7-47d7-acf7-37dbdafa15d2_1200x800.webp 424w, https://substackcdn.com/image/fetch/$s_!CtdE!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F94f8bf40-15c7-47d7-acf7-37dbdafa15d2_1200x800.webp 848w, https://substackcdn.com/image/fetch/$s_!CtdE!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F94f8bf40-15c7-47d7-acf7-37dbdafa15d2_1200x800.webp 1272w, https://substackcdn.com/image/fetch/$s_!CtdE!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F94f8bf40-15c7-47d7-acf7-37dbdafa15d2_1200x800.webp 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><a href="https://www.vpdae.com/redirect/ap9y0q1160spq7um58llgjb4rq4">Infisical&#8217;s</a> Agent Vault gives AI agents access to services without ever handing them secrets. A proxy attaches credentials on the way out, with no code changes. You control what agents can reach and when access expires.</p><p>See it in a 30-minute live demo on Wednesday, October 21 at 1 PM ET.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://www.vpdae.com/redirect/ap9y0q1160spq7um58llgjb4rq4&quot;,&quot;text&quot;:&quot;Save your spot&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://www.vpdae.com/redirect/ap9y0q1160spq7um58llgjb4rq4"><span>Save your spot</span></a></p><p><em>(Thanks to <a href="https://www.vpdae.com/redirect/ap9y0q1160spq7um58llgjb4rq4">Infisical</a> for partnering on this newsletter.)</em></p></div><h4 style="text-align: center;"><strong>&#167;</strong></h4><p>Why does a site take several seconds to load, even if server responds in milliseconds?</p><p>Before a single page appears, the browser must establish connections, send requests, and receive responses&#8230; </p><p>Understanding underlying mechanisms changed how I think about HTTP &amp; web performance.</p><p>In this newsletter, I&#8217;ll break down must-know HTTP concepts. Some of these are foundational &amp; some are quite advanced.</p><p>All of them are useful to engineers building production-ready systems:</p><ol><li><p>Request-response cycle,</p></li><li><p>TCP handshake,</p></li><li><p>Persistent connections,</p></li><li><p>Pipelining,</p></li><li><p>HTTP head-of-line blocking,</p></li><li><p>Chunked transfer encoding,</p></li><li><p>Caching &amp; conditional requests,</p></li><li><p>Parallel connections &amp; domain sharding,</p></li><li><p>TCP slow start,</p></li><li><p>The binary framing layer,</p></li><li><p>Streams, messages &amp; frames.</p></li></ol><p>(&#8230;and much more in Part 2!)</p><p>For each, I&#8217;ll share:</p><ul><li><p>What it is &amp; how it works in simple words,</p></li><li><p>A real-world analogy,</p></li><li><p>Why it matters.</p></li></ul><p>Let&#8217;s go!</p><h4 style="text-align: center;"><strong>&#167;</strong></h4><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption"><span data-color="#3a7bf9" style="color: rgb(58, 123, 249);">Get my system design playbook for FREE on newsletter signup:</span></p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><ul><li><p><em><a href="https://newsletter.systemdesign.one/p/http-concepts-explained/?action=share">Share this letter</a> &amp; I&#8217;ll send you some rewards for the referrals.</em></p></li></ul><h4 style="text-align: center;"><strong>&#167;</strong></h4><h2><strong>1. Request-Response Cycle</strong></h2><p>Every HTTP exchange is one request and one response.</p><p>The client sends a method, a path, and a set of headers. The server replies with a status code, its own headers, and usually a body. In <code>HTTP/1.x</code>, the message format is text-based, so you could type a request by hand into a terminal and read the reply with your own eyes<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-1" href="#footnote-1" target="_self">1</a>.</p><p>The earliest version of the protocol supported a single method and returned a single file type. The next version added headers, status codes, POST, and HEAD.</p><p>Also, it opened a fresh connection for every exchange and threw it away afterward. (This decision shaped the next 20+ years of workarounds.)</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!qDj_!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1379ec92-f6d6-431f-b3de-e41585b87575_1536x1024.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!qDj_!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1379ec92-f6d6-431f-b3de-e41585b87575_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!qDj_!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1379ec92-f6d6-431f-b3de-e41585b87575_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!qDj_!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1379ec92-f6d6-431f-b3de-e41585b87575_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!qDj_!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1379ec92-f6d6-431f-b3de-e41585b87575_1536x1024.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!qDj_!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1379ec92-f6d6-431f-b3de-e41585b87575_1536x1024.png" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/1379ec92-f6d6-431f-b3de-e41585b87575_1536x1024.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!qDj_!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1379ec92-f6d6-431f-b3de-e41585b87575_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!qDj_!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1379ec92-f6d6-431f-b3de-e41585b87575_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!qDj_!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1379ec92-f6d6-431f-b3de-e41585b87575_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!qDj_!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1379ec92-f6d6-431f-b3de-e41585b87575_1536x1024.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><strong>Analogy</strong></h3><p>Ordering by postcard:</p><p>You write one item, mail it, and wait for one parcel.</p><p>To order a second item, you write a second postcard.</p><h3><strong>Why it matters</strong></h3><p>Debugging an unexpected status code starts with reading the raw exchange, not your application logs.</p><p>Send requests by hand with <code>curl</code> and inspect real traffic in the Chrome DevTools Network tab. The binary framing layer picks up from here, keeping these semantics and changing the bytes.</p><h2><strong>2. TCP Three-Way Handshake</strong></h2><p>Transmission Control Protocol (<strong>TCP</strong>) opens a connection with 3 messages before either side sends data.</p><ol><li><p>Client sends a synchronize packet.</p></li><li><p>Server replies with an acknowledgment plus its own sequence number.</p></li><li><p>Client acknowledges the number &amp; connection opens.</p></li></ol><p>Three messages cost one full round trip&#8230;</p><p>i.e., on a link with 100ms of latency, you burn 100ms before the first byte of your request leaves the machine.</p><p>Encryption adds more on top&#8230;</p><p>A Transport Layer Security (<strong>TLS</strong>) 1.2 handshake costs two extra round trips, and TLS 1.3 cut that to one.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!kqhV!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcb7259ae-ad6d-4774-bb33-4ac352130f04_1536x1024.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!kqhV!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcb7259ae-ad6d-4774-bb33-4ac352130f04_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!kqhV!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcb7259ae-ad6d-4774-bb33-4ac352130f04_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!kqhV!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcb7259ae-ad6d-4774-bb33-4ac352130f04_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!kqhV!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcb7259ae-ad6d-4774-bb33-4ac352130f04_1536x1024.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!kqhV!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcb7259ae-ad6d-4774-bb33-4ac352130f04_1536x1024.png" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/cb7259ae-ad6d-4774-bb33-4ac352130f04_1536x1024.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!kqhV!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcb7259ae-ad6d-4774-bb33-4ac352130f04_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!kqhV!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcb7259ae-ad6d-4774-bb33-4ac352130f04_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!kqhV!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcb7259ae-ad6d-4774-bb33-4ac352130f04_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!kqhV!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcb7259ae-ad6d-4774-bb33-4ac352130f04_1536x1024.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><strong>Analogy</strong></h3><p>A phone call:</p><p>You dial, they say hello, and you say hello back. And only on the fourth breath does anyone say something useful.</p><h3><strong>Why it matters</strong></h3><p>Connection setup shows up in your p99 long before your handler code does.</p><p>Measure it with <code>curl -w</code> and watch it across a full page load in WebPageTest. Persistent connections spread that cost across many requests.</p><h2><strong>3. Persistent Connections</strong></h2><p>A persistent connection stays open after a response, so the next request reuses it.</p><p><code>HTTP/1.1</code> made this the default behavior. The connection lives until either side sends a close header or a timeout fires, so one handshake now covers dozens of requests.</p><p>The saving compounds on real pages.</p><p>A page with 40 assets used to pay 40 handshakes; on a single persistent connection, it pays one.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!_NtK!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F56a3242c-ef12-4451-a51a-f51cc4694aa8_1536x1024.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!_NtK!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F56a3242c-ef12-4451-a51a-f51cc4694aa8_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!_NtK!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F56a3242c-ef12-4451-a51a-f51cc4694aa8_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!_NtK!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F56a3242c-ef12-4451-a51a-f51cc4694aa8_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!_NtK!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F56a3242c-ef12-4451-a51a-f51cc4694aa8_1536x1024.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!_NtK!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F56a3242c-ef12-4451-a51a-f51cc4694aa8_1536x1024.png" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/56a3242c-ef12-4451-a51a-f51cc4694aa8_1536x1024.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!_NtK!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F56a3242c-ef12-4451-a51a-f51cc4694aa8_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!_NtK!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F56a3242c-ef12-4451-a51a-f51cc4694aa8_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!_NtK!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F56a3242c-ef12-4451-a51a-f51cc4694aa8_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!_NtK!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F56a3242c-ef12-4451-a51a-f51cc4694aa8_1536x1024.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><strong>Analogy</strong></h3><p>Keeping the phone line open between questions instead of hanging up and redialing every time you think of something new.</p><h3><strong>Why it matters</strong></h3><p>Connection churn on a busy proxy exhausts temporary ports well before it exhausts bandwidth<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-2" href="#footnote-2" target="_self">2</a>.</p><p>Tune <code>keepalive_timeout</code> in nginx and <code>server.keepAliveTimeout</code> in Node.js, then check both against your load balancer. TCP slow start explains why a reused connection also runs faster.</p><h4 style="text-align: center;"><strong>&#167;</strong></h4><div class="captioned-button-wrap" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/p/http-concepts-explained?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;}" data-component-name="CaptionedButtonToDOM"><div class="preamble"><p class="cta-caption"><em><span data-color="#3a7bf9" style="color: rgb(58, 123, 249);">If something resonated with you today, share this letter. Because one idea, one action, can change everything.</span></em></p></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/p/http-concepts-explained?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://newsletter.systemdesign.one/p/http-concepts-explained?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p></div><h4 style="text-align: center;"><strong>&#167;</strong></h4><h2><strong>4. Pipelining</strong></h2><p>Pipelining lets a client send several requests back-to-back on one connection without waiting for each response.</p><p>The browser asks for two images at once rather than asking, waiting, then asking again. On paper, it removes a full round trip per request while using nothing more than the connection you already have.</p><p>Yet one rule ruined it:</p><p>Responses must return in the exact order the requests went out, so a slow first response stalls everything behind it. Buggy proxies mangled pipelined traffic on top of that.</p><p>(NOTE: Because of these problems, major browsers disabled <code>HTTP/1.1</code> pipelining, and it never gained widespread adoption.)</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!l6G1!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fddc2205a-fa7c-4dd6-802f-7285b2d510b8_1536x1024.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!l6G1!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fddc2205a-fa7c-4dd6-802f-7285b2d510b8_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!l6G1!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fddc2205a-fa7c-4dd6-802f-7285b2d510b8_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!l6G1!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fddc2205a-fa7c-4dd6-802f-7285b2d510b8_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!l6G1!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fddc2205a-fa7c-4dd6-802f-7285b2d510b8_1536x1024.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!l6G1!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fddc2205a-fa7c-4dd6-802f-7285b2d510b8_1536x1024.png" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/ddc2205a-fa7c-4dd6-802f-7285b2d510b8_1536x1024.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!l6G1!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fddc2205a-fa7c-4dd6-802f-7285b2d510b8_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!l6G1!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fddc2205a-fa7c-4dd6-802f-7285b2d510b8_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!l6G1!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fddc2205a-fa7c-4dd6-802f-7285b2d510b8_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!l6G1!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fddc2205a-fa7c-4dd6-802f-7285b2d510b8_1536x1024.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><strong>Analogy</strong></h3><p>Ordering all three courses at once:</p><p>The kitchen still serves them in the order you named them, so a slow starter holds up your main.</p><h3><strong>Why it matters</strong></h3><p>You will find pipelining disabled in <code>curl</code> and every browser, and now you know why.</p><p>Confirm the behavior with <code>curl</code> and watch real request ordering in Wireshark<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-3" href="#footnote-3" target="_self">3</a>. Multiplexing solves the same problem without the ordering rule.</p><h2><strong>5. HTTP Head-of-Line Blocking</strong></h2><p>Head-of-line blocking occurs when first item in a queue delays every item behind it<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-4" href="#footnote-4" target="_self">4</a>.</p><p>One <code>HTTP/1.1</code> connection processes one exchange at a time:</p><p>A slow database query on request one could keep four other requests waiting, and the network sits idle while the queue holds.</p><p>This is the ceiling <code>HTTP/1.1</code> could never lift&#8230; Persistent connections removed the handshake cost and left the queue completely intact.</p><p>Every workaround built on top of it, from parallel connections to asset bundling, exists to hide this single property.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!20yz!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0b040cb0-94b8-4a97-9bf4-a763225c9020_1536x1024.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!20yz!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0b040cb0-94b8-4a97-9bf4-a763225c9020_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!20yz!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0b040cb0-94b8-4a97-9bf4-a763225c9020_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!20yz!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0b040cb0-94b8-4a97-9bf4-a763225c9020_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!20yz!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0b040cb0-94b8-4a97-9bf4-a763225c9020_1536x1024.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!20yz!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0b040cb0-94b8-4a97-9bf4-a763225c9020_1536x1024.png" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/0b040cb0-94b8-4a97-9bf4-a763225c9020_1536x1024.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!20yz!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0b040cb0-94b8-4a97-9bf4-a763225c9020_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!20yz!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0b040cb0-94b8-4a97-9bf4-a763225c9020_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!20yz!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0b040cb0-94b8-4a97-9bf4-a763225c9020_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!20yz!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0b040cb0-94b8-4a97-9bf4-a763225c9020_1536x1024.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><strong>Analogy</strong></h3><p>A single supermarket checkout:</p><p>The shopper at the front needs a price check, and the four people behind them stand still.</p><h3><strong>Why it matters</strong></h3><p>Stair-stepped requests with visible gaps in a waterfall chart point straight at this.</p><p>Read the waterfall in WebPageTest and count concurrent requests in the Chrome DevTools Network panel. Parallel connections show the workaround the browsers actually picked.</p><h2><strong>6. Chunked Transfer Encoding</strong></h2><p>Chunked transfer encoding lets a server send a response in pieces without knowing the total size in advance.</p><p>The server marks the response as chunked and writes each piece with its size in hexadecimal on the line above. A &#8220;zero-length&#8221; chunk marks the end.</p><p>The browser could start rendering before the full response arrives.</p><p>In <code>HTTP/1.1</code>, chunked transfer encoding supports streaming generated HTML, server-sent events, and token-by-token model responses.</p><p><code>HTTP/2</code> and <code>HTTP/3</code> use DATA frames instead.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!rxmV!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F597213e2-ccd4-4aa2-acf1-622ea0cf50ca_1536x1024.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!rxmV!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F597213e2-ccd4-4aa2-acf1-622ea0cf50ca_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!rxmV!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F597213e2-ccd4-4aa2-acf1-622ea0cf50ca_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!rxmV!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F597213e2-ccd4-4aa2-acf1-622ea0cf50ca_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!rxmV!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F597213e2-ccd4-4aa2-acf1-622ea0cf50ca_1536x1024.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!rxmV!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F597213e2-ccd4-4aa2-acf1-622ea0cf50ca_1536x1024.png" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/597213e2-ccd4-4aa2-acf1-622ea0cf50ca_1536x1024.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!rxmV!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F597213e2-ccd4-4aa2-acf1-622ea0cf50ca_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!rxmV!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F597213e2-ccd4-4aa2-acf1-622ea0cf50ca_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!rxmV!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F597213e2-ccd4-4aa2-acf1-622ea0cf50ca_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!rxmV!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F597213e2-ccd4-4aa2-acf1-622ea0cf50ca_1536x1024.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><strong>Analogy</strong></h3><p>A publisher shipping a book one chapter at a time.</p><p>You start reading chapter one while the author is still writing chapter four.</p><h3><strong>Why it matters</strong></h3><p>Streaming responses depend on the HTTP version&#8217;s framing mechanism, and accidental buffering often stops streaming. </p><p>Check <code>proxy_buffering</code> in nginx and <code>res.flushHeaders</code> in Express before you blame your application code. Early hints apply the same idea to the wait before a response starts.</p><h2><strong>7. Caching and Conditional Requests</strong></h2><p>Caching stores a copy of a response so a later request skips the network entirely.</p><p><code>Cache-Control</code> sets the rules:</p><p>A <code>max-age</code> directive says how long a copy stays fresh; <code>no-cache</code> forces a check on every use, and <code>immutable</code> promises the file never changes under its URL.</p><p>Conditional requests handle the check cheaply:</p><p>The client sends its stored entity tag in an <code>If-None-Match</code> header, and a server holding the same version answers <code>304 Not Modified</code> without a body.</p><p>A cache hit costs zero bytes on the wire and zero milliseconds of server time, thus making it the cheapest performance work available to you.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!ws71!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F27dd19af-6a1d-4b18-a890-f996dbe39526_1536x1024.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!ws71!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F27dd19af-6a1d-4b18-a890-f996dbe39526_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!ws71!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F27dd19af-6a1d-4b18-a890-f996dbe39526_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!ws71!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F27dd19af-6a1d-4b18-a890-f996dbe39526_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!ws71!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F27dd19af-6a1d-4b18-a890-f996dbe39526_1536x1024.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!ws71!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F27dd19af-6a1d-4b18-a890-f996dbe39526_1536x1024.png" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/27dd19af-6a1d-4b18-a890-f996dbe39526_1536x1024.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!ws71!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F27dd19af-6a1d-4b18-a890-f996dbe39526_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!ws71!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F27dd19af-6a1d-4b18-a890-f996dbe39526_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!ws71!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F27dd19af-6a1d-4b18-a890-f996dbe39526_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!ws71!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F27dd19af-6a1d-4b18-a890-f996dbe39526_1536x1024.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><strong>Analogy</strong></h3><p>Asking a librarian whether the report changed since your last visit.</p><p>If nothing changed, you read the copy already on your desk.</p><h3><strong>Why it matters</strong></h3><p>Static assets &amp; API responses need opposite caching policies, and mixing them up costs freshness or bandwidth.</p><p>Set the rules at the edge in Cloudflare/Varnish and read the real cache hit ratio there rather than trusting your config. Conditional requests lose their value once you version filenames properly.</p><h4 style="text-align: center;"><strong>&#167;</strong></h4><div class="captioned-button-wrap" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/p/http-concepts-explained?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;}" data-component-name="CaptionedButtonToDOM"><div class="preamble"><p class="cta-caption"><em><span data-color="#3a7bf9" style="color: rgb(58, 123, 249);">Share this letter &amp; I&#8217;ll send you some rewards for the referrals!</span></em></p></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/p/http-concepts-explained?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://newsletter.systemdesign.one/p/http-concepts-explained?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p></div><h4 style="text-align: center;"><strong>&#167;</strong></h4><h2><strong>8. Parallel Connections and Domain Sharding</strong></h2><p>Browsers open several TCP connections (typically 6) to the same origin at once.</p><p>Six lanes beat one lane, so head-of-line blocking hurts less. Sites wanting more than six invented extra hostnames, serving images from one subdomain &amp; scripts from another. That bought another six per name.</p><p>This technique is called <em>domain sharding</em>&#8230; It shaped front-end architecture.</p><p>But it became obsolete the moment multiplexing arrived&#8230; yet survives in production long after the sites that used it moved on&#8230;</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!vVLA!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa15fba0c-18ce-4683-9004-4a5d697c0dc8_1536x1024.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!vVLA!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa15fba0c-18ce-4683-9004-4a5d697c0dc8_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!vVLA!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa15fba0c-18ce-4683-9004-4a5d697c0dc8_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!vVLA!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa15fba0c-18ce-4683-9004-4a5d697c0dc8_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!vVLA!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa15fba0c-18ce-4683-9004-4a5d697c0dc8_1536x1024.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!vVLA!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa15fba0c-18ce-4683-9004-4a5d697c0dc8_1536x1024.png" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/a15fba0c-18ce-4683-9004-4a5d697c0dc8_1536x1024.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!vVLA!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa15fba0c-18ce-4683-9004-4a5d697c0dc8_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!vVLA!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa15fba0c-18ce-4683-9004-4a5d697c0dc8_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!vVLA!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa15fba0c-18ce-4683-9004-4a5d697c0dc8_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!vVLA!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa15fba0c-18ce-4683-9004-4a5d697c0dc8_1536x1024.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><strong>Analogy</strong></h3><p>Opening more checkout lanes.</p><p>When six lanes fill up, you rent the shop next door and open six more.</p><h3><strong>Why it matters</strong></h3><p>Removing dead shards is one of the easier wins available on an older codebase.</p><p>Measure the before and after in WebPageTest and let Lighthouse<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-5" href="#footnote-5" target="_self">5</a> flag the redundant origins. TCP slow start explains why those extra connections start slow.</p><h2><strong>9. TCP Slow Start</strong></h2><p>Slow start is a congestion control algorithm where a new connection begins with a small send window &amp; then grows it each round trip.</p><p>TCP has no idea how much bandwidth sits between you and your user. Guessing high floods the path and drops packets, so it guesses low and probes upward instead.</p><p>The window typically starts at 10 packets, roughly 14 kilobytes, and doubles on every successful round trip. Packet loss halves it and switches the sender to a slower climb<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-6" href="#footnote-6" target="_self">6</a>.</p><p>Without this algorithm, every new connection would flood the path and collapse throughput for everyone sharing it.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!ZUjE!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9151f9ee-0375-49ec-9afd-f717d924e29a_1536x1024.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!ZUjE!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9151f9ee-0375-49ec-9afd-f717d924e29a_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!ZUjE!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9151f9ee-0375-49ec-9afd-f717d924e29a_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!ZUjE!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9151f9ee-0375-49ec-9afd-f717d924e29a_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!ZUjE!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9151f9ee-0375-49ec-9afd-f717d924e29a_1536x1024.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!ZUjE!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9151f9ee-0375-49ec-9afd-f717d924e29a_1536x1024.png" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/9151f9ee-0375-49ec-9afd-f717d924e29a_1536x1024.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!ZUjE!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9151f9ee-0375-49ec-9afd-f717d924e29a_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!ZUjE!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9151f9ee-0375-49ec-9afd-f717d924e29a_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!ZUjE!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9151f9ee-0375-49ec-9afd-f717d924e29a_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!ZUjE!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9151f9ee-0375-49ec-9afd-f717d924e29a_1536x1024.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><strong>Analogy</strong></h3><p>Merging onto a motorway:</p><p>You accelerate in stages instead of flooring it blind into traffic you cannot see yet.</p><h3><strong>Why it matters</strong></h3><p>The first 14 kilobytes of your HTML carry more weight than the next 500.</p><p>Put critical CSS &amp; your hero image reference inside that window and let Lighthouse confirm the render path. Persistent connections keep the window you already earned.</p><h2><strong>10. Binary Framing Layer</strong></h2><p>Binary framing layer encodes HTTP messages as binary frames instead of plain text lines.</p><p>Each frame carries a length, type, set of flags, and a stream identifier in a fixed nine-byte header. The methods, headers, and status codes stay exactly as they were.</p><p>You still send <code>GET</code>&#8230; You still read <code>404</code>. The bytes on the wire look completely different, and frames parse faster and pack tighter than text ever did.</p><p>The stream identifier is the important part, and the next concept explains why.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!iqGz!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3c0fb8b3-f0a8-4c6c-98fb-a66476e416ea_1536x1024.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!iqGz!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3c0fb8b3-f0a8-4c6c-98fb-a66476e416ea_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!iqGz!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3c0fb8b3-f0a8-4c6c-98fb-a66476e416ea_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!iqGz!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3c0fb8b3-f0a8-4c6c-98fb-a66476e416ea_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!iqGz!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3c0fb8b3-f0a8-4c6c-98fb-a66476e416ea_1536x1024.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!iqGz!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3c0fb8b3-f0a8-4c6c-98fb-a66476e416ea_1536x1024.png" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/3c0fb8b3-f0a8-4c6c-98fb-a66476e416ea_1536x1024.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!iqGz!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3c0fb8b3-f0a8-4c6c-98fb-a66476e416ea_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!iqGz!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3c0fb8b3-f0a8-4c6c-98fb-a66476e416ea_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!iqGz!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3c0fb8b3-f0a8-4c6c-98fb-a66476e416ea_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!iqGz!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3c0fb8b3-f0a8-4c6c-98fb-a66476e416ea_1536x1024.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><strong>Analogy</strong></h3><p>Shipping containers replacing loose cargo.</p><p>The goods are identical, and the uniform box is what lets a crane stack, sort, and interleave them at speed.</p><h3><strong>Why it matters</strong></h3><p>Reading <code>HTTP/2</code> traffic needs different tools than reading <code>HTTP/1.1</code> traffic.</p><p>Dump frames with nghttp and decode full sessions in Wireshark with a TLS key log. Streams &amp; frames explain what the identifier inside each frame does.</p><h2><strong>11. Streams, Messages, and Frames</strong></h2><p>A connection carries many streams; each stream carries one request-response pair, and each message splits into frames tagged with a stream identifier.</p><p>i.e., three levels, one job.</p><p>The identifier lets frames from different requests share the wire and still reassemble correctly at the far end.</p><p>Numbering follows a simple rule:</p><p>Client-initiated streams take odd numbers, server-initiated streams take even ones, and identifiers always increase.</p><p>A stream costs only a few bytes per frame to label; this is what makes concurrency on a single connection practical.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!Tybx!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0915f87b-2129-467b-ba5b-07fa48f271c8_1536x1024.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Tybx!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0915f87b-2129-467b-ba5b-07fa48f271c8_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!Tybx!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0915f87b-2129-467b-ba5b-07fa48f271c8_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!Tybx!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0915f87b-2129-467b-ba5b-07fa48f271c8_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!Tybx!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0915f87b-2129-467b-ba5b-07fa48f271c8_1536x1024.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!Tybx!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0915f87b-2129-467b-ba5b-07fa48f271c8_1536x1024.png" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/0915f87b-2129-467b-ba5b-07fa48f271c8_1536x1024.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!Tybx!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0915f87b-2129-467b-ba5b-07fa48f271c8_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!Tybx!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0915f87b-2129-467b-ba5b-07fa48f271c8_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!Tybx!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0915f87b-2129-467b-ba5b-07fa48f271c8_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!Tybx!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0915f87b-2129-467b-ba5b-07fa48f271c8_1536x1024.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><strong>Analogy</strong></h3><p>One highway with many lanes.</p><p>Every car carries a destination label, so they travel together and still arrive at different lanes.</p><h3><strong>Why it matters</strong></h3><p>Default stream limits rarely match the concurrency your service actually sees.</p><p>Set <code>http2_max_concurrent_streams</code> in nginx and the matching limit in Envoy against your real p99. Flow control decides how much data those streams may send.</p><h4 style="text-align: center;"><strong>&#167;</strong></h4><h2><strong>Final Words</strong></h2><p><code>HTTP/1.1</code> made the web faster, but it left one major problem unsolved: responses still had to arrive in order on each connection.</p><p><code>HTTP/2</code> addressed this with multiplexing&#8230; But TCP&#8217;s head-of-line blocking remained.</p><p>In Part 2, we&#8217;ll explore how <code>HTTP/2</code> &amp; <code>HTTP/3</code> tackled these bottlenecks, and why the web needed another major upgrade.</p><div><hr></div><p>If you find this newsletter valuable, share it with a friend, and subscribe if you haven&#8217;t already. There are <a href="http://newsletter.systemdesign.one/subscribe?group=true">group discounts</a>, <a href="http://newsletter.systemdesign.one/subscribe?gift=true">gift options</a>, and <a href="https://newsletter.systemdesign.one/leaderboard">referral rewards</a> available.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe now&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://newsletter.systemdesign.one/subscribe?"><span>Subscribe now</span></a></p><div><hr></div><div class="captioned-image-container"><figure><a class="image-link image2" target="_blank" href="https://www.linkedin.com/in/nk-systemdesign-one/" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!68aQ!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe4eab3a0-fda8-4df8-b651-0f976b223e9a_320x320.png 424w, https://substackcdn.com/image/fetch/$s_!68aQ!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe4eab3a0-fda8-4df8-b651-0f976b223e9a_320x320.png 848w, https://substackcdn.com/image/fetch/$s_!68aQ!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe4eab3a0-fda8-4df8-b651-0f976b223e9a_320x320.png 1272w, https://substackcdn.com/image/fetch/$s_!68aQ!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe4eab3a0-fda8-4df8-b651-0f976b223e9a_320x320.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!68aQ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe4eab3a0-fda8-4df8-b651-0f976b223e9a_320x320.png" width="160" height="160" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/e4eab3a0-fda8-4df8-b651-0f976b223e9a_320x320.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:320,&quot;width&quot;:320,&quot;resizeWidth&quot;:160,&quot;bytes&quot;:121414,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:&quot;https://www.linkedin.com/in/nk-systemdesign-one/&quot;,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://newsletter.systemdesign.one/i/136161496?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe4eab3a0-fda8-4df8-b651-0f976b223e9a_320x320.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!68aQ!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe4eab3a0-fda8-4df8-b651-0f976b223e9a_320x320.png 424w, https://substackcdn.com/image/fetch/$s_!68aQ!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe4eab3a0-fda8-4df8-b651-0f976b223e9a_320x320.png 848w, https://substackcdn.com/image/fetch/$s_!68aQ!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe4eab3a0-fda8-4df8-b651-0f976b223e9a_320x320.png 1272w, https://substackcdn.com/image/fetch/$s_!68aQ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe4eab3a0-fda8-4df8-b651-0f976b223e9a_320x320.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div></div></div></a><figcaption class="image-caption"><strong>Find me elsewhere: <a href="https://www.linkedin.com/in/nk-systemdesign-one/">LinkedIn</a> | <a href="https://x.com/intent/follow?screen_name=systemdesignone">Twitter</a> | <a href="https://www.instagram.com/systemdesignone/">Instagram</a></strong></figcaption></figure></div><div><hr></div><p><strong>Want to reach 250K+ tech professionals at scale? </strong>&#128240;</p><p>If your company wants to reach 250K+ tech professionals, <a href="https://newsletter.systemdesign.one/p/sponsorship">advertise with me</a>.</p><div><hr></div><p>Thank you for supporting this newsletter.</p><p>You are now 250,001+ readers strong, very close to 251k. Let&#8217;s try to get 251k readers by 17 October. Consider sharing this letter with your friends and get rewards.</p><p>Y&#8217;all are the best.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!6oWl!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e739087-a910-4643-be36-997b6dd5b4af_800x500.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!6oWl!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e739087-a910-4643-be36-997b6dd5b4af_800x500.png 424w, https://substackcdn.com/image/fetch/$s_!6oWl!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e739087-a910-4643-be36-997b6dd5b4af_800x500.png 848w, https://substackcdn.com/image/fetch/$s_!6oWl!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e739087-a910-4643-be36-997b6dd5b4af_800x500.png 1272w, https://substackcdn.com/image/fetch/$s_!6oWl!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e739087-a910-4643-be36-997b6dd5b4af_800x500.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!6oWl!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e739087-a910-4643-be36-997b6dd5b4af_800x500.png" width="590" height="368.75" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/2e739087-a910-4643-be36-997b6dd5b4af_800x500.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:500,&quot;width&quot;:800,&quot;resizeWidth&quot;:590,&quot;bytes&quot;:87878,&quot;alt&quot;:&quot;system design newsletter&quot;,&quot;title&quot;:&quot;system design newsletter&quot;,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://newsletter.systemdesign.one/i/163380418?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e739087-a910-4643-be36-997b6dd5b4af_800x500.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="system design newsletter" title="system design newsletter" srcset="https://substackcdn.com/image/fetch/$s_!6oWl!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e739087-a910-4643-be36-997b6dd5b4af_800x500.png 424w, https://substackcdn.com/image/fetch/$s_!6oWl!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e739087-a910-4643-be36-997b6dd5b4af_800x500.png 848w, https://substackcdn.com/image/fetch/$s_!6oWl!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e739087-a910-4643-be36-997b6dd5b4af_800x500.png 1272w, https://substackcdn.com/image/fetch/$s_!6oWl!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e739087-a910-4643-be36-997b6dd5b4af_800x500.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/p/http-concepts-explained?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://newsletter.systemdesign.one/p/http-concepts-explained?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p><div><hr></div><h3>References</h3><ul><li><p>RFC 9110, <em>HTTP Semantics</em>:<a href="https://www.rfc-editor.org/rfc/rfc9110.html"> https://www.rfc-editor.org/rfc/rfc9110.html</a></p></li><li><p>RFC 9111, <em>HTTP Caching</em>:<a href="https://www.rfc-editor.org/rfc/rfc9111.html"> https://www.rfc-editor.org/rfc/rfc9111.html</a></p></li><li><p>RFC 9112, <em>HTTP/1.1</em>:<a href="https://www.rfc-editor.org/rfc/rfc9112.html"> https://www.rfc-editor.org/rfc/rfc9112.html</a></p></li><li><p>RFC 9113, <em>HTTP/2</em>:<a href="https://www.rfc-editor.org/rfc/rfc9113.html"> https://www.rfc-editor.org/rfc/rfc9113.html</a></p></li><li><p>RFC 7541, <em>HPACK: Header Compression for HTTP/2</em>:<a href="https://www.rfc-editor.org/rfc/rfc7541.html"> https://www.rfc-editor.org/rfc/rfc7541.html</a></p></li><li><p>RFC 9218, <em>Extensible Prioritization Scheme for HTTP</em>:<a href="https://www.rfc-editor.org/rfc/rfc9218.html"> https://www.rfc-editor.org/rfc/rfc9218.html</a></p></li><li><p>RFC 8297, <em>An HTTP Status Code for Indicating Hints</em>:<a href="https://www.rfc-editor.org/rfc/rfc8297.html"> https://www.rfc-editor.org/rfc/rfc8297.html</a></p></li><li><p>RFC 9000, <em>QUIC: A UDP-Based Multiplexed and Secure Transport</em>:<a href="https://www.rfc-editor.org/rfc/rfc9000.html"> https://www.rfc-editor.org/rfc/rfc9000.html</a></p></li><li><p>RFC 9114, <em>HTTP/3</em>:<a href="https://www.rfc-editor.org/rfc/rfc9114.html"> https://www.rfc-editor.org/rfc/rfc9114.html</a></p></li><li><p>RFC 9204, <em>QPACK: Field Compression for HTTP/3</em>:<a href="https://www.rfc-editor.org/rfc/rfc9204.html"> https://www.rfc-editor.org/rfc/rfc9204.html</a></p></li><li><p>HTTP Archive, <em>Web Almanac, HTTP chapter</em>:<a href="https://almanac.httparchive.org/en/2024/http"> https://almanac.httparchive.org/en/2024/http</a></p></li><li><p>Chrome for Developers, <em>Removing HTTP/2 Server Push from Chrome</em>:<a href="https://developer.chrome.com/blog/removing-push"> https://developer.chrome.com/blog/removing-push</a></p></li></ul><div class="footnote" data-component-name="FootnoteToDOM"><a id="footnote-1" href="#footnote-anchor-1" class="footnote-number" contenteditable="false" target="_self">1</a><div class="footnote-content"><p>HTTP/2 and HTTP/3 preserve the core HTTP semantics but use binary framing instead of HTTP/1.x&#8217;s text-based message format.</p></div></div><div class="footnote" data-component-name="FootnoteToDOM"><a id="footnote-2" href="#footnote-anchor-2" class="footnote-number" contenteditable="false" target="_self">2</a><div class="footnote-content"><ul><li><p>Connection churn: Opening and closing too many connections wastes resources.</p></li><li><p>Temporary ports: Each outgoing connection uses a temporary port. Too many short-lived connections can exhaust available ports.</p></li><li><p>Keep-alive timeout: Controls how long an idle connection stays open for reuse.</p></li><li><p>Load balancer: Its timeout settings should work with your server&#8217;s settings to avoid unexpected disconnections.</p></li><li><p>TCP slow start: New connections start with a limited sending rate. Reusing an existing connection avoids repeating this slow growth phase.</p></li></ul><p>Reuse connections to reduce connection setup overhead, avoid port exhaustion, and improve performance.</p></div></div><div class="footnote" data-component-name="FootnoteToDOM"><a id="footnote-3" href="#footnote-anchor-3" class="footnote-number" contenteditable="false" target="_self">3</a><div class="footnote-content"><p>Wireshark is a free, open-source network analysis tool that captures and examines network traffic in detail.</p></div></div><div class="footnote" data-component-name="FootnoteToDOM"><a id="footnote-4" href="#footnote-anchor-4" class="footnote-number" contenteditable="false" target="_self">4</a><div class="footnote-content"><ul><li><p>Head-of-line blocking: One slow request delays other requests waiting behind it.</p></li><li><p>HTTP/1.1: A single connection handles responses one at a time, in order.</p></li><li><p>Example: Five requests share one connection. The first request takes 5 seconds because of a slow database query. The other four must wait.</p></li><li><p>Persistent connections: Reuse the same connection to avoid repeated TCP handshakes, but they don&#8217;t solve head-of-line blocking.</p></li><li><p>Parallel connections: Browsers open multiple connections so a slow request on one connection doesn&#8217;t block requests on others.</p></li></ul><p>HTTP/2 addresses this with multiplexing!</p></div></div><div class="footnote" data-component-name="FootnoteToDOM"><a id="footnote-5" href="#footnote-anchor-5" class="footnote-number" contenteditable="false" target="_self">5</a><div class="footnote-content"><p>Google Lighthouse is a free, open-source tool that analyzes a website and identifies ways to improve its performance, accessibility, and SEO.</p></div></div><div class="footnote" data-component-name="FootnoteToDOM"><a id="footnote-6" href="#footnote-anchor-6" class="footnote-number" contenteditable="false" target="_self">6</a><div class="footnote-content"><p>The initial congestion window varies by implementation. Traditional loss-based TCP congestion control reduces its window after detecting congestion, while algorithms such as BBR use different mechanisms. The diagram illustrates a simplified loss-based congestion-control scenario.</p></div></div>]]></content:encoded></item><item><title><![CDATA[AI Engineering Playbook 🔥]]></title><description><![CDATA[Download My Playbook for Free]]></description><link>https://newsletter.systemdesign.one/p/ai-engineering-roadmap-for-beginners</link><guid isPermaLink="false">https://newsletter.systemdesign.one/p/ai-engineering-roadmap-for-beginners</guid><dc:creator><![CDATA[Neo Kim]]></dc:creator><pubDate>Tue, 13 Oct 2026 16:24:46 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/0a58b035-765c-4fb6-9e8e-93951f1d6aca_1280x720.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Once upon a time, I spent hours solving competitive math problems.</p><p>But I used to forget important formulas &amp; theorems quickly&#8230; and because of this, it became difficult to apply them when solving new problems.</p><p>Until one day, I started making notes on key ideas &amp; revisiting them regularly. This helped me absorb what I learned and become better than my past self.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://newsletter.systemdesign.one/subscribe" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!FpKu!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa4a6f4b2-a07a-4726-89cd-1861652f1423_1080x1350.png 424w, https://substackcdn.com/image/fetch/$s_!FpKu!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa4a6f4b2-a07a-4726-89cd-1861652f1423_1080x1350.png 848w, https://substackcdn.com/image/fetch/$s_!FpKu!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa4a6f4b2-a07a-4726-89cd-1861652f1423_1080x1350.png 1272w, https://substackcdn.com/image/fetch/$s_!FpKu!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa4a6f4b2-a07a-4726-89cd-1861652f1423_1080x1350.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!FpKu!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa4a6f4b2-a07a-4726-89cd-1861652f1423_1080x1350.png" width="254" height="317.5" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/a4a6f4b2-a07a-4726-89cd-1861652f1423_1080x1350.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1350,&quot;width&quot;:1080,&quot;resizeWidth&quot;:254,&quot;bytes&quot;:720001,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:&quot;https://newsletter.systemdesign.one/subscribe&quot;,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://newsletter.systemdesign.one/i/220181721?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa4a6f4b2-a07a-4726-89cd-1861652f1423_1080x1350.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!FpKu!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa4a6f4b2-a07a-4726-89cd-1861652f1423_1080x1350.png 424w, https://substackcdn.com/image/fetch/$s_!FpKu!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa4a6f4b2-a07a-4726-89cd-1861652f1423_1080x1350.png 848w, https://substackcdn.com/image/fetch/$s_!FpKu!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa4a6f4b2-a07a-4726-89cd-1861652f1423_1080x1350.png 1272w, https://substackcdn.com/image/fetch/$s_!FpKu!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa4a6f4b2-a07a-4726-89cd-1861652f1423_1080x1350.png 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Now I&#8217;ve created a playbook to help you learn AI engineering.</p><p>It gives you:</p><ul><li><p>Core concepts behind modern AI systems.</p></li><li><p>Condensed notes to understand the AI engineering stack.</p></li><li><p>Must-know techniques to build, deploy, debug &amp; scale AI apps.</p></li></ul><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption"><strong><mark data-color="#ffff00" style="background-color: rgb(255, 255, 0); color: rgb(0, 0, 0);">Join this newsletter right now and download my playbook for FREE:</mark></strong></p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><p style="text-align: center;">(Wait a few minutes for the welcome email &amp; check all folders.)</p><p>PS - This newsletter exists because of your support. Thank you.</p>]]></content:encoded></item><item><title><![CDATA[How to Scale an “AI App” to 1 Million Users in Production]]></title><description><![CDATA[#189: Part 7 - Generative AI Masterclass]]></description><link>https://newsletter.systemdesign.one/p/llm-inference-cost-optimization</link><guid isPermaLink="false">https://newsletter.systemdesign.one/p/llm-inference-cost-optimization</guid><dc:creator><![CDATA[Louis-François Bouchard]]></dc:creator><pubDate>Mon, 12 Oct 2026 11:30:16 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/a248d777-d0fd-4da0-bf04-c7573a9a721d_1280x720.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p></p><div class="captioned-image-container"><figure><a class="image-link image2" target="_blank" href="https://newsletter.systemdesign.one/subscribe?yearly=true" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!RKN7!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3689f342-2008-4ce6-b968-16461682508b_1280x300.png 424w, https://substackcdn.com/image/fetch/$s_!RKN7!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3689f342-2008-4ce6-b968-16461682508b_1280x300.png 848w, https://substackcdn.com/image/fetch/$s_!RKN7!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3689f342-2008-4ce6-b968-16461682508b_1280x300.png 1272w, https://substackcdn.com/image/fetch/$s_!RKN7!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3689f342-2008-4ce6-b968-16461682508b_1280x300.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!RKN7!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3689f342-2008-4ce6-b968-16461682508b_1280x300.png" width="1280" height="300" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/3689f342-2008-4ce6-b968-16461682508b_1280x300.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:300,&quot;width&quot;:1280,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:24224,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:&quot;&quot;,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:&quot;https://newsletter.systemdesign.one/subscribe?yearly=true&quot;,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://newsletter.systemdesign.one/i/192435842?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3689f342-2008-4ce6-b968-16461682508b_1280x300.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!RKN7!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3689f342-2008-4ce6-b968-16461682508b_1280x300.png 424w, https://substackcdn.com/image/fetch/$s_!RKN7!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3689f342-2008-4ce6-b968-16461682508b_1280x300.png 848w, https://substackcdn.com/image/fetch/$s_!RKN7!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3689f342-2008-4ce6-b968-16461682508b_1280x300.png 1272w, https://substackcdn.com/image/fetch/$s_!RKN7!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3689f342-2008-4ce6-b968-16461682508b_1280x300.png 1456w" sizes="100vw" fetchpriority="high"></picture><div></div></div></a></figure></div><ul><li><p><em><a href="https://newsletter.systemdesign.one/p/llm-inference-cost-optimization/?action=share">Share this letter</a> &amp; I&#8217;ll send you some rewards for the referrals.</em></p></li></ul><div><hr></div><p>Traditional software gets cheaper per user as it scales.</p><p>Yet AI software on an API does not!&#8230;</p><p>Every request runs a model, every model call has a per-token price, and the price stays &#8220;flat&#8221; whether ten people use it or a million do.</p><p>Self-hosting changes this, as GPU costs amortize across users.</p><p>But it introduces a new problem: <em>capacity planning, hardware management, and a minimum spend that has to be justified before the first user arrives.</em></p><p>So scaling an AI product forces cost decisions most software never encounters, and most engineering teams face them only after users have already arrived.</p><p>Building an AI system that produces correct, grounded, and safe responses is one challenge&#8230;Serving those responses reliably and affordably to a million users is another.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!bZpH!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F761f1bff-41a3-48f3-b068-1871624222ae_1698x805.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!bZpH!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F761f1bff-41a3-48f3-b068-1871624222ae_1698x805.png 424w, https://substackcdn.com/image/fetch/$s_!bZpH!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F761f1bff-41a3-48f3-b068-1871624222ae_1698x805.png 848w, https://substackcdn.com/image/fetch/$s_!bZpH!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F761f1bff-41a3-48f3-b068-1871624222ae_1698x805.png 1272w, https://substackcdn.com/image/fetch/$s_!bZpH!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F761f1bff-41a3-48f3-b068-1871624222ae_1698x805.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!bZpH!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F761f1bff-41a3-48f3-b068-1871624222ae_1698x805.png" width="1456" height="690" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/761f1bff-41a3-48f3-b068-1871624222ae_1698x805.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:690,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:1014671,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://newsletter.systemdesign.one/i/219568117?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F761f1bff-41a3-48f3-b068-1871624222ae_1698x805.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!bZpH!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F761f1bff-41a3-48f3-b068-1871624222ae_1698x805.png 424w, https://substackcdn.com/image/fetch/$s_!bZpH!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F761f1bff-41a3-48f3-b068-1871624222ae_1698x805.png 848w, https://substackcdn.com/image/fetch/$s_!bZpH!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F761f1bff-41a3-48f3-b068-1871624222ae_1698x805.png 1272w, https://substackcdn.com/image/fetch/$s_!bZpH!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F761f1bff-41a3-48f3-b068-1871624222ae_1698x805.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">At one million daily users, admission control protects capacity, cache hits bypass inference, misses route across serving tiers, and traced outcomes plus reviewed feedback drive eval-gated releases.</figcaption></figure></div><p>This newsletter shows you how to redesign a <a href="https://newsletter.systemdesign.one/p/ai-chat-assistant">chat assistant</a> for large-scale production, reduce inference costs &amp; build the infrastructure needed to handle millions of users.</p><h4 style="text-align: center;"><strong>&#167;</strong></h4><div class="callout-block" data-callout="true"><h2><strong><a href="https://link.svix.com/system-01">Make your agents realtime (Partner)</a></strong></h2><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://link.svix.com/system-01" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!v-ZP!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fde8a87d9-0b90-4a74-857c-be37bba398f2_2000x1047.png 424w, https://substackcdn.com/image/fetch/$s_!v-ZP!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fde8a87d9-0b90-4a74-857c-be37bba398f2_2000x1047.png 848w, https://substackcdn.com/image/fetch/$s_!v-ZP!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fde8a87d9-0b90-4a74-857c-be37bba398f2_2000x1047.png 1272w, https://substackcdn.com/image/fetch/$s_!v-ZP!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fde8a87d9-0b90-4a74-857c-be37bba398f2_2000x1047.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!v-ZP!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fde8a87d9-0b90-4a74-857c-be37bba398f2_2000x1047.png" width="1456" height="762" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/de8a87d9-0b90-4a74-857c-be37bba398f2_2000x1047.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:762,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:212365,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:&quot;&quot;,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:&quot;https://link.svix.com/system-01&quot;,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://newsletter.systemdesign.one/i/211217193?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fde8a87d9-0b90-4a74-857c-be37bba398f2_2000x1047.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!v-ZP!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fde8a87d9-0b90-4a74-857c-be37bba398f2_2000x1047.png 424w, https://substackcdn.com/image/fetch/$s_!v-ZP!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fde8a87d9-0b90-4a74-857c-be37bba398f2_2000x1047.png 848w, https://substackcdn.com/image/fetch/$s_!v-ZP!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fde8a87d9-0b90-4a74-857c-be37bba398f2_2000x1047.png 1272w, https://substackcdn.com/image/fetch/$s_!v-ZP!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fde8a87d9-0b90-4a74-857c-be37bba398f2_2000x1047.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Agents are killing your apps with compounding latency across workflows.</p><p>Your customers want their agent workflows to be efficient, so you&#8217;re losing deals if you don&#8217;t make them event-driven. Webhooks solve this issue, but implementing them is harder than it looks.</p><p>If you homebrewed your webhooks and are getting pinged at 4 in the morning for failed deliveries, this is for you.</p><p><strong><a href="https://link.svix.com/system-01">Svix</a></strong> handles everything: retries, idempotency, security, and compliance. Qualified startups get $12,000 in free credits.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://link.svix.com/system-01&quot;,&quot;text&quot;:&quot;Get started for free&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://link.svix.com/system-01"><span>Get started for free</span></a></p><p><em>(Thanks to <strong><a href="https://link.svix.com/system-01">Svix</a></strong> for partnering on this newsletter.)</em></p></div><h4 style="text-align: center;"><strong>&#167;</strong></h4><div class="captioned-image-container"><figure><a class="image-link image2" target="_blank" href="https://www.youtube.com/@whatsai?sub_confirmation=1" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Cu_b!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F14cb8599-59a4-4587-932c-fd09ddd0a595_1100x220.png 424w, https://substackcdn.com/image/fetch/$s_!Cu_b!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F14cb8599-59a4-4587-932c-fd09ddd0a595_1100x220.png 848w, https://substackcdn.com/image/fetch/$s_!Cu_b!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F14cb8599-59a4-4587-932c-fd09ddd0a595_1100x220.png 1272w, https://substackcdn.com/image/fetch/$s_!Cu_b!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F14cb8599-59a4-4587-932c-fd09ddd0a595_1100x220.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!Cu_b!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F14cb8599-59a4-4587-932c-fd09ddd0a595_1100x220.png" width="1100" height="220" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/14cb8599-59a4-4587-932c-fd09ddd0a595_1100x220.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:220,&quot;width&quot;:1100,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:&quot;https://www.youtube.com/@whatsai?sub_confirmation=1&quot;,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!Cu_b!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F14cb8599-59a4-4587-932c-fd09ddd0a595_1100x220.png 424w, https://substackcdn.com/image/fetch/$s_!Cu_b!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F14cb8599-59a4-4587-932c-fd09ddd0a595_1100x220.png 848w, https://substackcdn.com/image/fetch/$s_!Cu_b!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F14cb8599-59a4-4587-932c-fd09ddd0a595_1100x220.png 1272w, https://substackcdn.com/image/fetch/$s_!Cu_b!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F14cb8599-59a4-4587-932c-fd09ddd0a595_1100x220.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div></div></div></a></figure></div><p>I want to introduce <a href="https://louisbouchard.substack.com/">Louis-Fran&#231;ois Bouchard</a> as a guest author.</p><p>He&#8217;s a best-selling author (<a href="https://amzn.to/4bqYU9b">Building LLMs for Production</a>), the co-founder of <a href="https://academy.towardsai.net/?ref=1f9b29">Towards AI</a>, and the creator of the YouTube Channel, <a href="https://www.youtube.com/@whatsai?sub_confirmation=1">What&#8217;s AI</a>, where he helps people understand AI and learn how to apply it in the real world. Through his development work with clients, content, teaching, AI training programs, and the new Mentorship program at <a href="https://academy.towardsai.net/?ref=1f9b29">Towards AI Academy</a>, Louis focuses on making AI practical for builders, engineers, and curious learners alike.</p><p>At Towards AI, he and his team train AI engineers through courses and mentorship built for every stage, from beginner to advanced. That educational mission and the real-world experience building for his clients are exactly why I wanted him in this newsletter series.</p><h4 style="text-align: center;"><strong>&#167;</strong></h4><p><em><strong>Here&#8217;s what&#8217;s inside this newsletter:</strong></em></p><ul><li><p><strong>Your $193,000 AI bill could drop to $48,000.</strong> Three optimizations cut the projected monthly cost by 75% at 100,000 daily conversations. Most teams overlook where the biggest savings actually come from.</p></li><li><p><strong>Your smartest AI model could be your biggest money leak.</strong> Up to 70% of requests in our example go to a cheaper model, while the expensive one handles the rest. The surprising part is what happens to the total bill.</p></li><li><p><strong>At 100K users, renting AI could beat owning GPUs.</strong> A $48,000 API bill competes with a $49,560 self-hosted setup. At 1 million daily conversations, the economics flip dramatically.</p></li><li><p><strong>One GPU benchmark could wipe out $182,000 in monthly savings.</strong> Self-hosting looks 38% cheaper at scale, until lower throughput forces you to rent dozens of additional GPUs.</p></li><li><p><strong>Your AI could pass every test and still break in production.</strong> The hidden gaps between evaluations, guardrails, and real user behavior, and why production failures need to become tests before the next release.</p></li></ul><p></p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption"><span data-color="#3a7bf9" style="color: rgb(58, 123, 249);">Golden members get all letters like these!&#8230;</span></p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><p></p><h4 style="text-align: center;"><strong>&#167;</strong></h4><h2>Should You Plan for a Million Users?</h2><p>For most AI products, planning for a million users too early wastes time &amp; money.</p><p>An API key and a credit card are often enough to launch a prototype. You should focus on building a useful product before investing in complex infrastructure.</p><p>But as your user base grows, these challenges become HARD to ignore:</p><ul><li><p><strong>Cost increases with usage.</strong> A chat assistant costs approximately $0.064 per conversation using GPT-5.4. </p><ul><li><p>At 100 daily users, the monthly model bill reaches roughly $193.</p></li><li><p>At 100,000 daily users, it grows to $193,000.</p></li><li><p>At one million, it reaches $1.93 million.</p></li></ul><p>Model routing &amp; caching reduce the cost per conversation, but total spending still grows with usage.</p></li><li><p><strong>Latency increases under heavy traffic.</strong> An AI system that responds in 400 milliseconds with ten concurrent users could take two seconds with a thousand. As incoming requests approach the system&#8217;s serving capacity, queues grow &amp; response times increase.</p></li><li><p><strong>Small prompt changes become expensive.</strong> Adding 200 tokens to your system prompt across one million daily requests produces six billion additional input tokens per month. At GPT-5.4&#8217;s input price of $2.50 per million tokens, that adds $15,000 to your monthly bill before prompt caching.</p></li><li><p><strong>API rate limits restrict scalability.</strong> AI providers enforce limits on requests &amp; tokens per minute. As traffic grows, these limits could prevent your application from serving more users, even when your infrastructure has spare capacity.</p></li><li><p><strong>Unit economics determine profitability.</strong> If each conversation costs more than the revenue it generates, acquiring more users increases your losses. You need to measure cost per conversation against the revenue it produces.</p></li></ul><p>These challenges determine when your AI application needs more advanced production infrastructure.</p><p>So before moving beyond a managed API, evaluate these four constraints:</p><ul><li><p><strong>Unit economics:</strong> Does the cost per successful conversation support your target profit margin?</p></li><li><p><strong>Service level objectives (SLOs):</strong> Does the system meet your latency, availability, and reliability targets?</p></li><li><p><strong>Rate limits:</strong> Does your provider support your expected request and token volume?</p></li><li><p><strong>Data residency:</strong> Does the provider meet your requirements for where user data gets processed &amp; stored?</p></li></ul><p>As long as your API provider meets these requirements, staying with managed infrastructure is often the &#8220;simplest&#8221; choice.</p><p>The rest of this newsletter shows you how to estimate costs, identify scalability bottlenecks, and decide when more advanced infrastructure becomes necessary.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/p/llm-inference-cost-optimization?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://newsletter.systemdesign.one/p/llm-inference-cost-optimization?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p><h4 style="text-align: center;"><strong>&#167;</strong></h4><h2>Target System Design</h2><p>Our goal is to scale an AI chat assistant to <em>one million daily users</em> while keeping inference costs under control.</p><p>For simplicity, we assume each daily active user starts one conversation. Each conversation contains eight requests to the language model.</p><p>Let&#8217;s define the token usage for each request:</p><ul><li><p><strong>Static input:</strong> 1,024 tokens for the system prompt and a fixed few-shot examples. This meets OpenAI&#8217;s minimum prefix length for prompt caching.</p></li><li><p><strong>Dynamic input:</strong> 400 tokens for conversation history &amp; current user message.</p></li><li><p><strong>Output:</strong> 300 billed tokens generated by the model.</p></li></ul><p>Each request therefore processes 1,424 input tokens &amp; generates 300 output tokens.</p><p>With eight requests per conversation, the total becomes:</p><ul><li><p><strong>Input:</strong> 11,392 tokens per conversation.</p></li><li><p><strong>Output:</strong> 2,400 tokens per conversation.</p></li></ul><p>(These are illustrative assumptions. In production, you should replace them with actual token usage collected from your application.)</p><h3>System Requirements</h3><p>Our production system must meet seven requirements:</p><ul><li><p><strong>Cost:</strong> Keep model costs below a budget of $0.02 per conversation. Determine your actual budget from revenue per successful conversation and required gross margin.</p></li><li><p><strong>Latency:</strong> Define service level objectives (<strong>SLOs</strong>) for each workload, including time to first token (<strong>TTFT</strong>) for interactive chat &amp; completion deadlines for asynchronous jobs.</p></li><li><p><strong>Availability:</strong> Maintain 99.9% uptime, including during model &amp; prompt updates.</p></li><li><p><strong>Safe deployments:</strong> Support model changes with a reliable rollback mechanism.</p></li><li><p><strong>Observability:</strong> Record token usage, cost, latency &amp; model information for each request.</p></li><li><p><strong>Feedback:</strong> Collect user feedback &amp; incorporate reviewed examples into evaluation datasets.</p></li><li><p><strong>Cost forecasting:</strong> Estimate the financial impact of every change before deployment.</p></li></ul><h3>Our Approach</h3><p>We&#8217;ll make five major design decisions:</p><ol><li><p><strong>Cost engineering:</strong> Reduce model costs by 50% to 80% through model routing, semantic caching, and prompt optimization.</p></li><li><p><strong>Inference optimization:</strong> Explore quantization, GPU utilization, and serving frameworks to determine when self-hosting makes financial and technical sense.</p></li><li><p><strong>Production infrastructure:</strong> Handle concurrent traffic, control overload, and deploy model updates without disrupting users.</p></li><li><p><strong>Data flywheels:</strong> Use production feedback to improve evaluation datasets &amp; model quality over time.</p></li><li><p><strong>Capacity and cost estimation:</strong> Calculate infrastructure requirements &amp; monthly costs at 10,000, 100,000, and one million daily users.</p></li></ol><p>We&#8217;ll start with cost engineering because the budget per request influences every infrastructure decision that follows&#8230;</p><h4 style="text-align: center;"><strong>&#167;</strong></h4><h2>Part 1: Cost Engineering</h2><p>Let&#8217;s dive in!</p><h3>Three Types of Token Costs</h3><p>Every request to a language model creates costs based on token usage.</p><p>For this example, we&#8217;ll focus on three types:</p><ul><li><p><strong>Uncached input tokens:</strong> Tokens you send to the model, including the system prompt, conversation history, and user message, without a cached-input discount.</p></li><li><p><strong>Cached input tokens:</strong> Previously processed input prefixes that the provider reuses at a discounted rate.</p></li><li><p><strong>Output tokens:</strong> Tokens the model generates, including any billable reasoning tokens.</p></li></ul><p>Output tokens typically cost more than input tokens because generating tokens one at a time (decode) has lower throughput than processing input tokens in parallel (prefill).</p><p>Some providers charge separately for extra features:</p><ul><li><p><strong>Cache writes:</strong> Anthropic charges approximately 1.25 to 2 times the standard input rate to create a cache entry, depending on its duration. Cache reads cost approximately 10% of the standard input rate.</p></li><li><p><strong>Tool usage:</strong> Certain tools or external services introduce extra charges.</p></li><li><p><strong>Priority processing:</strong> Higher-priority inference tiers could have different pricing.</p></li><li><p><strong>Data residency:</strong> Regional processing requirements could affect the final bill.</p></li></ul><p>So check your provider&#8217;s billing structure before estimating production costs.</p><p>For the two OpenAI models below, output tokens cost <em>six times </em>as much as uncached input tokens. </p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!o0ez!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb0f68e48-7673-40e0-b078-a1c00f767bfd_1774x887.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!o0ez!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb0f68e48-7673-40e0-b078-a1c00f767bfd_1774x887.png 424w, https://substackcdn.com/image/fetch/$s_!o0ez!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb0f68e48-7673-40e0-b078-a1c00f767bfd_1774x887.png 848w, https://substackcdn.com/image/fetch/$s_!o0ez!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb0f68e48-7673-40e0-b078-a1c00f767bfd_1774x887.png 1272w, https://substackcdn.com/image/fetch/$s_!o0ez!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb0f68e48-7673-40e0-b078-a1c00f767bfd_1774x887.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!o0ez!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb0f68e48-7673-40e0-b078-a1c00f767bfd_1774x887.png" width="1456" height="728" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/b0f68e48-7673-40e0-b078-a1c00f767bfd_1774x887.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:728,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:1263650,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:&quot;&quot;,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://newsletter.systemdesign.one/i/219568117?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb0f68e48-7673-40e0-b078-a1c00f767bfd_1774x887.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!o0ez!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb0f68e48-7673-40e0-b078-a1c00f767bfd_1774x887.png 424w, https://substackcdn.com/image/fetch/$s_!o0ez!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb0f68e48-7673-40e0-b078-a1c00f767bfd_1774x887.png 848w, https://substackcdn.com/image/fetch/$s_!o0ez!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb0f68e48-7673-40e0-b078-a1c00f767bfd_1774x887.png 1272w, https://substackcdn.com/image/fetch/$s_!o0ez!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb0f68e48-7673-40e0-b078-a1c00f767bfd_1774x887.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Standard short-context token prices for GPT-5.4 and GPT-5.4-mini as of 2026.</figcaption></figure></div><p>These prices are a historical snapshot&#8230; Model availability &amp; pricing change over time, so verify the latest rates before making deployment decisions.</p><p>Next, let&#8217;s use these token prices to calculate the monthly cost of serving thousands/millions of conversations.</p><h3>Estimation Funnel</h3><p>Estimate costs before building the system.</p><p>Start with requests per day, estimate input, output, and cached-input tokens per request, then apply their respective rates. Multiply the daily cost by the number of days to get the monthly bill.</p><p>The arithmetic is straightforward&#8230;The challenge is accounting for every cost.</p><p>Retries &amp; failed calls increase usage.</p><p>Agent loops multiply model calls per task. Embeddings and reranking add separate costs. Image and video processing introduce media-specific charges.</p><p>Semantic caching adds infrastructure costs, while provider-side prompt caching may introduce cache-write charges. Storage, network egress, observability, and idle GPU capacity when self-hosting also contribute to the total.</p><p>Even a complete cost estimate does not tell you whether the system produces useful answers. Track cost per successful task alongside total spending to measure <em>efficiency</em>.</p><h3>Cost per Successful Task</h3><p>Cost per successful task measures total spending divided by the number of tasks completed successfully.</p><p>Track it alongside quality and latency to understand the actual cost of delivering useful results. For example, suppose each conversation costs $0.064 on average. If 10% of conversations fail, the cost per successful conversation increases to approximately $0.071, assuming the average cost already includes failed attempts and retries.</p><p>The same principle applies to self-hosted models. Estimate the required token throughput, translate it into GPU capacity &amp; calculate infrastructure costs.</p><p>H100 GPU rentals may cost around $2 to $6 per GPU-hour, depending on the provider, region, and rental terms. Newer Blackwell-class accelerators may have higher hourly rental prices.</p><p>But GPU rental prices alone do not determine inference cost:</p><p>Utilization, batching efficiency, memory requirements &amp; model quality also affect the cost per successful task.</p><h3>Faster or Cheaper, and When You Get Both</h3><p>Latency &amp; cost typically involve tradeoffs.</p><p>More powerful GPUs and lower batching levels reduce response times but increase cost per request.</p><p>Some optimizations improve both.</p><p>Smaller models require fewer resources, while shorter prompts reduce input processing. Prompt caching, quantization, and efficient attention kernels also reduce latency and cost, depending on the workload.</p><p>Streaming lets users see the response as it&#8217;s generated.</p><p>This improves perceived responsiveness without necessarily reducing total computation. Measure time to first token (<strong>TTFT</strong>), total response time &amp; stream reliability separately.</p><p>Batching improves GPU utilization by processing multiple requests together.</p><p>Yet excessive batching can increase tail latency (p99). So interactive applications need different batching configurations from background workloads.</p><p>For tasks that do not require immediate responses, consider a provider&#8217;s Batch API. These asynchronous endpoints process requests within a specified window at a lower price.</p><p>OpenAI&#8217;s Batch API offers a 50% discount on supported workloads with a 24-hour completion window. Other providers have different pricing and processing limits.</p><h3>Reasoning-Token Line Item</h3><p>Some reasoning models generate internal reasoning tokens that users never see.</p><p>These tokens count toward output usage &amp; get billed at output token rates.</p><p>This increases costs when switching models. Different models use different default reasoning-effort settings. If you leave the setting unspecified, a model upgrade could generate more reasoning tokens and increase the bill, even when the visible response stays the same.</p><p>Set reasoning effort explicitly where supported.</p><p>Track reasoning-token usage and total output costs for each request. Increase reasoning effort only when evaluations show enough improvement in answer quality to justify the extra cost.</p><h3>The Anchor: 100K Conversations a Day</h3><p>Picture an AI assistant handling 100,000 conversations per day, with eight model requests per conversation.</p><p>Each request uses 1,424 input tokens and 300 billed output tokens.</p><p>Over 30 days, the system processes 34.176 billion input tokens and 7.2 billion output tokens.</p><p>At GPT-5.4 rates of $2.50 per million input tokens and $15 per million output tokens, the monthly cost is:</p><ul><li><p>Input: $85,440</p></li><li><p>Output: $108,000</p></li><li><p>Total: $193,440 or $0.064 per conversation.</p></li></ul><p>Suppose 70% of requests use GPT-5.4-mini, which costs $0.75 per million input tokens and $4.50 per million output tokens.</p><p>Assuming identical token usage across both models, routing reduces the monthly bill to approximately $98,635, a 49% reduction.</p><p>Plus, adding semantic caching, provider-side prompt caching, and shorter prompts could further reduce costs. Under a scenario with a 30% semantic-cache hit rate and additional prompt optimizations, the monthly bill could approach $48,000.</p><p>That represents approximately 75% savings compared with the original system.</p><p>Actual savings depend on traffic patterns, cache hit rates, token usage &amp; model quality. Validate these assumptions against production measurements before relying on the estimate.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!35RP!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc88f9382-c58f-4b18-958c-cb594ea298eb_2048x994.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!35RP!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc88f9382-c58f-4b18-958c-cb594ea298eb_2048x994.png 424w, https://substackcdn.com/image/fetch/$s_!35RP!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc88f9382-c58f-4b18-958c-cb594ea298eb_2048x994.png 848w, https://substackcdn.com/image/fetch/$s_!35RP!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc88f9382-c58f-4b18-958c-cb594ea298eb_2048x994.png 1272w, https://substackcdn.com/image/fetch/$s_!35RP!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc88f9382-c58f-4b18-958c-cb594ea298eb_2048x994.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!35RP!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc88f9382-c58f-4b18-958c-cb594ea298eb_2048x994.png" width="1456" height="707" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/c88f9382-c58f-4b18-958c-cb594ea298eb_2048x994.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:707,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!35RP!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc88f9382-c58f-4b18-958c-cb594ea298eb_2048x994.png 424w, https://substackcdn.com/image/fetch/$s_!35RP!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc88f9382-c58f-4b18-958c-cb594ea298eb_2048x994.png 848w, https://substackcdn.com/image/fetch/$s_!35RP!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc88f9382-c58f-4b18-958c-cb594ea298eb_2048x994.png 1272w, https://substackcdn.com/image/fetch/$s_!35RP!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc88f9382-c58f-4b18-958c-cb594ea298eb_2048x994.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">At 100K daily conversations, modeled workload produces 34.176B input and 7.2B billed output tokens per month. Routing, caching, and prompt trimming reduce the model-token estimate from $193,000 to $48,000, excluding retry overhead and adjacent infrastructure costs.</figcaption></figure></div><h3>Model Routing</h3><p>Model routing sends simpler requests to a cheaper model &amp; complex requests to a more capable model.</p><p>A lightweight classifier or a smaller language model could decide which model should handle each request.</p><p>At scale, this significantly reduces inference costs. For example, routing 70% of requests to a cheaper model could lower the monthly bill. But the actual split depends on your workload and should come from production traffic &amp; evaluation results.</p><p>The biggest risk is routing a complex request to a model that cannot handle it reliably. The smaller model may produce a confident but incorrect answer, reducing quality without an obvious failure.</p><p>To prevent this, evaluate the route system against the original system using a representative test dataset. Measure task success, factual accuracy, relevance, and other application-specific metrics.</p><p>The router should reduce costs while keeping quality within predefined acceptance thresholds. Track routing accuracy and cost per successful task to verify that the savings are worthwhile.</p><h3>Semantic Caching</h3><p>Model routing reduces costs by sending simpler requests to cheaper models.</p><p>Semantic caching goes further by reusing previously generated answers for similar questions, thus avoiding repeated model inference.</p><p>Semantic caching uses embeddings to find previously answered questions with similar meanings. If a cached answer passes the similarity and validity checks, the system returns it without calling the language model.</p><p>GPTCache is one open-source library for implementing this approach.</p><p>Cache effectiveness depends on traffic patterns. Frequently repeated questions, such as FAQs, produce higher hit rates, while unique or context-dependent queries benefit less.</p><p>Yet two failure modes require attention:</p><p><strong>1. False cache hit</strong></p><p>Two questions may have similar embeddings but require different answers.</p><p>Reusing an incorrect or outdated response silently reduces answer quality. Similarity thresholds, time-to-live (<strong>TTL</strong>) policies, and extra validation help reduce this risk.</p><p><strong>2. Cross-user data leakage</strong></p><p>Cache entries must be isolated by tenant &amp; authorization scope.</p><p>Cache validity should account for relevant prompt, model, and retrieval-index versions, along with data freshness requirements. Authorization checks must remain enforced even when a cached answer exists.</p><p>Personalized responses should only be cached within properly isolated scopes.</p><p>Requests that perform actions or modify data must execute the required operations rather than replay a cached response.</p><p>Finally, semantic caching has its own costs, including embedding generation, vector search, and cache storage. So measure cache hit rate, false-hit rate, latency, and net cost savings to determine whether it improves the system.</p><h3>Prompt Optimization</h3><p>The system prompt gets included in every model request.</p><p>So its length directly affects input-token usage and cost. In our example, the static prefix accounts for nearly three-quarters of each request&#8217;s input tokens.</p><p>Shortening instructions and removing unnecessary few-shot examples reduce token usage. But the actual savings depend on how much of the prompt already benefits from caching. Plus, removing important instructions reduces answer quality, so evaluate every change before deployment.</p><p>Prompt caching introduces another consideration:</p><p>Providers reuse previously processed input when requests share an identical token prefix. To maximize cache hits, place reusable instructions and examples first, followed by dynamic content such as user questions and retrieved documents.</p><p>For OpenAI&#8217;s automatic prompt caching, eligible prompts must contain at least 1,024 tokens. Shortening a reusable prefix below this threshold eliminates its caching benefit and potentially increases costs, even when the prompt becomes shorter.</p><p>In our cost example, we retain the 1,024-token static prefix and optimize the dynamic portion instead. This preserves the opportunity for prompt-cache savings while reducing unnecessary input tokens.</p><p>The goal is to minimize total input cost without sacrificing response quality.</p><p>Compare cached-token usage, total spending, and evaluation results before deciding which prompt version performs best.</p><h3>When Smaller Models Are Good Enough</h3><p>Smaller models significantly reduce inference costs, but only if they maintain acceptable answer quality.</p><p>Evaluate each candidate model against a representative test dataset that includes common requests, complex queries, and edge cases. Measure task success, factual accuracy, relevance, and other application-specific metrics.</p><p>If the smaller model meets predefined quality thresholds, it passes the quality gate. Before deployment, it must also satisfy latency, reliability, security &amp; rollback requirements.</p><p>Use evaluation results to decide when a smaller model is sufficient rather than relying on model size or benchmark rankings alone.</p><p>Model routing reduces average inference costs by selecting cheaper models, while semantic caching avoids repeated model calls.</p><p>The next step is optimizing the inference stack itself to reduce the computational cost of generating each token&#8230;</p><h4 style="text-align: center;"><strong>&#167;</strong></h4><div class="callout-block" data-callout="true"><p style="text-align: center;"><em><strong><span data-color="#3a7bf9" style="color: rgb(58, 123, 249);">Reminder: this is a teaser of the subscriber-only newsletter series, exclusive to my golden members.</span></strong></em></p><p>When you upgrade, you&#8217;ll get:</p><ul><li><p><strong>Simple breakdown of real-world architectures</strong></p></li><li><p>Frameworks you can plug into your work or business</p></li><li><p><strong>Proven systems behind ChatGPT, Perplexity, and Copilot</strong></p></li></ul><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/subscribe&quot;,&quot;text&quot;:&quot;Unlock Full Access&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://newsletter.systemdesign.one/subscribe"><span>Unlock Full Access</span></a></p><p><em>Ready for the best part?</em></p></div><h4 style="text-align: center;"><strong>&#167;</strong></h4><h2>Part 2: Inference Optimization and Self-Hosting</h2>
      <p>
          <a href="https://newsletter.systemdesign.one/p/llm-inference-cost-optimization">
              Read more
          </a>
      </p>
   ]]></content:encoded></item><item><title><![CDATA[Claude Code on Autopilot: Schedule It Once and Walk Away]]></title><description><![CDATA[#188: Part 4 - Claude Code 24/7]]></description><link>https://newsletter.systemdesign.one/p/claude-code-website-monitoring-routine</link><guid isPermaLink="false">https://newsletter.systemdesign.one/p/claude-code-website-monitoring-routine</guid><dc:creator><![CDATA[Neo Kim]]></dc:creator><pubDate>Tue, 06 Oct 2026 14:08:32 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/0470935b-085f-4f9d-9b1e-0b455c0164bc_1280x720.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p></p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption"><span data-color="#3a7bf9" style="color: rgb(58, 123, 249);">Get my system design playbook for FREE on newsletter signup:</span></p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><ul><li><p><em><a href="https://newsletter.systemdesign.one/p/claude-code-website-monitoring-routine/?action=share">Share this letter</a> &amp; I&#8217;ll send you some rewards for the referrals.</em></p></li></ul><div><hr></div><p><span>Every task you give Claude runs while you sit there &amp; watch it.</span></p><p><span>But some work does not need your presence at all&#8230; You can check your website for availability issues daily, watch a repo for changes, or let Claude send you a weekly report via Slack.</span></p><p><span>Anthropic created two different features for this:</span></p><ul><li><p><span>Claude Code calls them </span><strong><span>Routines</span></strong><a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-1" href="#footnote-1" target="_self">1</a><span>,</span></p></li><li><p><span>Cowork</span><a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-2" href="#footnote-2" target="_self">2</a><span> calls them </span><strong><span>Scheduled Tasks</span></strong><span>.</span></p></li></ul><p><span>You could create them from scratch or simply by prompting Claude.</span></p><p><span>Some of these automations run on your laptop. To make sure they work, your </span>laptop<span> needs to stay awake. But you can also run your code in the cloud on Anthropic&#8217;s servers. This works even when your laptop is closed.</span></p><p><span>Let&#8217;s build two of them in this newsletter &amp; explore their limitations.</span></p><p><span>At the end, we&#8217;ll also look at guardrails, because we all know that conversations between us and AI can leak, and we should take steps to make them safer.</span></p><p><span>Let&#8217;s get started!</span></p><h4 style="text-align: center;"><strong>&#167;</strong></h4><div class="callout-block" data-callout="true"><h3><strong><a href="https://getunblocked.com/events/how-to-stop-babysitting-your-agents-oct-7/?utm_source=systemdesign&amp;utm_medium=email&amp;utm_campaign=primary">[Webinar] How to stop babysitting your agents (Partner)</a></strong></h3><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://getunblocked.com/events/how-to-stop-babysitting-your-agents-oct-7/?utm_source=systemdesign&amp;utm_medium=email&amp;utm_campaign=primary" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Nfs2!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F54df2fb6-ecbc-4828-ab3c-8386f50f0fba_1600x900.png 424w, https://substackcdn.com/image/fetch/$s_!Nfs2!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F54df2fb6-ecbc-4828-ab3c-8386f50f0fba_1600x900.png 848w, https://substackcdn.com/image/fetch/$s_!Nfs2!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F54df2fb6-ecbc-4828-ab3c-8386f50f0fba_1600x900.png 1272w, https://substackcdn.com/image/fetch/$s_!Nfs2!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F54df2fb6-ecbc-4828-ab3c-8386f50f0fba_1600x900.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!Nfs2!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F54df2fb6-ecbc-4828-ab3c-8386f50f0fba_1600x900.png" width="1456" height="819" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/54df2fb6-ecbc-4828-ab3c-8386f50f0fba_1600x900.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:819,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:1098056,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:&quot;https://getunblocked.com/events/how-to-stop-babysitting-your-agents-oct-7/?utm_source=systemdesign&amp;utm_medium=email&amp;utm_campaign=primary&quot;,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://newsletter.systemdesign.one/i/218241166?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F54df2fb6-ecbc-4828-ab3c-8386f50f0fba_1600x900.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!Nfs2!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F54df2fb6-ecbc-4828-ab3c-8386f50f0fba_1600x900.png 424w, https://substackcdn.com/image/fetch/$s_!Nfs2!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F54df2fb6-ecbc-4828-ab3c-8386f50f0fba_1600x900.png 848w, https://substackcdn.com/image/fetch/$s_!Nfs2!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F54df2fb6-ecbc-4828-ab3c-8386f50f0fba_1600x900.png 1272w, https://substackcdn.com/image/fetch/$s_!Nfs2!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F54df2fb6-ecbc-4828-ab3c-8386f50f0fba_1600x900.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Agents can generate code. Getting it right for your system, team conventions, and past decisions is the hard part. You end up wasting time and tokens in correction loops.</p><p>More MCPs give agents access to information but not understanding. The teams pulling ahead use a context layer to give agents exactly what they need.</p><p><strong><a href="https://getunblocked.com/events/how-to-stop-babysitting-your-agents-oct-7/?utm_source=systemdesign&amp;utm_medium=email&amp;utm_campaign=primary">Join live on Oct 7 (FREE)</a></strong> to see:</p><ul><li><p>Where teams get stuck on the AI maturity curve</p></li><li><p>How a context layer solves for quality, efficiency, and cost</p></li><li><p>Live demo: the same coding task with and without a context layer</p></li></ul><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://getunblocked.com/events/how-to-stop-babysitting-your-agents-oct-7/?utm_source=systemdesign&amp;utm_medium=email&amp;utm_campaign=primary&quot;,&quot;text&quot;:&quot;Register now&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://getunblocked.com/events/how-to-stop-babysitting-your-agents-oct-7/?utm_source=systemdesign&amp;utm_medium=email&amp;utm_campaign=primary"><span>Register now</span></a></p><p>(Thanks to <a href="https://getunblocked.com/events/how-to-stop-babysitting-your-agents-oct-7/?utm_source=systemdesign&amp;utm_medium=email&amp;utm_campaign=primary">Unblocked</a> for partnering on this newsletter.)</p></div><h4 style="text-align: center;"><strong>&#167;</strong></h4><p>I want to introduce <strong><a href="https://www.learnwithmeai.com/welcome">Gencay</a></strong> as the guest author.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://www.learnwithmeai.com/welcome" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!i2sU!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F79774ca3-8968-45f9-bb13-fb23bc404b4f_1442x668.png 424w, https://substackcdn.com/image/fetch/$s_!i2sU!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F79774ca3-8968-45f9-bb13-fb23bc404b4f_1442x668.png 848w, https://substackcdn.com/image/fetch/$s_!i2sU!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F79774ca3-8968-45f9-bb13-fb23bc404b4f_1442x668.png 1272w, https://substackcdn.com/image/fetch/$s_!i2sU!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F79774ca3-8968-45f9-bb13-fb23bc404b4f_1442x668.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!i2sU!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F79774ca3-8968-45f9-bb13-fb23bc404b4f_1442x668.png" width="1442" height="668" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/79774ca3-8968-45f9-bb13-fb23bc404b4f_1442x668.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:668,&quot;width&quot;:1442,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:&quot;&quot;,&quot;type&quot;:null,&quot;href&quot;:&quot;https://www.learnwithmeai.com/welcome&quot;,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!i2sU!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F79774ca3-8968-45f9-bb13-fb23bc404b4f_1442x668.png 424w, https://substackcdn.com/image/fetch/$s_!i2sU!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F79774ca3-8968-45f9-bb13-fb23bc404b4f_1442x668.png 848w, https://substackcdn.com/image/fetch/$s_!i2sU!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F79774ca3-8968-45f9-bb13-fb23bc404b4f_1442x668.png 1272w, https://substackcdn.com/image/fetch/$s_!i2sU!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F79774ca3-8968-45f9-bb13-fb23bc404b4f_1442x668.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>He has been building with AI since ChatGPT first shipped in 2022.</p><p>For years, he built for clients on Upwork: 7,300+ hours of it.</p><p>Now he builds in the open and shares every piece of it on <strong><a href="https://www.learnwithmeai.com/welcome">LearnAIWithMe</a></strong>, every Monday, Wednesday &amp; Friday.</p><p>Not just theory for its own sake. One real build at a time, from idea to running system.</p><h4 style="text-align: center;"><strong>&#167;</strong></h4><p><em>Here&#8217;s what you&#8217;ll find inside this newsletter:</em></p><ul><li><p><strong>What a schedule changes about an agent.</strong> Routines in Claude Code, Scheduled Tasks in Cowork, and why a job that runs without you is a different kind of worker than one you watch.</p></li><li><p><strong>Where your automation actually runs.</strong> Local routines that need your computer awake, cloud routines that keep running with the laptop closed, and the triggers (schedule, Git event, API) and delivery options (push, email, Slack) the cloud version adds.</p></li><li><p><strong>Headless mode, the engine underneath.</strong> <code>claude -p</code> runs Claude with no chat window at all, and every scheduled job is built on it.</p></li><li><p><strong>Two routines you can build, and how to reach them from anywhere.</strong> A morning site check and a GitHub watcher, each created from a single prompt, then dispatched from your phone with reports landing in Slack or a custom Telegram bot.</p></li><li><p><strong>How to trust a job you were asleep for.</strong> Guardrails on permissions, keys, and connectors, a validation harness where tests and hooks check the agent&#8217;s work, and reports worth reading.</p></li></ul><p></p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption"><span data-color="#3a7bf9" style="color: rgb(58, 123, 249);">Golden members get all letters like these!&#8230;</span></p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><p></p><h4 style="text-align: center;"><strong>&#167;</strong></h4><h2><span>What routines actually are</span></h2><p><span>Everything Claude does normally starts with you!</span></p><p><span>You type, it works, and when you leave the desk, the work stops&#8230;</span></p><p><span>A schedule </span><em><span>breaks</span></em><span> this link... The agent stops being an extension of your hands and becomes a worker in your absence, since the job now runs whether you are there/not. And that changes what you use it for, because the tasks worth scheduling are the ones you kept forgetting/postponing.</span></p><p><span>Routines let Claude work on a schedule you define.</span></p><p><span>They have two different modes:</span></p><ul><li><p><span>Local</span></p></li><li><p><span>Cloud</span></p></li></ul><p><span>(We&#8217;ll cover both in the next sections.)</span></p><p><span>Let&#8217;s first create a routine manually&#8230;</span></p><h3><span>Manual</span></h3><p><span>To create a routine, open your Claude app, click on &#8220;Routines&#8221; and &#8220;New routine&#8221;.</span></p><p><span>You will see two options:</span></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!uVs2!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb65cb926-88ab-4118-84db-0974f5ac1a5f_2048x1121.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!uVs2!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb65cb926-88ab-4118-84db-0974f5ac1a5f_2048x1121.png 424w, https://substackcdn.com/image/fetch/$s_!uVs2!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb65cb926-88ab-4118-84db-0974f5ac1a5f_2048x1121.png 848w, https://substackcdn.com/image/fetch/$s_!uVs2!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb65cb926-88ab-4118-84db-0974f5ac1a5f_2048x1121.png 1272w, https://substackcdn.com/image/fetch/$s_!uVs2!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb65cb926-88ab-4118-84db-0974f5ac1a5f_2048x1121.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!uVs2!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb65cb926-88ab-4118-84db-0974f5ac1a5f_2048x1121.png" width="1456" height="797" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/b65cb926-88ab-4118-84db-0974f5ac1a5f_2048x1121.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:797,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!uVs2!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb65cb926-88ab-4118-84db-0974f5ac1a5f_2048x1121.png 424w, https://substackcdn.com/image/fetch/$s_!uVs2!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb65cb926-88ab-4118-84db-0974f5ac1a5f_2048x1121.png 848w, https://substackcdn.com/image/fetch/$s_!uVs2!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb65cb926-88ab-4118-84db-0974f5ac1a5f_2048x1121.png 1272w, https://substackcdn.com/image/fetch/$s_!uVs2!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb65cb926-88ab-4118-84db-0974f5ac1a5f_2048x1121.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h4><span>Local</span></h4><p><span>Click on </span><strong><span>Local</span></strong><span>.</span></p><p><span>You can choose the model, set permissions &amp; schedule when you want the routine to run.</span></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!D5Hn!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1cc7915d-c5ea-420e-8eb6-32d5f3e2ca3e_1716x1310.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!D5Hn!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1cc7915d-c5ea-420e-8eb6-32d5f3e2ca3e_1716x1310.png 424w, https://substackcdn.com/image/fetch/$s_!D5Hn!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1cc7915d-c5ea-420e-8eb6-32d5f3e2ca3e_1716x1310.png 848w, https://substackcdn.com/image/fetch/$s_!D5Hn!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1cc7915d-c5ea-420e-8eb6-32d5f3e2ca3e_1716x1310.png 1272w, https://substackcdn.com/image/fetch/$s_!D5Hn!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1cc7915d-c5ea-420e-8eb6-32d5f3e2ca3e_1716x1310.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!D5Hn!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1cc7915d-c5ea-420e-8eb6-32d5f3e2ca3e_1716x1310.png" width="1456" height="1112" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/1cc7915d-c5ea-420e-8eb6-32d5f3e2ca3e_1716x1310.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1112,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!D5Hn!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1cc7915d-c5ea-420e-8eb6-32d5f3e2ca3e_1716x1310.png 424w, https://substackcdn.com/image/fetch/$s_!D5Hn!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1cc7915d-c5ea-420e-8eb6-32d5f3e2ca3e_1716x1310.png 848w, https://substackcdn.com/image/fetch/$s_!D5Hn!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1cc7915d-c5ea-420e-8eb6-32d5f3e2ca3e_1716x1310.png 1272w, https://substackcdn.com/image/fetch/$s_!D5Hn!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1cc7915d-c5ea-420e-8eb6-32d5f3e2ca3e_1716x1310.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h4><span>Cloud</span></h4><p><span>Now let&#8217;s check the cloud version.</span></p><p><span>This version lets you customize a lot more.</span></p><p><span>First, you can select a trigger: </span><strong><span>Schedule, Git Event, or API.</span></strong></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!ZJy1!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F10f8aaee-58c4-4621-9e16-bc760d33dca0_1528x1638.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!ZJy1!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F10f8aaee-58c4-4621-9e16-bc760d33dca0_1528x1638.png 424w, https://substackcdn.com/image/fetch/$s_!ZJy1!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F10f8aaee-58c4-4621-9e16-bc760d33dca0_1528x1638.png 848w, https://substackcdn.com/image/fetch/$s_!ZJy1!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F10f8aaee-58c4-4621-9e16-bc760d33dca0_1528x1638.png 1272w, https://substackcdn.com/image/fetch/$s_!ZJy1!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F10f8aaee-58c4-4621-9e16-bc760d33dca0_1528x1638.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!ZJy1!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F10f8aaee-58c4-4621-9e16-bc760d33dca0_1528x1638.png" width="1456" height="1561" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/10f8aaee-58c4-4621-9e16-bc760d33dca0_1528x1638.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1561,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!ZJy1!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F10f8aaee-58c4-4621-9e16-bc760d33dca0_1528x1638.png 424w, https://substackcdn.com/image/fetch/$s_!ZJy1!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F10f8aaee-58c4-4621-9e16-bc760d33dca0_1528x1638.png 848w, https://substackcdn.com/image/fetch/$s_!ZJy1!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F10f8aaee-58c4-4621-9e16-bc760d33dca0_1528x1638.png 1272w, https://substackcdn.com/image/fetch/$s_!ZJy1!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F10f8aaee-58c4-4621-9e16-bc760d33dca0_1528x1638.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><span>Next, click </span><strong><span>Notifications</span></strong><span> to choose how the report is delivered to you.</span></p><p><span>The options are </span><strong><span>push notifications in the app, email, or Slack.</span></strong></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!bG0U!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2935fd72-1f15-44e1-b314-28768115633a_1506x660.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!bG0U!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2935fd72-1f15-44e1-b314-28768115633a_1506x660.png 424w, https://substackcdn.com/image/fetch/$s_!bG0U!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2935fd72-1f15-44e1-b314-28768115633a_1506x660.png 848w, https://substackcdn.com/image/fetch/$s_!bG0U!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2935fd72-1f15-44e1-b314-28768115633a_1506x660.png 1272w, https://substackcdn.com/image/fetch/$s_!bG0U!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2935fd72-1f15-44e1-b314-28768115633a_1506x660.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!bG0U!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2935fd72-1f15-44e1-b314-28768115633a_1506x660.png" width="1456" height="638" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/2935fd72-1f15-44e1-b314-28768115633a_1506x660.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:638,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!bG0U!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2935fd72-1f15-44e1-b314-28768115633a_1506x660.png 424w, https://substackcdn.com/image/fetch/$s_!bG0U!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2935fd72-1f15-44e1-b314-28768115633a_1506x660.png 848w, https://substackcdn.com/image/fetch/$s_!bG0U!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2935fd72-1f15-44e1-b314-28768115633a_1506x660.png 1272w, https://substackcdn.com/image/fetch/$s_!bG0U!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2935fd72-1f15-44e1-b314-28768115633a_1506x660.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><span>You can also enable &#8220;Auto-fix&#8221; for pull requests.</span></p><p><span>This way Claude watches CI and review comments and pushes fixes automatically.</span></p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!hGV0!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9f2f90db-752c-4738-9151-0d36d06e79c4_1474x340.png 424w, https://substackcdn.com/image/fetch/$s_!hGV0!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9f2f90db-752c-4738-9151-0d36d06e79c4_1474x340.png 848w, https://substackcdn.com/image/fetch/$s_!hGV0!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9f2f90db-752c-4738-9151-0d36d06e79c4_1474x340.png 1272w, https://substackcdn.com/image/fetch/$s_!hGV0!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9f2f90db-752c-4738-9151-0d36d06e79c4_1474x340.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!hGV0!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9f2f90db-752c-4738-9151-0d36d06e79c4_1474x340.png" width="1456" height="336" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/9f2f90db-752c-4738-9151-0d36d06e79c4_1474x340.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:336,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!hGV0!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9f2f90db-752c-4738-9151-0d36d06e79c4_1474x340.png 424w, https://substackcdn.com/image/fetch/$s_!hGV0!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9f2f90db-752c-4738-9151-0d36d06e79c4_1474x340.png 848w, https://substackcdn.com/image/fetch/$s_!hGV0!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9f2f90db-752c-4738-9151-0d36d06e79c4_1474x340.png 1272w, https://substackcdn.com/image/fetch/$s_!hGV0!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9f2f90db-752c-4738-9151-0d36d06e79c4_1474x340.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div></div></div></div></figure></div><h3><span>Create Routine by prompting</span></h3><p><span>Let&#8217;s create one routine by </span><em><span>prompting</span></em><span>.</span></p><p><span>Here is a site I run&#8230; I change it often, so pages break sometimes, and I check them by hand.</span></p><p><span>Follow along with any site you run, since the prompt only needs a URL.</span></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!blI-!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c732049-2cce-4279-9bbe-5465671486ad_2048x1040.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!blI-!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c732049-2cce-4279-9bbe-5465671486ad_2048x1040.png 424w, https://substackcdn.com/image/fetch/$s_!blI-!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c732049-2cce-4279-9bbe-5465671486ad_2048x1040.png 848w, https://substackcdn.com/image/fetch/$s_!blI-!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c732049-2cce-4279-9bbe-5465671486ad_2048x1040.png 1272w, https://substackcdn.com/image/fetch/$s_!blI-!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c732049-2cce-4279-9bbe-5465671486ad_2048x1040.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!blI-!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c732049-2cce-4279-9bbe-5465671486ad_2048x1040.png" width="1456" height="739" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/7c732049-2cce-4279-9bbe-5465671486ad_2048x1040.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:739,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!blI-!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c732049-2cce-4279-9bbe-5465671486ad_2048x1040.png 424w, https://substackcdn.com/image/fetch/$s_!blI-!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c732049-2cce-4279-9bbe-5465671486ad_2048x1040.png 848w, https://substackcdn.com/image/fetch/$s_!blI-!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c732049-2cce-4279-9bbe-5465671486ad_2048x1040.png 1272w, https://substackcdn.com/image/fetch/$s_!blI-!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c732049-2cce-4279-9bbe-5465671486ad_2048x1040.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><span>Let&#8217;s hand this task off to Claude Code to save </span><em><span>time</span></em><span>.</span></p><p><span>Here is the prompt:</span></p><div class="highlighted_code_block" data-attrs="{&quot;language&quot;:&quot;plaintext&quot;,&quot;nodeId&quot;:&quot;5834099d-ca31-4e20-bf48-526b3ba82569&quot;}" data-component-name="HighlightedCodeBlockToDOM"><pre class="shiki"><code class="language-plaintext">Create a routine that runs every morning at 8.
It opens my AI Labs and AI Academy pages in Chrome,
checks for broken links, layout problems, and console errors.

If everything is fine, report one line saying clean.
Otherwise, list only the real issues to Slack.</code></pre></div><p><span>After pasting this prompt, Claude asks for my approval.</span></p><p><span>The first approval is to list the scheduled tasks.</span></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!mer0!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe0d47c21-ad97-4356-9f02-723ebba1e572_1652x558.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!mer0!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe0d47c21-ad97-4356-9f02-723ebba1e572_1652x558.png 424w, https://substackcdn.com/image/fetch/$s_!mer0!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe0d47c21-ad97-4356-9f02-723ebba1e572_1652x558.png 848w, https://substackcdn.com/image/fetch/$s_!mer0!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe0d47c21-ad97-4356-9f02-723ebba1e572_1652x558.png 1272w, https://substackcdn.com/image/fetch/$s_!mer0!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe0d47c21-ad97-4356-9f02-723ebba1e572_1652x558.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!mer0!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe0d47c21-ad97-4356-9f02-723ebba1e572_1652x558.png" width="1456" height="492" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/e0d47c21-ad97-4356-9f02-723ebba1e572_1652x558.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:492,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!mer0!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe0d47c21-ad97-4356-9f02-723ebba1e572_1652x558.png 424w, https://substackcdn.com/image/fetch/$s_!mer0!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe0d47c21-ad97-4356-9f02-723ebba1e572_1652x558.png 848w, https://substackcdn.com/image/fetch/$s_!mer0!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe0d47c21-ad97-4356-9f02-723ebba1e572_1652x558.png 1272w, https://substackcdn.com/image/fetch/$s_!mer0!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe0d47c21-ad97-4356-9f02-723ebba1e572_1652x558.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><span>Next, it asks a couple of questions&#8230;</span></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!LDL7!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F629e69f8-d35f-431f-9477-f40b8e2e8e7a_1612x808.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!LDL7!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F629e69f8-d35f-431f-9477-f40b8e2e8e7a_1612x808.png 424w, https://substackcdn.com/image/fetch/$s_!LDL7!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F629e69f8-d35f-431f-9477-f40b8e2e8e7a_1612x808.png 848w, https://substackcdn.com/image/fetch/$s_!LDL7!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F629e69f8-d35f-431f-9477-f40b8e2e8e7a_1612x808.png 1272w, https://substackcdn.com/image/fetch/$s_!LDL7!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F629e69f8-d35f-431f-9477-f40b8e2e8e7a_1612x808.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!LDL7!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F629e69f8-d35f-431f-9477-f40b8e2e8e7a_1612x808.png" width="1456" height="730" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/629e69f8-d35f-431f-9477-f40b8e2e8e7a_1612x808.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:730,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!LDL7!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F629e69f8-d35f-431f-9477-f40b8e2e8e7a_1612x808.png 424w, https://substackcdn.com/image/fetch/$s_!LDL7!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F629e69f8-d35f-431f-9477-f40b8e2e8e7a_1612x808.png 848w, https://substackcdn.com/image/fetch/$s_!LDL7!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F629e69f8-d35f-431f-9477-f40b8e2e8e7a_1612x808.png 1272w, https://substackcdn.com/image/fetch/$s_!LDL7!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F629e69f8-d35f-431f-9477-f40b8e2e8e7a_1612x808.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><span>After my answers, it started working on it&#8230;</span></p><p><span>Then it asked the final question before creating this routine.</span></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!uPgR!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F86838e2f-e9bf-4df4-95a4-3ab5188d7ad5_1604x886.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!uPgR!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F86838e2f-e9bf-4df4-95a4-3ab5188d7ad5_1604x886.png 424w, https://substackcdn.com/image/fetch/$s_!uPgR!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F86838e2f-e9bf-4df4-95a4-3ab5188d7ad5_1604x886.png 848w, https://substackcdn.com/image/fetch/$s_!uPgR!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F86838e2f-e9bf-4df4-95a4-3ab5188d7ad5_1604x886.png 1272w, https://substackcdn.com/image/fetch/$s_!uPgR!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F86838e2f-e9bf-4df4-95a4-3ab5188d7ad5_1604x886.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!uPgR!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F86838e2f-e9bf-4df4-95a4-3ab5188d7ad5_1604x886.png" width="1456" height="804" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/86838e2f-e9bf-4df4-95a4-3ab5188d7ad5_1604x886.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:804,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!uPgR!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F86838e2f-e9bf-4df4-95a4-3ab5188d7ad5_1604x886.png 424w, https://substackcdn.com/image/fetch/$s_!uPgR!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F86838e2f-e9bf-4df4-95a4-3ab5188d7ad5_1604x886.png 848w, https://substackcdn.com/image/fetch/$s_!uPgR!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F86838e2f-e9bf-4df4-95a4-3ab5188d7ad5_1604x886.png 1272w, https://substackcdn.com/image/fetch/$s_!uPgR!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F86838e2f-e9bf-4df4-95a4-3ab5188d7ad5_1604x886.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><span>I allowed &amp; it created the routine.</span></p><p><span>When I went back to the routine tab, I saw mine.</span></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!RY5l!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4157e04f-d2d3-4203-873a-f8b030187ce6_2048x927.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!RY5l!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4157e04f-d2d3-4203-873a-f8b030187ce6_2048x927.png 424w, https://substackcdn.com/image/fetch/$s_!RY5l!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4157e04f-d2d3-4203-873a-f8b030187ce6_2048x927.png 848w, https://substackcdn.com/image/fetch/$s_!RY5l!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4157e04f-d2d3-4203-873a-f8b030187ce6_2048x927.png 1272w, https://substackcdn.com/image/fetch/$s_!RY5l!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4157e04f-d2d3-4203-873a-f8b030187ce6_2048x927.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!RY5l!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4157e04f-d2d3-4203-873a-f8b030187ce6_2048x927.png" width="1456" height="659" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/4157e04f-d2d3-4203-873a-f8b030187ce6_2048x927.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:659,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!RY5l!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4157e04f-d2d3-4203-873a-f8b030187ce6_2048x927.png 424w, https://substackcdn.com/image/fetch/$s_!RY5l!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4157e04f-d2d3-4203-873a-f8b030187ce6_2048x927.png 848w, https://substackcdn.com/image/fetch/$s_!RY5l!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4157e04f-d2d3-4203-873a-f8b030187ce6_2048x927.png 1272w, https://substackcdn.com/image/fetch/$s_!RY5l!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4157e04f-d2d3-4203-873a-f8b030187ce6_2048x927.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><span>Testing The Routine</span></h3><p><span>I clicked on the routine and &#8220;Run now&#8221;.</span></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!5a0y!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F56c9dbf7-1f5e-4f4b-bd34-3789e24c069b_2048x962.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!5a0y!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F56c9dbf7-1f5e-4f4b-bd34-3789e24c069b_2048x962.png 424w, https://substackcdn.com/image/fetch/$s_!5a0y!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F56c9dbf7-1f5e-4f4b-bd34-3789e24c069b_2048x962.png 848w, https://substackcdn.com/image/fetch/$s_!5a0y!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F56c9dbf7-1f5e-4f4b-bd34-3789e24c069b_2048x962.png 1272w, https://substackcdn.com/image/fetch/$s_!5a0y!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F56c9dbf7-1f5e-4f4b-bd34-3789e24c069b_2048x962.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!5a0y!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F56c9dbf7-1f5e-4f4b-bd34-3789e24c069b_2048x962.png" width="1456" height="684" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/56c9dbf7-1f5e-4f4b-bd34-3789e24c069b_2048x962.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:684,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!5a0y!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F56c9dbf7-1f5e-4f4b-bd34-3789e24c069b_2048x962.png 424w, https://substackcdn.com/image/fetch/$s_!5a0y!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F56c9dbf7-1f5e-4f4b-bd34-3789e24c069b_2048x962.png 848w, https://substackcdn.com/image/fetch/$s_!5a0y!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F56c9dbf7-1f5e-4f4b-bd34-3789e24c069b_2048x962.png 1272w, https://substackcdn.com/image/fetch/$s_!5a0y!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F56c9dbf7-1f5e-4f4b-bd34-3789e24c069b_2048x962.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><span>And I saw this notification:</span></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!Jgh4!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F96ae069f-b032-4bfb-a5c1-5b574943e2e7_2048x1281.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Jgh4!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F96ae069f-b032-4bfb-a5c1-5b574943e2e7_2048x1281.png 424w, https://substackcdn.com/image/fetch/$s_!Jgh4!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F96ae069f-b032-4bfb-a5c1-5b574943e2e7_2048x1281.png 848w, https://substackcdn.com/image/fetch/$s_!Jgh4!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F96ae069f-b032-4bfb-a5c1-5b574943e2e7_2048x1281.png 1272w, https://substackcdn.com/image/fetch/$s_!Jgh4!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F96ae069f-b032-4bfb-a5c1-5b574943e2e7_2048x1281.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!Jgh4!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F96ae069f-b032-4bfb-a5c1-5b574943e2e7_2048x1281.png" width="1456" height="911" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/96ae069f-b032-4bfb-a5c1-5b574943e2e7_2048x1281.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:911,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!Jgh4!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F96ae069f-b032-4bfb-a5c1-5b574943e2e7_2048x1281.png 424w, https://substackcdn.com/image/fetch/$s_!Jgh4!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F96ae069f-b032-4bfb-a5c1-5b574943e2e7_2048x1281.png 848w, https://substackcdn.com/image/fetch/$s_!Jgh4!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F96ae069f-b032-4bfb-a5c1-5b574943e2e7_2048x1281.png 1272w, https://substackcdn.com/image/fetch/$s_!Jgh4!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F96ae069f-b032-4bfb-a5c1-5b574943e2e7_2048x1281.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><span>To see more details, you need to click on History. (I clicked it twice by mistake.)</span></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!8fg-!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdc2b1280-218e-4875-8339-e853b9652788_2048x1140.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!8fg-!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdc2b1280-218e-4875-8339-e853b9652788_2048x1140.png 424w, https://substackcdn.com/image/fetch/$s_!8fg-!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdc2b1280-218e-4875-8339-e853b9652788_2048x1140.png 848w, https://substackcdn.com/image/fetch/$s_!8fg-!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdc2b1280-218e-4875-8339-e853b9652788_2048x1140.png 1272w, https://substackcdn.com/image/fetch/$s_!8fg-!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdc2b1280-218e-4875-8339-e853b9652788_2048x1140.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!8fg-!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdc2b1280-218e-4875-8339-e853b9652788_2048x1140.png" width="1456" height="810" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/dc2b1280-218e-4875-8339-e853b9652788_2048x1140.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:810,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!8fg-!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdc2b1280-218e-4875-8339-e853b9652788_2048x1140.png 424w, https://substackcdn.com/image/fetch/$s_!8fg-!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdc2b1280-218e-4875-8339-e853b9652788_2048x1140.png 848w, https://substackcdn.com/image/fetch/$s_!8fg-!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdc2b1280-218e-4875-8339-e853b9652788_2048x1140.png 1272w, https://substackcdn.com/image/fetch/$s_!8fg-!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdc2b1280-218e-4875-8339-e853b9652788_2048x1140.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><span>And Claude took control of my laptop &amp; started testing&#8230;</span></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!hWXs!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc282e643-055b-4c18-9505-5b96b0e8ea60_2048x1179.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!hWXs!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc282e643-055b-4c18-9505-5b96b0e8ea60_2048x1179.png 424w, https://substackcdn.com/image/fetch/$s_!hWXs!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc282e643-055b-4c18-9505-5b96b0e8ea60_2048x1179.png 848w, https://substackcdn.com/image/fetch/$s_!hWXs!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc282e643-055b-4c18-9505-5b96b0e8ea60_2048x1179.png 1272w, https://substackcdn.com/image/fetch/$s_!hWXs!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc282e643-055b-4c18-9505-5b96b0e8ea60_2048x1179.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!hWXs!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc282e643-055b-4c18-9505-5b96b0e8ea60_2048x1179.png" width="1456" height="838" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/c282e643-055b-4c18-9505-5b96b0e8ea60_2048x1179.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:838,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!hWXs!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc282e643-055b-4c18-9505-5b96b0e8ea60_2048x1179.png 424w, https://substackcdn.com/image/fetch/$s_!hWXs!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc282e643-055b-4c18-9505-5b96b0e8ea60_2048x1179.png 848w, https://substackcdn.com/image/fetch/$s_!hWXs!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc282e643-055b-4c18-9505-5b96b0e8ea60_2048x1179.png 1272w, https://substackcdn.com/image/fetch/$s_!hWXs!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc282e643-055b-4c18-9505-5b96b0e8ea60_2048x1179.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><span>When I check Claude app, I see Claude took screenshots &amp; analyzed the pages.</span></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!4LFn!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2ac00a41-e627-45c5-84c6-57a74dae2957_1690x1672.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!4LFn!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2ac00a41-e627-45c5-84c6-57a74dae2957_1690x1672.png 424w, https://substackcdn.com/image/fetch/$s_!4LFn!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2ac00a41-e627-45c5-84c6-57a74dae2957_1690x1672.png 848w, https://substackcdn.com/image/fetch/$s_!4LFn!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2ac00a41-e627-45c5-84c6-57a74dae2957_1690x1672.png 1272w, https://substackcdn.com/image/fetch/$s_!4LFn!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2ac00a41-e627-45c5-84c6-57a74dae2957_1690x1672.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!4LFn!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2ac00a41-e627-45c5-84c6-57a74dae2957_1690x1672.png" width="1456" height="1440" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/2ac00a41-e627-45c5-84c6-57a74dae2957_1690x1672.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1440,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!4LFn!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2ac00a41-e627-45c5-84c6-57a74dae2957_1690x1672.png 424w, https://substackcdn.com/image/fetch/$s_!4LFn!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2ac00a41-e627-45c5-84c6-57a74dae2957_1690x1672.png 848w, https://substackcdn.com/image/fetch/$s_!4LFn!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2ac00a41-e627-45c5-84c6-57a74dae2957_1690x1672.png 1272w, https://substackcdn.com/image/fetch/$s_!4LFn!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2ac00a41-e627-45c5-84c6-57a74dae2957_1690x1672.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><span>A couple of minutes later, I got this Slack message:</span></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!ypMs!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F72dc0202-c356-4a66-b45a-7a2778db68a1_2048x1502.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!ypMs!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F72dc0202-c356-4a66-b45a-7a2778db68a1_2048x1502.png 424w, https://substackcdn.com/image/fetch/$s_!ypMs!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F72dc0202-c356-4a66-b45a-7a2778db68a1_2048x1502.png 848w, https://substackcdn.com/image/fetch/$s_!ypMs!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F72dc0202-c356-4a66-b45a-7a2778db68a1_2048x1502.png 1272w, https://substackcdn.com/image/fetch/$s_!ypMs!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F72dc0202-c356-4a66-b45a-7a2778db68a1_2048x1502.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!ypMs!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F72dc0202-c356-4a66-b45a-7a2778db68a1_2048x1502.png" width="1456" height="1068" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/72dc0202-c356-4a66-b45a-7a2778db68a1_2048x1502.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1068,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!ypMs!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F72dc0202-c356-4a66-b45a-7a2778db68a1_2048x1502.png 424w, https://substackcdn.com/image/fetch/$s_!ypMs!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F72dc0202-c356-4a66-b45a-7a2778db68a1_2048x1502.png 848w, https://substackcdn.com/image/fetch/$s_!ypMs!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F72dc0202-c356-4a66-b45a-7a2778db68a1_2048x1502.png 1272w, https://substackcdn.com/image/fetch/$s_!ypMs!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F72dc0202-c356-4a66-b45a-7a2778db68a1_2048x1502.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><span>&#8220;Clean&#8221;, simple yet effective.</span></p><p><span>Here is a more detailed report:</span></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!GwDr!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F673fbfb6-2f2c-45bd-95cd-8baf17224f9e_1612x1352.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!GwDr!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F673fbfb6-2f2c-45bd-95cd-8baf17224f9e_1612x1352.png 424w, https://substackcdn.com/image/fetch/$s_!GwDr!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F673fbfb6-2f2c-45bd-95cd-8baf17224f9e_1612x1352.png 848w, https://substackcdn.com/image/fetch/$s_!GwDr!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F673fbfb6-2f2c-45bd-95cd-8baf17224f9e_1612x1352.png 1272w, https://substackcdn.com/image/fetch/$s_!GwDr!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F673fbfb6-2f2c-45bd-95cd-8baf17224f9e_1612x1352.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!GwDr!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F673fbfb6-2f2c-45bd-95cd-8baf17224f9e_1612x1352.png" width="1456" height="1221" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/673fbfb6-2f2c-45bd-95cd-8baf17224f9e_1612x1352.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1221,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!GwDr!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F673fbfb6-2f2c-45bd-95cd-8baf17224f9e_1612x1352.png 424w, https://substackcdn.com/image/fetch/$s_!GwDr!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F673fbfb6-2f2c-45bd-95cd-8baf17224f9e_1612x1352.png 848w, https://substackcdn.com/image/fetch/$s_!GwDr!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F673fbfb6-2f2c-45bd-95cd-8baf17224f9e_1612x1352.png 1272w, https://substackcdn.com/image/fetch/$s_!GwDr!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F673fbfb6-2f2c-45bd-95cd-8baf17224f9e_1612x1352.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p></p><h4 style="text-align: center;"><strong>&#167;</strong></h4><div class="callout-block" data-callout="true"><p style="text-align: center;"><em><strong><span data-color="#3a7bf9" style="color: rgb(58, 123, 249);">Reminder: this is a teaser of the subscriber-only newsletter, exclusive to my golden members.</span></strong></em></p><p>When you upgrade, you&#8217;ll get:</p><ul><li><p><strong>A complete Claude Code setup, from your first session to a fully automated one.</strong></p></li><li><p>Deep dives into how MCP, subagents, agent orchestration &amp; hooks actually work inside Claude Code.</p></li><li><p><strong>How real Claude Code projects handle context, scale &amp; running without you.</strong></p></li></ul><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/subscribe&quot;,&quot;text&quot;:&quot;Unlock Full Access&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://newsletter.systemdesign.one/subscribe"><span>Unlock Full Access</span></a></p><p>(If this newsletter has helped you become a better software engineer, consider subscribing to support my work.)</p></div><h4 style="text-align: center;"><strong>&#167;</strong></h4><h2><span>Routine vs scheduled</span></h2>
      <p>
          <a href="https://newsletter.systemdesign.one/p/claude-code-website-monitoring-routine">
              Read more
          </a>
      </p>
   ]]></content:encoded></item><item><title><![CDATA[64 AI Concepts Every Software Engineer Should Know]]></title><description><![CDATA[#187: Part 1 - Generative AI, RAG, AI Agents, and 14 others.]]></description><link>https://newsletter.systemdesign.one/p/ai-concepts-explained-for-beginners</link><guid isPermaLink="false">https://newsletter.systemdesign.one/p/ai-concepts-explained-for-beginners</guid><dc:creator><![CDATA[Neo Kim]]></dc:creator><pubDate>Mon, 05 Oct 2026 13:44:22 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/f0daa132-5989-4780-8c01-725680a43fde_1280x720.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p></p><p></p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption"><span data-color="#3a7bf9" style="color: rgb(58, 123, 249);">Get my system design playbook for FREE on newsletter signup:</span></p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><ul><li><p><em><a href="https://newsletter.systemdesign.one/p/ai-concepts-explained-for-beginners/?action=share">Share this letter</a> &amp; I&#8217;ll send you some rewards for the referrals.</em></p></li></ul><div><hr></div><p>You ask an AI assistant a question&#8230;</p><p>&#8230;it answers in <em>seconds</em>!</p><p>But what actually happened between your question &amp; the answer?</p><p>Behind that simple interaction sits a stack of models, tokens, context, retrieval, tools, evaluations, and infrastructure.</p><p>These 64 concepts<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-1" href="#footnote-1" target="_self">1</a> will help you understand how modern AI systems actually work, from the neural networks underneath them to agents &amp; applications built on top.</p><p>Curious to know how many were new to you:</p><ol><li><p>Large language models,</p></li><li><p>Generative AI,</p></li><li><p>Machine learning,</p></li><li><p>Deep learning,</p></li><li><p>Neural networks,</p></li><li><p>Retrieval augmented generation,</p></li><li><p>AI agents &amp; agentic workflows,</p></li><li><p>Prompt engineering,</p></li><li><p>Transformers and self-attention,</p></li><li><p>Embeddings,</p></li><li><p>Tool &amp; function calling,</p></li><li><p>Model evaluation and evaluation-driven development,</p></li><li><p>Context engineering,</p></li><li><p>Multimodal AI,</p></li><li><p>Fine tuning and instruction tuning,</p></li><li><p>Tokenization,</p></li><li><p>Reasoning &amp; chain-of-thought.</p></li></ol><p>( &#8230;and much more in Parts 2, 3 &amp; 4!)</p><p>For each, I&#8217;ll share:</p><ul><li><p>What it means and how it works,</p></li><li><p>A real-world analogy where useful,</p></li><li><p>A common mistake to avoid,</p></li><li><p>Why it matters.</p></li></ul><p>Let&#8217;s dive in!</p><h4 style="text-align: center;"><strong>&#167;</strong></h4><div class="callout-block" data-callout="true"><h2><strong><a href="https://link.svix.com/system-01">Make your agents realtime (Partner)</a></strong></h2><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://link.svix.com/system-01" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!v-ZP!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fde8a87d9-0b90-4a74-857c-be37bba398f2_2000x1047.png 424w, https://substackcdn.com/image/fetch/$s_!v-ZP!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fde8a87d9-0b90-4a74-857c-be37bba398f2_2000x1047.png 848w, https://substackcdn.com/image/fetch/$s_!v-ZP!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fde8a87d9-0b90-4a74-857c-be37bba398f2_2000x1047.png 1272w, https://substackcdn.com/image/fetch/$s_!v-ZP!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fde8a87d9-0b90-4a74-857c-be37bba398f2_2000x1047.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!v-ZP!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fde8a87d9-0b90-4a74-857c-be37bba398f2_2000x1047.png" width="1456" height="762" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/de8a87d9-0b90-4a74-857c-be37bba398f2_2000x1047.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:762,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:212365,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:&quot;&quot;,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:&quot;https://link.svix.com/system-01&quot;,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://newsletter.systemdesign.one/i/211217193?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fde8a87d9-0b90-4a74-857c-be37bba398f2_2000x1047.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!v-ZP!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fde8a87d9-0b90-4a74-857c-be37bba398f2_2000x1047.png 424w, https://substackcdn.com/image/fetch/$s_!v-ZP!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fde8a87d9-0b90-4a74-857c-be37bba398f2_2000x1047.png 848w, https://substackcdn.com/image/fetch/$s_!v-ZP!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fde8a87d9-0b90-4a74-857c-be37bba398f2_2000x1047.png 1272w, https://substackcdn.com/image/fetch/$s_!v-ZP!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fde8a87d9-0b90-4a74-857c-be37bba398f2_2000x1047.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Agents are killing your apps with compounding latency across workflows.</p><p>Your customers want their agent workflows to be efficient, so you&#8217;re losing deals if you don&#8217;t make them event-driven. Webhooks solve this issue, but implementing them is harder than it looks.</p><p>If you homebrewed your webhooks and are getting pinged at 4 in the morning for failed deliveries, this is for you.</p><p><strong><a href="https://link.svix.com/system-01">Svix</a></strong> handles everything: retries, idempotency, security, and compliance. Qualified startups get $12,000 in free credits.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://link.svix.com/system-01&quot;,&quot;text&quot;:&quot;Get started for free&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://link.svix.com/system-01"><span>Get started for free</span></a></p><p><em>(Thanks to <strong><a href="https://link.svix.com/system-01">Svix</a></strong> for partnering on this newsletter.)</em></p></div><h4 style="text-align: center;"><strong>&#167;</strong></h4><h2><strong>1. Large Language Models</strong></h2><blockquote><p>A large language model (<strong>LLM</strong>) is an AI model trained on vast amounts of text to learn language patterns.</p></blockquote><p>You type <em>&#8220;Why is the sky blue?&#8221;</em> into a chatbot &amp; hit Enter.</p><p>The model processes your question as small pieces of text called <strong>tokens</strong>. A token could be a word, part of a word, or punctuation. A typical text-generating LLM selects the next token based on your question and the text it has produced so far.</p><p>Then it repeats. Token by token, the answer appears&#8230;</p><p>During training, the model adjusted internal numerical values called <strong>weights</strong>. Those weights capture patterns useful for language tasks, from sentence structure to relationships between ideas.</p><p>Your prompt gives the model context for its response.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!wtQ-!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc9c9f692-b971-4c90-b9f1-c3ee44f0a364_1448x1086.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!wtQ-!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc9c9f692-b971-4c90-b9f1-c3ee44f0a364_1448x1086.png 424w, https://substackcdn.com/image/fetch/$s_!wtQ-!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc9c9f692-b971-4c90-b9f1-c3ee44f0a364_1448x1086.png 848w, https://substackcdn.com/image/fetch/$s_!wtQ-!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc9c9f692-b971-4c90-b9f1-c3ee44f0a364_1448x1086.png 1272w, https://substackcdn.com/image/fetch/$s_!wtQ-!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc9c9f692-b971-4c90-b9f1-c3ee44f0a364_1448x1086.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!wtQ-!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc9c9f692-b971-4c90-b9f1-c3ee44f0a364_1448x1086.png" width="1448" height="1086" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/c9c9f692-b971-4c90-b9f1-c3ee44f0a364_1448x1086.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1086,&quot;width&quot;:1448,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!wtQ-!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc9c9f692-b971-4c90-b9f1-c3ee44f0a364_1448x1086.png 424w, https://substackcdn.com/image/fetch/$s_!wtQ-!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc9c9f692-b971-4c90-b9f1-c3ee44f0a364_1448x1086.png 848w, https://substackcdn.com/image/fetch/$s_!wtQ-!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc9c9f692-b971-4c90-b9f1-c3ee44f0a364_1448x1086.png 1272w, https://substackcdn.com/image/fetch/$s_!wtQ-!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc9c9f692-b971-4c90-b9f1-c3ee44f0a364_1448x1086.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><strong>Analogy</strong></h3><p>Think of autocomplete on your phone:</p><p>You type <em>&#8220;See you,&#8221;</em> and the keyboard suggests &#8220;<em>soon</em>.&#8221; An LLM takes this idea much further, using the available conversation to produce paragraphs, explanations, or code.</p><p>Predicting a plausible continuation still leaves room for a wrong answer.</p><h3><strong>Common mistake</strong></h3><p>Trusting an answer simply because it sounds <em>confident</em>.</p><p>Suppose you ask for code to resize an image. The model could suggest a function with a sensible name that the library doesn&#8217;t provide. Check the documentation &amp; run the code before you rely on it.</p><h3><strong>Why it matters</strong></h3><p>You don&#8217;t need a separate model for every language task.</p><p>Different instructions let the same LLM summarize a document, explain an error, or draft a reply.</p><p>That flexibility makes LLMs useful across many software products.</p><h2><strong>2. Generative AI</strong></h2><blockquote><p>Generative AI produces content such as text, images, audio &amp; video from patterns learned during training.</p></blockquote><p>Ask an image generator for <em>&#8220;a red bicycle outside a bakery.&#8221;</em></p><p>Your description guides the model toward a result. Some image models begin with random noise and refine it step by step into an image. A text model uses a different process, typically selecting tokens in sequence.</p><p>Both produce content from learned patterns.</p><p>Generative AI is the broader category&#8230; Text-generating LLMs are one example; image and audio generators are others.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!lCDy!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3c2c80fa-3ada-44d6-b5fc-6158721873e8_1448x1086.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!lCDy!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3c2c80fa-3ada-44d6-b5fc-6158721873e8_1448x1086.png 424w, https://substackcdn.com/image/fetch/$s_!lCDy!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3c2c80fa-3ada-44d6-b5fc-6158721873e8_1448x1086.png 848w, https://substackcdn.com/image/fetch/$s_!lCDy!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3c2c80fa-3ada-44d6-b5fc-6158721873e8_1448x1086.png 1272w, https://substackcdn.com/image/fetch/$s_!lCDy!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3c2c80fa-3ada-44d6-b5fc-6158721873e8_1448x1086.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!lCDy!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3c2c80fa-3ada-44d6-b5fc-6158721873e8_1448x1086.png" width="1448" height="1086" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/3c2c80fa-3ada-44d6-b5fc-6158721873e8_1448x1086.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1086,&quot;width&quot;:1448,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!lCDy!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3c2c80fa-3ada-44d6-b5fc-6158721873e8_1448x1086.png 424w, https://substackcdn.com/image/fetch/$s_!lCDy!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3c2c80fa-3ada-44d6-b5fc-6158721873e8_1448x1086.png 848w, https://substackcdn.com/image/fetch/$s_!lCDy!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3c2c80fa-3ada-44d6-b5fc-6158721873e8_1448x1086.png 1272w, https://substackcdn.com/image/fetch/$s_!lCDy!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3c2c80fa-3ada-44d6-b5fc-6158721873e8_1448x1086.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><strong>Analogy</strong></h3><p>Think of a cook who has learned from hundreds of recipes:</p><p>You ask for a spicy vegetable dish. The cook draws on familiar ingredients and techniques to prepare something that fits your request.</p><p>You still taste the dish before you serve it.</p><h3><strong>Common mistake</strong></h3><p>Assuming the generated content is ready to use.</p><p>An image could contain unreadable labels. A summary could leave out a key decision. Review the result against your original request and check factual details against the source.</p><h3><strong>Why it matters</strong></h3><p>Generative AI helps you get from a blank page to a first draft.</p><p>A few notes become an email, a description becomes an illustration, or a specification becomes starter code.</p><p>The next step is to check whether the draft does the job.</p><h2><strong>3. Machine Learning</strong></h2><blockquote><p>Machine Learning (<strong>ML</strong>) is a branch of AI in which systems learn from data to perform tasks without explicit rules for every situation.</p></blockquote><p>Think about the spam folder in your inbox:</p><p>You could write a rule to block every email containing <em>&#8220;free money.&#8221;</em> But a spammer could change the wording; also, a legitimate email could contain the same phrase. A model learns from examples &amp; considers several clues together.</p><p>Those clues help it assess a new message.</p><p>For a spam filter, training examples typically include the correct label:<em> spam or legitimate</em>. The training algorithm adjusts the model to reduce classification errors.</p><p>This approach is called <strong>supervised learning</strong>.</p><p>Other approaches discover patterns in unlabeled data or learn through rewards after actions.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!9-tF!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01348b3-cb95-4017-84c9-217cdf57ae08_1448x1086.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!9-tF!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01348b3-cb95-4017-84c9-217cdf57ae08_1448x1086.png 424w, https://substackcdn.com/image/fetch/$s_!9-tF!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01348b3-cb95-4017-84c9-217cdf57ae08_1448x1086.png 848w, https://substackcdn.com/image/fetch/$s_!9-tF!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01348b3-cb95-4017-84c9-217cdf57ae08_1448x1086.png 1272w, https://substackcdn.com/image/fetch/$s_!9-tF!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01348b3-cb95-4017-84c9-217cdf57ae08_1448x1086.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!9-tF!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01348b3-cb95-4017-84c9-217cdf57ae08_1448x1086.png" width="1448" height="1086" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/c01348b3-cb95-4017-84c9-217cdf57ae08_1448x1086.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1086,&quot;width&quot;:1448,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!9-tF!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01348b3-cb95-4017-84c9-217cdf57ae08_1448x1086.png 424w, https://substackcdn.com/image/fetch/$s_!9-tF!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01348b3-cb95-4017-84c9-217cdf57ae08_1448x1086.png 848w, https://substackcdn.com/image/fetch/$s_!9-tF!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01348b3-cb95-4017-84c9-217cdf57ae08_1448x1086.png 1272w, https://substackcdn.com/image/fetch/$s_!9-tF!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01348b3-cb95-4017-84c9-217cdf57ae08_1448x1086.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><strong>Analogy</strong></h3><p>Think about how you learn to choose ripe fruit:</p><p>Someone shows you a few good examples &amp; a few bad ones. Over time, you connect color, firmness, and smell with ripeness.</p><p>You use those clues on your next trip to the market.</p><h3><strong>Common mistake</strong></h3><p>Testing the model with the same examples it learned from.</p><p>That&#8217;s like giving a student the practice questions again and calling it a final exam. Reserve separate examples to check whether the model handles unfamiliar data.</p><h3><strong>Why it matters</strong></h3><p>ML helps when a task involves more patterns than you could reasonably capture in hand-written rules. It powers uses such as spam filters, fraud detection, and product recommendations.</p><p>The goal is useful predictions on new data.</p><h4 style="text-align: center;"><strong>&#167;</strong></h4><div class="captioned-button-wrap" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/p/ai-concepts-explained-for-beginners?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;}" data-component-name="CaptionedButtonToDOM"><div class="preamble"><p class="cta-caption"><em><span data-color="#3a7bf9" style="color: rgb(58, 123, 249);">If something resonated with you today, share this letter. Because one idea, one action, can change everything.</span></em></p></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/p/ai-concepts-explained-for-beginners?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://newsletter.systemdesign.one/p/ai-concepts-explained-for-beginners?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p></div><h4 style="text-align: center;"><strong>&#167;</strong></h4><h2><strong>4. Deep Learning</strong></h2><blockquote><p>Deep learning (<strong>DL</strong><a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-2" href="#footnote-2" target="_self">2</a>) is a type of ML that uses multiple layers of a neural network to learn complex patterns.</p></blockquote><p>Show a model a photograph of a bicycle.</p><p>The model receives numbers representing the image&#8217;s pixels. Early layers often detect simple features, such as edges &amp; textures. Later layers combine these features into more complex shapes &amp; object parts.</p><p>Together, the layers help the model recognize the bicycle.</p><p>The word &#8220;deep&#8221; refers to these multiple layers. During training, the model adjusts them to improve its predictions.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!HfUv!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9faf1b2e-a4ee-4c07-9747-d99e37e345d1_1448x1086.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!HfUv!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9faf1b2e-a4ee-4c07-9747-d99e37e345d1_1448x1086.png 424w, https://substackcdn.com/image/fetch/$s_!HfUv!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9faf1b2e-a4ee-4c07-9747-d99e37e345d1_1448x1086.png 848w, https://substackcdn.com/image/fetch/$s_!HfUv!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9faf1b2e-a4ee-4c07-9747-d99e37e345d1_1448x1086.png 1272w, https://substackcdn.com/image/fetch/$s_!HfUv!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9faf1b2e-a4ee-4c07-9747-d99e37e345d1_1448x1086.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!HfUv!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9faf1b2e-a4ee-4c07-9747-d99e37e345d1_1448x1086.png" width="1448" height="1086" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/9faf1b2e-a4ee-4c07-9747-d99e37e345d1_1448x1086.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1086,&quot;width&quot;:1448,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!HfUv!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9faf1b2e-a4ee-4c07-9747-d99e37e345d1_1448x1086.png 424w, https://substackcdn.com/image/fetch/$s_!HfUv!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9faf1b2e-a4ee-4c07-9747-d99e37e345d1_1448x1086.png 848w, https://substackcdn.com/image/fetch/$s_!HfUv!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9faf1b2e-a4ee-4c07-9747-d99e37e345d1_1448x1086.png 1272w, https://substackcdn.com/image/fetch/$s_!HfUv!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9faf1b2e-a4ee-4c07-9747-d99e37e345d1_1448x1086.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><strong>Analogy</strong></h3><p>Think of a sketch as someone draws it:</p><p>First, you see a few lines and circles. Add a frame, handlebars, and pedals, and the shapes start to make sense together.</p><p>A deep network also combines simpler features into more complex patterns.</p><h3><strong>Common mistake</strong></h3><p>Assuming every problem needs deep learning.</p><p>For a small spreadsheet of house sizes and prices, a simpler model could do the job well. Test a basic approach first, then compare whether a deep network earns its extra cost and complexity.</p><h3><strong>Why it matters</strong></h3><p>Images, speech &amp; language contain patterns which are difficult to describe with fixed rules. Deep learning helps models learn useful features from these inputs.</p><p>It underpins tools such as speech recognition &amp; image classification.</p><h2><strong>5. Neural Networks</strong></h2><blockquote><p>A neural network<a class="footnote-anchor" data-component-name="FootnoteAnchorToDOM" id="footnote-anchor-3" href="#footnote-3" target="_self">3</a> is a model made of connected mathematical units that transform inputs into an output.</p></blockquote><p>Let&#8217;s look inside the layers we just discussed:</p><p>Each unit receives numbers, combines them, and passes a result onward. Connections have adjustable values called weights, which control how much each input contributes. Layers also apply transformations, so the network learns more complex relationships.</p><p>The final layer produces the model&#8217;s output.</p><p>During training, an error measure tells the system how far its predictions are from the target. The training algorithm uses this feedback to adjust weights and other values across the network.</p><p>Tiny adjustments, repeated across many examples, change its predictions.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!3ju9!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc59287b4-bd90-49b6-b573-1487d7b2d367_1448x1086.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!3ju9!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc59287b4-bd90-49b6-b573-1487d7b2d367_1448x1086.png 424w, https://substackcdn.com/image/fetch/$s_!3ju9!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc59287b4-bd90-49b6-b573-1487d7b2d367_1448x1086.png 848w, https://substackcdn.com/image/fetch/$s_!3ju9!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc59287b4-bd90-49b6-b573-1487d7b2d367_1448x1086.png 1272w, https://substackcdn.com/image/fetch/$s_!3ju9!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc59287b4-bd90-49b6-b573-1487d7b2d367_1448x1086.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!3ju9!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc59287b4-bd90-49b6-b573-1487d7b2d367_1448x1086.png" width="1448" height="1086" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/c59287b4-bd90-49b6-b573-1487d7b2d367_1448x1086.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1086,&quot;width&quot;:1448,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!3ju9!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc59287b4-bd90-49b6-b573-1487d7b2d367_1448x1086.png 424w, https://substackcdn.com/image/fetch/$s_!3ju9!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc59287b4-bd90-49b6-b573-1487d7b2d367_1448x1086.png 848w, https://substackcdn.com/image/fetch/$s_!3ju9!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc59287b4-bd90-49b6-b573-1487d7b2d367_1448x1086.png 1272w, https://substackcdn.com/image/fetch/$s_!3ju9!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc59287b4-bd90-49b6-b573-1487d7b2d367_1448x1086.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><strong>Analogy</strong></h3><p>Think of the controls on a sound mixing desk:</p><p>One control increases the voice; another reduces the drums. Changing the controls changes the final sound.</p><p>Weights play a similar role in how a network combines its inputs.</p><h3><strong>Common mistake</strong></h3><p>Taking the word &#8220;neural&#8221; too literally.</p><p>The name draws inspiration from the brain, but these units perform mathematical operations. A network&#8217;s ability to recognize a face or produce a sentence doesn&#8217;t establish human-like thought.</p><h3><strong>Why it matters</strong></h3><p>Neural networks provide the structure behind deep learning.</p><p>Understanding weights also explains what training changes: numerical values a model uses to turn an input into a prediction.</p><h2><strong>6. Retrieval-Augmented Generation</strong></h2><blockquote><p>RAG is an approach that retrieves relevant information &amp; supplies it to a model to help generate an answer.</p></blockquote><p>Ask a company assistant, <em>&#8220;How long do I have to return this order?&#8221;</em></p><p>The application searches the company&#8217;s documents for the relevant return policy. It places matching passages alongside your question. The model uses those passages to write its response.</p><p>The answer has information from the company&#8217;s documents.</p><p>Updating those documents gives the application new material to retrieve. It doesn&#8217;t require a new training run for every policy change.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!kftM!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fae54d9a8-e6c9-420d-b231-1c188d7cfdb7_1448x1086.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!kftM!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fae54d9a8-e6c9-420d-b231-1c188d7cfdb7_1448x1086.png 424w, https://substackcdn.com/image/fetch/$s_!kftM!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fae54d9a8-e6c9-420d-b231-1c188d7cfdb7_1448x1086.png 848w, https://substackcdn.com/image/fetch/$s_!kftM!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fae54d9a8-e6c9-420d-b231-1c188d7cfdb7_1448x1086.png 1272w, https://substackcdn.com/image/fetch/$s_!kftM!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fae54d9a8-e6c9-420d-b231-1c188d7cfdb7_1448x1086.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!kftM!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fae54d9a8-e6c9-420d-b231-1c188d7cfdb7_1448x1086.png" width="1448" height="1086" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/ae54d9a8-e6c9-420d-b231-1c188d7cfdb7_1448x1086.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1086,&quot;width&quot;:1448,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!kftM!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fae54d9a8-e6c9-420d-b231-1c188d7cfdb7_1448x1086.png 424w, https://substackcdn.com/image/fetch/$s_!kftM!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fae54d9a8-e6c9-420d-b231-1c188d7cfdb7_1448x1086.png 848w, https://substackcdn.com/image/fetch/$s_!kftM!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fae54d9a8-e6c9-420d-b231-1c188d7cfdb7_1448x1086.png 1272w, https://substackcdn.com/image/fetch/$s_!kftM!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fae54d9a8-e6c9-420d-b231-1c188d7cfdb7_1448x1086.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><strong>Analogy</strong></h3><p>Think of an open-book exam:</p><p>You find the relevant pages before you answer. Your result depends on both finding the right material &amp; interpreting it correctly.</p><h3><strong>Common mistake</strong></h3><p>Assuming retrieval guarantees a correct answer.</p><p>The search could return an outdated policy and/or miss an exception. So check the retrieved passages &amp; final response. When the evidence is missing, the assistant should say so.</p><h3><strong>Why it matters</strong></h3><p>RAG helps assistants answer questions about private or frequently updated information. Also it gives readers source material to check against the answer.</p><h2><strong>7. AI Agents and Agentic Workflows</strong></h2><blockquote><p>An AI agent uses a model to choose actions toward a goal and respond to the results. Agentic workflows organize model and tool steps into a process.</p></blockquote><p>Ask an assistant to investigate a failed software test.</p><p>An agent would inspect the error, open a file, make an edit, and rerun the tests. The result helps it decide what to do next. A fixed workflow follows a prewritten sequence, such as &#8220;<em>summarize the ticket, classify it, then route it.</em>&#8221;</p><p>The difference is how much control the model has over the next step.</p><p>The term &#8220;agentic workflow&#8221; covers a range of designs. Some follow mostly fixed steps; others give the model more freedom to choose.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!rMan!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F39e6f430-a713-45f6-bd1d-395847b2d4cf_1448x1086.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!rMan!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F39e6f430-a713-45f6-bd1d-395847b2d4cf_1448x1086.png 424w, https://substackcdn.com/image/fetch/$s_!rMan!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F39e6f430-a713-45f6-bd1d-395847b2d4cf_1448x1086.png 848w, https://substackcdn.com/image/fetch/$s_!rMan!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F39e6f430-a713-45f6-bd1d-395847b2d4cf_1448x1086.png 1272w, https://substackcdn.com/image/fetch/$s_!rMan!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F39e6f430-a713-45f6-bd1d-395847b2d4cf_1448x1086.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!rMan!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F39e6f430-a713-45f6-bd1d-395847b2d4cf_1448x1086.png" width="1448" height="1086" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/39e6f430-a713-45f6-bd1d-395847b2d4cf_1448x1086.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1086,&quot;width&quot;:1448,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!rMan!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F39e6f430-a713-45f6-bd1d-395847b2d4cf_1448x1086.png 424w, https://substackcdn.com/image/fetch/$s_!rMan!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F39e6f430-a713-45f6-bd1d-395847b2d4cf_1448x1086.png 848w, https://substackcdn.com/image/fetch/$s_!rMan!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F39e6f430-a713-45f6-bd1d-395847b2d4cf_1448x1086.png 1272w, https://substackcdn.com/image/fetch/$s_!rMan!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F39e6f430-a713-45f6-bd1d-395847b2d4cf_1448x1086.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3><strong>Analogy</strong></h3><p>Think of a technician with a checklist:</p><p>For a routine inspection, the checklist determines the order. For an unfamiliar fault, the technician chooses the next check based on what the previous one revealed.</p><h3><strong>Common mistake</strong></h3><p>Giving an agent tools without clear limits.</p><p>A failed action could trigger repeated retries/unwanted changes. So set permissions, spending limits, and stopping conditions. Plus, require approval before consequential actions, such as a payment/deletion.</p><h3><strong>Why it matters</strong></h3><p>Agents help with tasks whose steps depend on discoveries along the way.</p><p>Fixed workflows work when the sequence is known and predictable execution matters.</p><h4 style="text-align: center;"><strong>&#167;</strong></h4><div class="callout-block" data-callout="true"><p style="text-align: center;"><em><strong><span data-color="#3a7bf9" style="color: rgb(58, 123, 249);">Reminder: this is a teaser of the subscriber-only newsletter series, exclusive to my golden members.</span></strong></em></p><p>When you upgrade, you&#8217;ll get:</p><ul><li><p><strong>Simple breakdown of real-world architectures</strong></p></li><li><p>Frameworks you can plug into your work or business</p></li><li><p><strong>Proven systems behind ChatGPT, Perplexity, and Copilot</strong></p></li></ul><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://newsletter.systemdesign.one/subscribe&quot;,&quot;text&quot;:&quot;Unlock Full Access&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://newsletter.systemdesign.one/subscribe"><span>Unlock Full Access</span></a></p><p><em>Ready for the best part?</em></p></div><h4 style="text-align: center;"><strong>&#167;</strong></h4><h2><strong>8. Prompt Engineering</strong></h2>
      <p>
          <a href="https://newsletter.systemdesign.one/p/ai-concepts-explained-for-beginners">
              Read more
          </a>
      </p>
   ]]></content:encoded></item></channel></rss>