Don’t go back to the starting line
Inngest makes your code durable by default.
Simply wrapping your workflows in Inngest’s step.run provides automatic step-level checkpointing & retries, with zero infra to manage.
(Thanks to Inngest for partnering on this newsletter.)
Onward…
You type a question into ChatGPT:
“Why is the sky blue?”
… and hit Enter!
A moment later, the answer starts appearing.
From your side, that is the entire interaction…
But behind the screen, an entire “inference” system has already started working.
And it has 3 HARD problems to solve:
How to generate tokens quickly when each step needs access to a huge model?
How to serve many users without running out of GPU memory & capacity?
How to make responses fast for users without wasting expensive hardware?
To answer those questions, we’ll follow one request from the moment you hit Enter to the moment the answer reaches your screen.
Along the way, we will see how tokenization, prefill, decode, KV cache, batching, memory management, and distributed serving all fit together.
Let’s start with the request itself…
Share this letter & I’ll send you some rewards for the referrals.
Inference
A model goes through two main stages: training & inference.
During training, the model learns its parameters--weights. It processes large amounts of data & repeatedly adjusts those weights to get better at predicting what comes next. Once a trained model gets deployed, those weights are then typically fixed!
Then the inference begins:
When you send the model a prompt, it does not retrain itself on your question. It uses the weights it already learned to produce a response. This process is called inference.
Training creates the model…Inference runs it.
(This distinction becomes important at scale.)
Training requires enormous computing power, but it occurs during model development. Inference occurs every time someone sends a request. The same trained checkpoint can therefore perform inference again and again for as long as it is being served.
Our request is one of those inference jobs.
But “Why is the sky blue?” cannot be handed directly to the model as a string of English text. It first has to be converted into the representation the model expects.
That begins with tokenization…
Tokenization
A token is a piece of text. Sometimes, it is a complete word, and sometimes it is only part of one. Spaces & punctuation also appear as part of tokens.
Large Language Models (LLMs) work with tokens, not words.
So our question, “Why is the sky blue?”, could be split like this:
[”Why”, “ is”, “ the”, “ sky”, “ blue”, “?”]
The exact split depends on the tokenizer.
Another model may divide the same sentence differently. Each token maps to an integer in the model’s vocabulary. These integers (token IDs) become the model’s discrete input before it maps them to learned vector representations.
That’s why tokens appear everywhere when you use an LLM.
API pricing get typically measured in tokens.
Context windows get measured in tokens.
Generation speed gets measured in tokens per second.
The tokenizer still needs a method for deciding where to split the text.
One common method is Byte Pair Encoding (BPE1). It builds a vocabulary by repeatedly finding common adjacent units & merging them. Frequent patterns could eventually become individual tokens, while less common words often get split into several pieces.
This gives the tokenizer a useful compromise.
It does not need a vocabulary containing every possible word. Plus, it does not have to process normal text one character at a time.
Another tokenization framework you may come across is SentencePiece2.
SentencePiece applies algorithms such as BPE/Unigram3 directly to raw text, including whitespace. This makes it useful for languages where spaces are not reliable word boundaries.
By the end of tokenization, our sentence is no longer just text.
It has become a sequence of token IDs. Before the model processes those tokens, however, we need to consider a limit.
That limit is the context window.
Context Window
It’s the maximum number of tokens a model works with at one time.
Your visible prompt is not necessarily the only thing taking up this space.
A chat request also contains earlier messages, system instructions, retrieved documents, tool definitions, or other information the application sends to the model. Depending on the model & API, generated output also needs to fit within the applicable sequence or output limits. This is why long conversations eventually need some form of context management.
What happens at the limit depends on the application:
A system could remove older messages, summarize them, retrieve only relevant parts, or reject a request that is too large. The model’s context window should NOT be confused with a permanent memory of everything that has happened in conversation.
Context length matters for another reason: longer prompts require more work.
During attention4, the model uses information from different positions in the sequence to understand how the tokens relate to one another. As the sequence grows, the model has more information to process & store. This increases computation and memory requirements, although the exact cost depends on the model architecture and attention implementation.
A larger context window also does not mean the model will use every part equally well.
Information buried in the middle of a long prompt can be harder to retrieve than information closer to the beginning/end.
Our question is tiny by comparison: “Why is the sky blue?”
Once those tokens are inside the available context, the model begins processing them.
Prefill
Prefill is the stage where the model processes tokens in your prompt before it starts generating an answer.
LLMs are built from many layers stacked on top of one another.
Each layer takes the representation produced by the previous layer, processes it, and passes a new representation forward. By the time the data reaches final layers, the model has built a richer representation of the prompt.
Inside these layers is a mechanism called attention.
Attention helps the model decide which earlier tokens are useful when processing the current token. In our question, for example, when the model processes blue, information from sky could be relevant to understanding what the sentence is asking.
LLMs used for text generation normally use causal attention.
Causal attention means a token uses information from earlier positions, but not from positions ahead of it.
This preserves the left-to-right structure needed for predicting the next token. During prefill, however, every token in our prompt already exists: “Why is the sky blue?”. The GPU can therefore process work across the prompt positions in parallel rather than waiting for each token to be generated first.
While processing the prompt, attention creates keys & values for each token.
You can think of the key as information used to determine when a token is relevant. The value contains the information the model can retrieve. The model keeps these keys & values in the KV cache.
The KV cache will be useful again during generation.
Storing the keys & values means the model does not need to recompute the same key and value projections for every previous token at every generation step.
(We will look much more closely at the KV cache later.)
Once the prompt has passed through the model, it must decide which token comes next.
The model assigns a raw score to every possible token in its vocabulary. These scores are called logits. A decoding or sampling strategy uses the logits to select the first output token.
Depending on the strategy, this can mean choosing the strongest candidate or sampling from several likely candidates.
The work done before the first generated token contributes to a metric called Time to First Token (TTFT).
TTFT measures the time between sending a request and receiving the first part of the response. Prefill contributes to this delay, but it is not the only source. A request can also spend time traveling over the network, waiting for available GPU capacity, or sitting in a queue while the serving system decides which request should run next.
Once the first output token has been chosen, inference enters a different phase…
Decode
Decode is the stage where the model generates a response one token at a time.
Suppose the response begins: “The sky appears blue because...”
The model cannot generate the entire sentence in one shot. It first chooses a token such as ‘The’. That token becomes part of the sequence. The model then uses the updated sequence to choose the next token.
This is autoregressive generation5.
Each newly generated token depends on the tokens before it.
The process looks roughly like this:
The
The sky
The sky appears
The sky appears blue
At each decode step, the model still needs information from tokens it has already processed.
Instead of recomputing the keys & values for all previous tokens, it reuses them from the KV cache created during prefill. The model computes the key and value for the newest token and adds them to the cache. It then passes the newest token through its layers and produces another set of logits: scores for the possible next token.
The decoding strategy uses those logits to select the next token.
The selected token joins the sequence, and another decode step begins. This process continues until the model reaches a stopping condition, such as an end-of-sequence token or a configured output limit.
During prefill, ALL prompt tokens are already available, so the hardware can process work across many positions in parallel. During ordinary autoregressive decode, the next token does not exist until the previous one has been selected. This sequential dependency makes output generation behave differently from input processing.
There is also a hardware cost inside this loop.
Every decode step runs through the model’s layers & repeatedly accesses a large set of model weights. For large models, moving this data through GPU memory could become a major bottleneck.
This is where the simple story of “model predicts the next token” starts turning into a system design problem…
Streaming Responses
Streaming is the process of sending parts of the model’s response to the user while the rest of the response is still being generated.
We have already seen how decode works…
The model generates one token, adds it to the sequence, then generates the next one. Without streaming, the server could wait for the entire decode process to finish before returning the answer.
That would leave you looking at an empty screen while the model generates the full response.
Streaming changes when the output reaches you.
Suppose the model is generating: “The sky appears blue because...”
As parts of the response become available, the server could start sending them back to the application. Your browser then displays the beginning while the model is still generating the rest…
Streaming does not make the model generate tokens faster.
It changes when you receive those tokens. For an interactive application such as a chatbot, this makes the response feel more responsive.
How the response reaches your browser
The server needs a way to keep sending data after the request has started.
One common option is Server-Sent Events (SSE):
SSE keeps an HTTP connection open so server can continue sending events to the client instead of returning one response and immediately closing the exchange.
The communication moves in one direction: server sends updates & client receives them.
Server → Client
This works well for text generation because most of the response data travels from the server to the client after you send your prompt.
Another option is WebSockets.
A WebSocket creates a persistent connection where both sides can send data:
Server ↔ Client
This two-way communication is useful when the client needs to keep sending information while receiving results. A real-time voice assistant is one example because the user can continue sending audio while the server sends information back.
There is one more detail between decode & what appears on your screen: buffering.
The server does not have to send every generated token as a separate update. It could collect several tokens and send them together as one chunk.
For example:
Token 1 → send
Token 2 → send
Token 3 → send
OR
Tokens 1–3 → send together
The choice affects how the response feels.
Sending smaller chunks more frequently makes the response appear smoother. Grouping more tokens into each chunk reduces some of the overhead from repeatedly sending tiny updates.
So there is a tradeoff between responsiveness & communication overhead.
Our request has now completed the first part of its journey6:
Text → Tokens → Prefill → Decode → Stream
We started with: “Why is the sky blue?”
We have followed it from ordinary text to the response appearing on the screen. For one request, this process is manageable. A production AI service has to repeat it for many users while their requests compete for the same hardware.
This is where inference starts becoming a HARD systems problem.
Reminder: this is a teaser of the subscriber-only newsletter, exclusive to my golden members.
When you upgrade, you’ll get:
High-level architecture of real-world systems.
Deep dive into how popular real-world systems work.
How real-world systems handle scale, reliability & performance.
(If this newsletter has helped you become a better software engineer, consider subscribing to support my work.)