Build for the long run
Long-running agents need endurance.
They must endure waiting for users and weather tool failures. They must take each new jump in model intelligence in stride.
Inngest makes long-running agents durable, observable and improvable over time.
Build for the long run with Inngest »
(Thanks to Inngest for partnering on this newsletter.)
§
The 80/20 rule states that 80% of results come from 20% of actions.
It is the ONE rule you must obey & this is the most important rule when it comes to learning something new.
Let me explain...
Imagine you want to fill a bottle with pebbles, sand & water.
If you pour in the sand & water first, you’ll struggle to fit the big pebbles.
But if you start with the pebbles, the sand automagically fills the gaps between them, and the water fills the remaining space.
i.e., do the 20% that makes the real difference (80%) first.
And this is why I’m writing these letters1 on AI engineering concepts for you.
I want you to win in the AI era by understanding the “fundamental” AI concepts.
Inside this newsletter, you’ll find:
Mixture-of-Experts,
Large Language Model Serving & Inference Engines,
Synthetic Data & Self-Instruction,
LLM-as-a-Judge,
Direct Preference Optimization,
Knowledge Distillation,
Temperature & Sampling,
Gradient Descent & Optimization,
Backpropagation,
Loss Functions,
Generalization,
Scaling Laws,
AI Alignment,
AI Safety & Guardrails.
For each, we’ll cover:
What it means & how it works,
A real-world analogy where useful,
A common mistake to avoid,
Why it matters.
Let’s go!
§
Share this letter & I’ll send you some rewards for the referrals.
§
35. Mixture-of-Experts
Mixture-of-Experts (MoE2) is a model design that uses a router to direct information to neural network components3 called experts. Common sparse designs activate only some experts for each token.
Let’s look inside a model as it processes a question:
Imagine one layer contains eight experts. A router selects two experts to process a token’s numerical representation. Then the layer combines their results & passes the information onward.
The next token could use a different pair of experts.
The router learns how to assign work during training. Each expert performs part of the model’s calculations, while other components also contribute to the response.
(These experts work within the same model.)
Analogy
Think of a workshop with several workers & a coordinator.
The coordinator assigns each task to a few workers. Those workers contribute their results, and the work continues through the workshop.
Not every worker needs to handle every piece.
Common mistake
Assuming each expert has a fixed subject - mathematics, coding, or history.
The word “expert” describes a neural network component. Training shapes its behavior & router’s choices, but those choices don’t necessarily follow recognizable subject categories.
Why it matters
Sparse MoE allows a model to have more parameters without using all its experts for every token. Only selected experts perform calculations, thus reducing computation while giving the model more capacity to learn patterns.
Yet all expert weights still need to be stored in memory/storage.
So using fewer experts per token reduces computation, but doesn’t reduce the model’s memory requirements.
36. LLM Serving and Inference Engines
LLM serving makes a trained model available to receive requests & return responses. An inference engine is the software that runs the model’s calculations.
You submit “Explain this error” through a chatbot:
The serving system accepts your request & passes it to the software that runs the model.
Inference engine uses the model’s weights & your input to generate tokens.
The serving system sends those tokens back to your screen.
Your answer arrives through several coordinated steps.
Now imagine hundreds of users submitting questions at once...
The system schedules work within the available memory & computing capacity. Some requests need longer responses than others. Requests that exceed available capacity must wait or get rejected.
The model needs software around it to serve real users.
Analogy
Think of a restaurant’s kitchen & service team4:
The kitchen prepares the meals… The wider restaurant operation accepts orders, coordinates workload, and delivers food to customers.
Preparing a meal is one part of serving a restaurant full of people.
Common mistake
Testing one request & assuming the system is ready for traffic.
A setup that responds quickly to one user could slow down when many requests arrive at once5. Test realistic request volumes & response lengths, then check waiting times, failures & memory use.
Why it matters
LLM serving helps applications handle requests from many users.
While inference engines run the model’s calculations to generate responses. Together, they affect response time, hardware efficiency, and how reliably an application serves users.
Choosing a suitable engine & testing the complete serving system helps turn a trained model into a dependable service.
37. Synthetic Data and Self-Instruction
Synthetic data is “artificially created” data used for purposes such as training/testing. Self-instruction6 uses a language model to generate tasks & corresponding responses for further training.
Imagine you need more training examples for a model that classifies “customer support” messages.
You ask a language model to generate fictional messages about billing & technical problems.
With self-instruction, the model also creates instructions & expected answers for those messages.
Each example contains a task & its correct label:
Instruction: Classify this customer message.
Message: “My card got charged twice.”
Expected answer: Billing.
The system then checks generated examples & removes incorrect, repetitive, and/or unsuitable ones.
The remaining examples could be used to fine-tune the model…
Also generating examples alone doesn’t change the model’s weights… A separate training step uses those examples to update the weights, thus helping the model learn to classify similar messages.
Analogy
Think of preparing practice questions with an answer key:
You create exercises to cover different situations. Before anyone studies from them, you check whether the questions make sense & if the answers are correct.
Faulty practice material teaches faulty lessons.
Common mistake
Treating a large generated dataset as a high-quality dataset.
Generated examples could contain wrong answers, repeated patterns, or unrealistic situations. Check their accuracy & variety, then evaluate the trained model on separate examples that reflect real use.
Why it matters
Synthetic data helps expand training material when suitable examples are scarce.
Self-instruction provides one way to create examples for instruction-based tasks.
The value comes from useful, checked examples rather than volume alone.
§
§
38. LLM-as-a-Judge
LLM-as-a-Judge uses a language model to evaluate another model’s responses against predefined criteria, such as factual accuracy, relevance, or completeness, and assigns scores/rankings based on those criteria.
Ask an assistant to shorten a customer’s complaint.
The customer wrote, “My order arrived three days late, and the box was damaged.” But the assistant produces “The order arrived late.” You give a judge model both versions & ask it to check whether the summary includes both problems.
The judge could flag the missing detail: damaged box.
Your instructions define what the judge checks. It could score one response, explain a problem, or compare two answers & select the better one.
Clear criteria give the review a specific purpose.
Analogy
Think of a teacher checking a student’s work:
The teacher compares the answer to the assignment requirements. A judge model performs a similar review using the instructions and reference material you supply.
Common mistake
Treating the judge’s score as proof of correctness.
A judge could overlook an error or favor a longer answer without good reason. Compare some of its reviews with careful human assessments. When comparing 2 answers, swap their order to check whether its preference changes.
Why it matters
Reading every response by hand takes time.
LLM judges help teams review large collections of answers & identify responses that need closer inspection.
Their usefulness depends on how reliably their assessments match qualities you need.
39. Direct Preference Optimization
Direct Preference Optimization (DPO) “trains” a language model to favor preferred responses over less preferred alternatives using pairs of answers to the same request.
Imagine you want an assistant to reply politely to customers.
For “My order is late,” one response says, “Just wait.” Another says, “I’m sorry about the delay. Please send your order number.” A reviewer marks the second response as the better choice.
The pair then becomes a training example.
DPO uses many such pairs to adjust the model’s weights toward preferred responses during training. It uses recorded choices directly, without first training a separate reward model to score answers.
Choosing a response records feedback; the training step changes the model.
Analogy
Think of helping someone write customer replies:
You show two drafts & point out which one better follows the company’s standards. Repeated practice guides how the person writes future replies.
DPO guides a model through similar comparisons.
Common mistake
Choosing preferred answers based only on how pleasant they sound.
A polite reply could promise a refund the customer doesn’t qualify for. Review accuracy & company policy alongside tone, so the training examples reward useful behavior.
Why it matters
DPO provides a simpler way to train from preferences than approaches which require a separate reward model and reinforcement learning.
The quality of those preferences shapes the behavior the model learns.
40. Knowledge Distillation
Knowledge distillation trains a student model7 to learn from a teacher model’s outputs. A common goal is to preserve useful abilities in a smaller model.
Imagine a large model recognizes cats & dogs well, but your application needs something smaller.
You show photos to the teacher model & collect its predictions. The student processes the same photos, and the training algorithm compares its predictions to the teacher’s. Then it adjusts student’s weights to bring their predictions closer together.
The teacher’s outputs guide the student’s training.
Some methods also use the teacher’s confidence across possible answers. This gives the student more information than a single label such as “cat.” The student learns through its own weight adjustments.
After training, the student handles new photos independently.
Analogy
Think of a teacher helping a student practice.
The teacher demonstrates answers, and the student learns from those examples. The student attempts new questions later without the teacher beside them.
Practice doesn’t guarantee the student matches the teacher on every question.
Common mistake
Assuming the student inherits every ability the teacher has.
A student trained on cat & dog photos hasn’t necessarily learned to recognize other animals. It could also repeat the teacher’s errors. Test it on separate examples which reflect the job you need it to perform.
Why it matters
Distillation helps teams develop models which need less memory and computation for a chosen task. A smaller student could serve requests faster or run on more limited hardware.
Check whether those savings preserve the quality your application needs.
41. Temperature and Sampling
Sampling is how an AI model chooses the next token during inference. Tokens the model considers likely have a higher chance of being chosen.
Temperature controls how predictable that choice is by adjusting the probabilities to make likely options more dominant or give less likely options more chance.
Ask an assistant to suggest a name for your bakery:
The model builds each suggestion one token at a time. At each step, it assigns probabilities to possible next tokens. Sampling chooses among those options according to their chances.
Different choices lead to different names.
Lower temperature favors the leading options more strongly, usually producing less variation across attempts. Higher temperature gives less likely options more chance, usually producing more varied suggestions.
Yet some results could be unusual/awkward… More variation doesn’t guarantee a better name.
These settings affect token selection without changing the model’s weights.
Analogy
Think of drawing a slip from a bowl.
Several slips carry the same word, while others carry different words. A word with more slips has a higher chance of being selected.
Temperature changes the balance of slips; sampling makes the draw.
Common mistake
Treating temperature as a quality control.
A higher setting doesn’t guarantee originality, and a lower setting doesn’t guarantee accuracy. Review results for readability & relevance, and check factual answers against reliable sources.
Why it matters
Different tasks need different amounts of variation…
Name suggestions benefit from a range of options, while extracting details from a receipt benefits from consistent results.
Test the settings against the task rather than using one value for everything.
§
Reminder: this is a teaser of the subscriber-only newsletter, exclusive to my golden members.
When you upgrade, you’ll get:
High-level architecture of real-world systems.
Deep dive into how popular real-world systems work.
How real-world systems handle scale, reliability & performance.
(If this newsletter has helped you become a better software engineer, consider subscribing to support my work.)