How to Build a Domain Expert AI
#162: Part 4 - Generative AI Masterclass
Share this letter & I’ll send you some rewards for the referrals.
A general-purpose AI model, even with access to clinical notes, still does not write like a clinician…
Give the model access to legal documents, and it still does not draft like a lawyer…
The vocabulary is familiar, but the format and conventions are NOT present by default.
Ask a foundation model to write up a patient visit, and you get:
“The patient is a 70-year-old man with a history of hypertension
and hyperlipidemia who presents to the emergency department
complaining of substernal chest pain, pressure-like,
radiating to the left arm and jaw, with nausea, diaphoresis,
and shortness of breath.”While a clinician writes:
“70 y/o M w/ h/o HTN, HLD presents to ED c/o substernal CP.
Pressure-like, radiates to L arm and jaw. + nausea, diaphoresis, SOB.”Same content…
One is a paragraph; the other is a chart entry. The general model knows the medicine; it doesn’t know how clinicians write it down.
Fine-tuning is how you get them there.
Rather than prompting the model into a style on every call, you train those patterns directly into the model’s weights so they become its defaults. Fine-tuning is how tools like Harvey write like junior associates at large law firms.
Glass Health produces clinical notes in the format clinicians use.
BloombergGPT reasons over financial data the way analysts expect.
GitHub Copilot responds fast enough to feel like autocomplete.
Each of these is a general-purpose model trained in a narrow domain.
This newsletter covers how fine-tuning works, when it is the right choice over prompting or RAG, and how to build it.
Let’s dive in!
[Webinar] 8 levels of context maturity in AI-native engineering (Partner)
AI is in your engineering workflow. While the token spend shows it, the throughput doesn’t. The human is very much still in the loop, and that’s a context problem.
Join live July 23 (FREE) to learn:
The 8 levels of context maturity: where most teams are stuck and what the ceiling looks like at each stage
Why more MCPs, rules, and skills provide agents access but not understanding
How leading engineering teams are using a context engine to make the most of their agents
(Thanks, Unblocked, for partnering on this newsletter.)
I want to reintroduce Louis-François Bouchard as the author of this newsletter.
He’s a best-selling author (Building LLMs for Production), the co-founder of Towards AI, and the creator of the YouTube Channel, What’s AI, where he helps people understand AI and learn how to apply it in the real world.
Through his development work with clients and his content, teaching, and AI training programs on the Towards AI Academy, Louis focuses on making AI practical for builders, engineers, and curious learners alike.
At Towards AI, he and his team train AI engineers through courses built for every stage, from beginner to advanced. That educational mission and the real-world experience building for his clients are exactly why I wanted him in this newsletter series.
Here’s what’s inside this newsletter:
What fine-tuning change in a model? Behavior, style, and domain reasoning patterns.
Low-Rank Adaptation (LoRA) and Quantized Low-Rank Adaptation (QLoRA). The methods that make fine-tuning practical by modifying less than 1% of a model’s weights.
Data quality. Instruction-response pairs, dataset size, synthetic data, and the gap between training distribution and production.
The decision framework. When prompting is enough, when RAG is the right addition, when fine-tuning is warranted, and when all three are needed together.
Alignment beyond Supervised Fine-Tuning (SFT). Reinforcement Learning from Human Feedback (RLHF), Direct Preference Optimization (DPO), model merging, and distillation.
A hands-on comparison. Three configurations for the same clinical SOAP note task: base model, fine-tuned model, and frontier model ceiling.
Golden members get all letters like these!…
Why Build Your Own Domain Expert AI?
Models such as GPT-5.5, Claude Opus 4.7, and Gemini 2.5 Pro are foundation models: Large Language Models (LLMs) trained on broad, internet-scale text to perform many tasks reasonably well rather than one task very well.
Their defaults are calibrated for a wide audience, not a specific domain.
For a domain product, this means out-of-the-box quality is just the starting point, and specialist users typically expect more…
Domain vocabulary and reasoning. A foundation model recognizes terms such as “Transient Ischemic Attack” and “force majeure,” but recognition is not the same as application. It does not consistently apply the National Institutes of Health (NIH) Stroke Scale, the standard scoring system clinicians use to grade stroke severity. It does not draft a force majeure clause the way a specific law firm drafts it. Fine-tuning bakes those specialized patterns into the model’s weights so they appear by default, without requiring them to be prompted on every call.
Tone and style consistency. Every domain has a house style. Brand voice for marketing, a formal style for legal memos, a structured format for clinical chart notes. Harvey reads like a junior associate at a large law firm because the underlying model has been customized for legal drafting work. Prompting can imitate a style for a turn or two, but the model usually drifts back to its default voice within a few turns.
Latency and model size. A fine-tuned 7-billion-parameter (7B) model often beats a foundation model on a narrow task. On a single Graphics Processing Unit (GPU), a fine-tuned 7B model can produce its first output token in under 300 milliseconds. This metric is called Time-To-First-Token (TTFT). For a product like code autocomplete, where suggestions must appear before the user types the next keystroke, this latency is the difference between a usable and an unusable product. GitHub Copilot is fine-tuned for exactly this reason.
Cost at scale. Above a few hundred thousand queries per month (lower against a flagship-plus-RAG configuration, higher against a mini-class API), a self-hosted fine-tuned model is cheaper than paying for foundation-model Application Programming Interface (API) calls. Below this volume, the work required to build the dataset and run training usually costs more than a year’s worth of foundation-model API bills.
Data privacy. Some training data cannot leave your infrastructure. Patient charts are regulated under the Health Insurance Portability and Accountability Act (HIPAA), the attorney-client privilege protects legal documents, and customer records may be regulated under the General Data Protection Regulation (GDPR) or System and Organization Controls 2 (SOC 2). None of this data can be sent to a third-party API for processing. Open-weight fine-tuning addresses this constraint by keeping the model, training data, and inference traffic within your own Virtual Private Cloud (VPC), behind your own authentication, and with your own audit trail.
Before deciding to fine-tune, you must confirm prompting and Retrieval-Augmented Generation (RAG) do NOT meet your quality bar.
Most people overestimate how much fine-tuning will buy them over a well-engineered prompt combined with retrieval. Fine-tuning is the right answer when one of the five reasons above applies. But it is the wrong answer when you are reaching for it because the prompting felt inconvenient.
Once you have decided to fine-tune, three design questions determine the system: which base model to start with, which dataset to build on, and whether to pair fine-tuning with retrieval.
What We Are Building
The system has four components:
Base model is the pretrained Large Language Model (LLM) the fine-tuning pipeline starts from.
Dataset is a set of instruction-response pairs capturing the behavior the model should produce by default.
Fine-tuning pipeline uses that dataset to shift the base model’s behavior toward the target domain.
API endpoint serves the resulting model, paired with the retrieval layer from newsletter 3, for use cases where the domain also requires fresh facts.
Requirements
System must produce outputs in the domain style and format without per-query examples, and apply domain reasoning without Chain-of-Thought prompting on every call.
It must refuse out-of-scope questions in a calibrated way rather than attempting an answer.
The entire training run must be reproducible, with a locked dataset, locked random seeds, and a versioned model card.
A production ship would also verify that general capability regresses by less than 5% on Massive Multi-task Language Understanding, Professional version (MMLU-Pro) or MT-Bench, and that inference cost is at least 50% below the foundation-model baseline. Both gates are described in Parts 3 and 4 but are not run in Part 5’s walkthrough.
One advanced feature covered later: preference tuning with Direct Preference Optimization (DPO), used to correct behaviors that Supervised Fine-Tuning (SFT) cannot reach reliably.
What this system does not do
It does not call tools or take actions, which is the focus of a future newsletter.
Also, it does not guarantee factual freshness, because the model’s training data is fixed at the point of fine-tuning. For current facts, RAG is still the right layer to add.
Deployment
Part 5 builds the open-weight path and documents the OpenAI fine-tuning path as a hosted alternative.
The open-weight path uses a base model such as Llama, Qwen, or Mistral with LoRA adapters served on vLLM1. The hosted path uses the OpenAI fine-tuning API.
Model selection
The right starting point is the smallest base model that already understands the domain.
Here are three criteria to look for:
Run a 20-question domain probe on the base model; if it cannot answer roughly half correctly, fine-tuning will not recover it, because fine-tuning shifts defaults and does not install knowledge the model never had.
Confirm fine-tuning support: on the open side, Llama, Qwen, and Mistral all support it; on the hosted side, OpenAI API currently exposes GPT-4.1 and GPT-4.1-mini for fine-tuning, since the GPT-5.x reasoning tier is not available for SFT at the time of writing.
Check license fit: Llama community license, Apache 2.0, and the OpenAI Terms of Service each impose different constraints on where the model can be hosted, which determines whether the model can run where your data lives.
Start in the 7B-8B parameter range and scale up only if the domain probe requires it.
Model choice is the last decision you make before training begins.
The next part covers what training actually does to the model you picked…
Part 1: How Fine-Tuning Changes a Model
Pretraining runs next-token prediction over trillions of tokens of text scraped from the internet, books, and code repositories.
By the end of the process, the model has compressed a broad representation of human language and knowledge into its weights. This is the base model: capable across many tasks, specialized in none.
Fine-tuning starts from that base and runs the same training process again, but on a much smaller dataset, typically a few thousand examples, chosen specifically to push the model’s behavior in one direction. The model is not learning language at this stage. Instead, it is learning which of its existing patterns to apply by default.
i.e., fine-tune for behavior, retrieve for knowledge.
Fine-tuning works by reinforcing patterns across many examples until they override the base model’s defaults. A reasoning pattern or output format repeated across thousands of training examples will stick.
A fact that appears once or twice will not, because the model has no reason to weigh it more than the vast body of knowledge it acquired during pretraining.
What Gets Updated During Training
Fine-tuning adjusts every parameter in the model slightly, one example at a time, until its behavior shifts toward what the dataset shows.
The primary cost here is memory:
A 7B model has 7 billion numbers stored on the GPU. Training requires storing a second number alongside each parameter to track how it should change, roughly doubling memory usage to 28 GB.
The optimizer adds a few more bookkeeping values per parameter on top of that, pushing the total GPU memory to 60-80 GB.
Consumer GPUs have 16 to 24 GB of memory, which is why a full fine-tune of a 7B model used to require a cluster and cost tens of thousands of dollars. Parameter-efficient methods (covered later) reduce the requirement to a single 16 GB GPU and bring the cost down to around $10 per run.
The size of the GPU bill forces the choice of training regime; the next decision a team has to make…
Three Training Regimes
There are three ways to fine-tune a model, each using a different kind of dataset:
Continued pretraining runs the same kind of training the base model already underwent, but on raw text from a single domain rather than the open internet. There are no labels; the model just reads a lot of domain text and gets better at predicting the next word in the domain. The dataset is large, ranging from hundreds of millions to billions of words, and the point of the exercise is to teach the model the field’s vocabulary. BloombergGPT started this way with financial text, and Med-PaLM (a medical fine-tuning of Google’s PaLM, Pathways Language Model) started this way with medical text. You should reach for continued pretraining only when the base model barely understands your domain.
Supervised Fine-Tuning (SFT), introduced in newsletter 1, trains on a list of questions paired with the answers the model must produce. The model learns to imitate those answers. Typically, the datasets contain 1,000 to 50,000 of these question-answer pairs. SFT is the right starting point for almost every fine-tuning project, because most domain problems are about style, format, and reasoning patterns rather than missing knowledge.
Preference tuning, which includes both Reinforcement Learning from Human Feedback (RLHF) and DPO, trains on comparisons. For each question, the dataset contains two possible answers and a label indicating which is better. The model learns to produce answers closer to the preferred one. Datasets are smaller, typically 500 to 5,000 pairs. Preference tuning is used to correct behaviors that SFT cannot reliably teach: calibrated refusals, a consistent tone, and output formats that hold up even when users push back.
The practical order is to start with SFT, ship it, identify the specific behaviors that still need work, and then add DPO to address those cases.
Continued pretraining enters the picture “only” if SFT stops improving and the base model clearly does not understand the domain vocabulary.
SFT handles the bulk of the work in a typical project, and the question it raises immediately is how to run it without an 80 GB GPU. The answer is a class of methods called Parameter-Efficient Fine-Tuning (PEFT), which trains only a small fraction of the model’s parameters rather than all of them.
LoRA is the most widely used method in this class…
Low-Rank Adaptation (LoRA)
LoRA leaves the base model’s weights completely frozen and instead trains a small side network alongside it.
The side network is built from two small matrices, conventionally labeled A and B. At inference time, the output of the side network is added on top of the frozen base model’s output.
Only the side network is trained; the base model never changes.
This works because the change required to turn a general model into a domain expert is mathematically small.
You do not need to adjust every parameter in the model; you only need to apply a targeted correction to the parameters that matter. The side network learns the correction. Its capacity is controlled by a setting called the rank, typically 8 or 16, which determines how much the side network can express.
In those settings, LoRA trains fewer than 1% of the parameters full fine-tuning would touch and recovers 90-95% of the quality on most tasks. GPU memory drops by an order of magnitude because the optimizer only tracks the small side network, and the risk of degrading the base model remains low because its weights never change.
Yet LoRA has two trade-offs against full fine-tuning:
First, on most tasks, it matches full fine-tuning in quality, but not always at parity. On hard tasks, full fine-tuning is, on average, a few percentage points stronger.
Second, the side network has a fixed capacity ceiling set by the rank: a rank-16 adapter can only encode so much change to the base model. For large domain shifts, such as moving a general model toward dense medical or legal reasoning, LoRA hits this ceiling and stops improving with more data.
Full fine-tuning has no such ceiling, but it incurs the highest GPU cost and carries the greatest risk of the model forgetting its general capabilities.
It is worth considering only when LoRA has plateaued, and the GPU budget allows it.
Reminder: this is a teaser of the subscriber-only newsletter series, exclusive to my golden members.
When you upgrade, you’ll get:
Simple breakdown of real-world architectures
Frameworks you can plug into your work or business
Proven systems behind ChatGPT, Perplexity, and Copilot
Ready for the best part?
Quantized Low-Rank Adaptation (QLoRA)
Keep reading with a 7-day free trial
Subscribe to The System Design Newsletter to keep reading this post and get 7 days of free access to the full post archives.
 | A guest post by
|