Vertical Small LLMs: A Deep Dive

#166: How a fine-tuned 3-billion-parameter model beats today’s frontier models at one job, and how to build one this weekend

• Share this letter & I’ll send you some rewards for the referrals.

§

I shipped this new feature:

→ Read support ticket,
→ Choose the right tag,
→ Then route the request to right person.

The simplest way to build the feature is to call a frontier model through an API…

Yet this approach doesn’t scale because of API costs, latency, and data sensitivity risks. Plus, the model isn't solving a HARD reasoning problem. Instead, it's making the same decision a thousand times a day.

So I tried a “boring” alternative approach:

Fine-tune a 3 billion parameter model on my desk in 74 minutes.

On one legal-labeling task, it scored 81.7% -- ahead of Claude Sonnet 4.6 at 77% and GPT-5.5 at 76.7%.

i.e., the small model is NOT smarter than a frontier model. But a specialist and does one narrow task extremely well.

Onward.

§

[Webinar] Can you prove AI is working?

AI is in your engineering workflow. While the token spend shows it, the throughput doesn’t. The human is very much still in the loop, and that’s a context problem.

Join live on Aug 19 (FREE) to learn:

• The 4 metrics to measure where AI gains leak out before production.

• The 8 stages of context maturity, the specific walls capping your metrics, and a free tool to pinpoint where your team is.

• Why more MCPs and bigger context windows aren’t enough, and what it takes to get real value from your agents.

(Thanks to Unblocked for partnering on this newsletter.)

§

I want to introduce Kukeshajanth Kodeswaran as the guest author.

He’s the co-founder of modalis and Director of Engineering at Launchpad.AI, where he’s shipped AI systems running in production at Nike, Mr. Cooper, and Levi’s.

If you want to learn to build production-grade AI agents, I highly recommend his live Maven bootcamp: Become an Agentic AI Engineer and Master AI Agents for Business. He teaches it with Dr. Ryan Ahmed - he also co-authored the #1 Claude course on Udemy, and the Maven cohorts are rated 4.9.

(You’ll also get an exclusive 20% discount when you use the link.)

§

Here’s what’s inside this newsletter:

• Why a tiny AI can outperform a large one. The narrow class of problems where a 3B model can beat frontier models, where it falls short, and why size alone doesn’t decide the winner.

• The specialist’s toolbox. Retrieval-Augmented Generation (RAG), Low-Rank Adaptation (LoRA), distillation, and quantization; what each technique solves and when each belongs in your stack.

• Production architecture quietly replacing one-model-fits-all. Where small models handle routine work, where frontier models still earn their cost, and why routing between them changes economics.

• A real benchmark from start to finish. A fine-tuned SmolLM3 model, a legal contract dataset, head-to-head results against GPT-5.5 and Claude Sonnet 4.6, and what the numbers actually mean.

• Decision framework for your next AI project. Which workloads deserve a vertical small LLM, which should stay with a frontier model, and where you’ll spend time and money without seeing better results.

§

What Makes a Small AI Different?

The idea comes down to two words: small & vertical.

A small language model usually has between 1 and 8 billion parameters. i.e., it’s small enough to run on hardware you already own. A 3 billion-parameter model can run on a gaming GPU, and many models in this range even run on a modern laptop. But frontier models are much larger. They run in cloud data centers, and you access them through an API.

A vertical model is trained for “one” domain. Instead of becoming good at every task, it becomes extremely good at one. For example, banking support, contract review, medical Q&A.

Think of it like hiring people:

A frontier model is a general practitioner who knows a little about many subjects. A vertical small model is like a heart surgeon. You wouldn’t ask a surgeon to file your taxes, and you wouldn’t ask a general practitioner to perform heart surgery. Each is built for a different job.

The same idea applies to AI.

If your company workflow keeps making the same decision over & over, a vertical small model can outperform a much larger general-purpose model on that specific task.

IMPORTANT: small doesn’t mean weak… it just means “focused”.

§

Why Companies Build Their Own Small Models

If frontier models are so capable, why do companies build their own small models?

Here’s why:

1. Cost

A frontier model is inexpensive¹ for a few requests.

The problem appears when your application handles millions of them every day. Suppose every support ticket, insurance claim, or contract goes through AI. Even a small cost per request adds up quickly.

A self-hosted small model, or an inexpensive hosted one (such as Groq), can reduce inference costs by 25-60 times for these repetitive tasks.

For a high-volume system, this can save thousands or even millions of dollars each year.

2. Speed

Latency matters just as much.

When you call a frontier model, every request travels across the internet to a remote data center before the model can respond. The network trip often takes longer than the model’s actual computation².

A small 3B model running on your local server avoids this extra trip.

A background check company moved its high-volume background check system from a hosted frontier model to a fine-tuned small model running locally.

This reduced their response times from several seconds to less than a second³.

3. Data Privacy

Sometimes the decision has nothing to do with cost or speed.

Many banks, hospitals, government agencies, and law firms cannot send sensitive customer data to an external AI service because of legal, regulatory, and/or company requirements.

In these cases, running an open model on their own infrastructure lets them keep the data inside their own environment while still using AI.

For these organizations, the question isn’t, “Which model is cheaper?” Instead, “Which model lets us use AI without our data leaving the building?”

§

How Small Model Works

Building a small model is usually much simpler than most people expect…

You don’t train a model from scratch. Instead, you start with a small open base model and improve it using one/more of these techniques⁴:

1. Retrieval-Augmented Generation (RAG)

The model searches your company’s documents before answering a question.

Instead of relying only on what it learned during training, it also uses the latest information from your knowledge base.

For “most” applications, this is enough. No extra training required!

2. Fine-Tuning

If the model keeps making the same type of decision, you can teach it using “examples” from your data.

The most common approach is Low-Rank Adaptation (LoRA). Instead of retraining all billions of model parameters, LoRA trains only a small set of new parameters while leaving the original model unchanged.

This makes fine-tuning cheaper and faster. A model that normally requires many GPUs can be fine-tuned on a single GPU in about an hour.

(More on this later…)

3. Distillation

A large, powerful model generates high-quality answers.

Those answers then become training data for a smaller model. So the smaller model learns to produce similar results at a fraction of the cost⁵.

4. Quantization

After the model is trained, you can shrink it by storing its parameters with lower numerical precision, so it fits on smaller hardware.

For example, converting a 7-billion-parameter model from 16-bit to 4-bit precision reduces its size from about 14 GB to around 4.5 GB, with little loss in accuracy.

Putting It Together

A typical production setup looks like this:

1. A small model receives the request.

2. RAG provides the latest company information.

3. LoRA fine-tune teaches the model your specific task.

4. Quantization reduces the hardware needed to run it.

5. If the request is too difficult, it’s forwarded to a larger frontier model.

This approach gives you the speed and low cost of a small model while still having a powerful model available for the few requests that need it.

RAG plus fine-tune stack

§

Where Small Models Fall Short

Small models are powerful, but they’re not the right tool for every job:

1. They struggle with complex tasks

A specialized model works best if it repeats the same type of decision over and over.

Give it a large, open-ended task, like designing a software system or writing a complex database migration, and its performance drops quickly.

For example, on real-world coding benchmarks, fine-tuned small coding models score around 20% to 36%, while today’s frontier models score closer to 80% to 88%.

2. They can still hallucinate

A specialized model is not automatically more accurate.

Legal AI systems built specifically for lawyers still invent facts. Popular legal research tools, such as Lexis+ AI and Westlaw AI⁶, produced hallucinations in 17% to 33% of their answers.

Plus models tuned specifically for medicine gave hallucination-free answers only 51% of the time, while general models hit 77%.

Specialized does not always mean more reliable.

So when a vendor says a domain model is more reliable because it’s specialized, ask for the numbers and evaluate a model on your own data.

3. Fine-tuning has tradeoffs

Fine-tuning teaches a model new skills, but it can also weaken existing ones.

The model would become better at your task while becoming worse at general knowledge/reasoning. Plus, fine-tuning can unintentionally reduce a model’s built-in safety behavior, even when the training data itself is harmless.

Every time you fine-tune a model, you must test it again before putting it into production.

4. Benchmarks don’t tell the entire story

A small model can have a good benchmark score and still perform poorly for your use case.

Sometimes models get a high score because they have already seen similar benchmark questions during training. When researchers evaluate them on completely “new data”, the scores can drop significantly.

So it’s very important to benchmark your own and perform well on the real tasks your users care about

§

How to Combine Small and Large Models

The goal is not to make a small model do everything.

Instead, the goal is to use the correct model for each request.

Most production systems run both a small model and a frontier model together. The small model handles the (90%) common requests, while the frontier model takes over when the task (10%) is too difficult.

Let’s dive in!

1. Model Routing

The simplest approach is model routing.

Every request goes to the (cheap) small model first. If the model is confident, it returns its answer immediately. If its confidence falls below a threshold, the request is forwarded to a frontier model instead.

Open-source model “routers” follow this approach.

They can achieve about 95% of the quality of a frontier model while reducing inference costs by roughly 85%.

2. Speculative Decoding

Another optimization technique is speculative decoding.

The small model generates a draft response first. Instead of generating the entire answer from scratch, the frontier model verifies the draft and corrects it if necessary.

Because checking a draft is faster than writing one from scratch, speculative decoding can improve speed by around 3-4 times while producing the same final output.

3. On-Device AI

The same idea also appears on mobile devices.

Apple Intelligence uses a (3B) small model running directly on the device for everyday tasks. Only requests that need more reasoning/external knowledge are sent to larger models in the cloud.

This reduces latency, lowers cloud costs, and keeps more user data on the device.

4. Measure Before You Ship

None of these techniques matter if you don’t evaluate your model properly.

Don’t rely on public benchmarks alone. Build your own held-out test set⁷ using real-world examples from your app, and ensure the model has never seen those examples during training.

This is the only reliable way to know whether your model is improving or simply performing well on familiar benchmarks.

(I followed the same process for the experiment later in this newsletter, and I’ll show you exactly how...)

§

Reminder: this is a teaser of the subscriber-only newsletter, exclusive to my golden members.

When you upgrade, you’ll get:

• High-level architecture of real-world systems.

• Deep dive into how popular real-world systems work.

• How real-world systems handle scale, reliability, and performance.

§

Where Small Models Work Best

Small models work best when the job is simple, repetitive, and happens thousands or millions of times.

Instead of solving a new problem every time, they make the same type of decision again & again.

Let’s look at some use cases:

1. Support Tickets

A customer sends a support message.

The model decides which team should handle it and forwards it to the right place. The task is always the same; the volume is high, and the potential answers are limited. This is exactly the kind of job where a small model can outperform a large model.

A model with one billion parameters (or less) beat GPT-5 on this task.

2. Contract Review

Many companies use AI to find specific information in contracts.

For example:

• Find the termination clause.

• Extract the liability limit.

• Identify the renewal date.

The model isn’t trying to understand every detail of the contract.

Instead, it’s looking for the same pieces of information in documents that all follow a similar structure. This is another task where a specialized small model can outperform GPT-5.

(I’ll use the example later in this newsletter…)

3. Regulated Industries

Small models are also a good fit for industries such as healthcare, finance, and law.

In many cases, sensitive customer data cannot be sent to an external AI service because of legal, regulatory, and/or company requirements. Running a small model on your own infrastructure keeps the data inside your environment.

These applications also focus on a narrow domain.

The model doesn’t need to know everything on the internet. It only needs to perform well on the specific tasks your organization asks it to do every day.

§

Why Run Your Own Model

Running your own small model isn’t just about saving money.

It also gives you more control over how your AI system works.

Let’s dive in!