When a smaller language model is the right call
Start from your task, not from a model. Build the evaluation set first, so "good enough" is a number agreed up front rather than a judgement call later.
Small language models get discussed as though the choice were ideological. It is not. A frontier model is the right default for almost everything, right up until one of three specific things stops being true.
The three reasons that actually justify it
- 01Cost at volume. At a few thousand calls a month, inference is a rounding error and you should not be reading this. At tens of millions, the arithmetic changes and a task-specific model can be an order of magnitude cheaper.
- 02Latency. Some interactions have a hard budget — an interactive loop, a keystroke-level suggestion, anything on-device. A smaller model that answers in 80ms can beat a better model that answers in 900ms.
- 03Data residency. Sometimes the data cannot leave your boundary, for regulatory or contractual reasons. A model you can run inside your own VPC or on the device is not an optimisation, it is the only option.
Notice what is not on that list: wanting to own the model, or the belief that smaller is inherently more elegant. Neither pays for the engineering.
Build the eval set before you choose anything
The most common failure is starting from a model. A team picks a promising open-weights model, fine-tunes it, and then argues about whether the output is good enough. That argument has no ending, because nobody agreed what "good enough" meant before they had a result to defend.
Write the evaluation set first. It converts a matter of taste into a number, before anyone is invested in a particular answer.
Ours is deliberately unglamorous: a few hundred real examples from the actual task, with the output we would accept, including the hard and ambiguous cases rather than a curated set of easy ones. It should hurt a bit to build.
Then agree the quality floor before benchmarking. Not "as good as possible" — an explicit threshold on an explicit metric, signed off by whoever owns the outcome.
Benchmark against a frontier baseline
Run the best available frontier model against your eval set. That is your ceiling and your reference point. Then compete against it with smaller candidates on your actual data.
Measure three things together, because optimising one in isolation is how projects go wrong:
| Measure | Why it matters |
|---|---|
| Quality against the floor | A cheaper model that misses the agreed threshold is not cheaper. It is broken. |
| Cost per thousand calls | At your real volume and prompt length, not a benchmark’s. |
| p95 latency | Under load, in your deployment, not on an idle GPU. |
Narrow tasks are where small models shine. Classification, extraction, routing, structured rewriting, constrained generation — tasks with a tight output space and plenty of examples. Open-ended reasoning across long context is where the gap is still wide and honest benchmarking will tell you so.
The recommendation is sometimes "do nothing"
If the smaller model clears the floor and the economics are real, ship it. If it does not, the honest recommendation is to stay where you are.
That is a real outcome, not a failed project. You end up knowing what your task costs, what quality you are actually getting, and at what volume the arithmetic would flip. You also keep the eval harness, which is the durable asset — it catches regressions on whatever you run next, frontier or not.
We would rather tell a client their current setup is fine and lose the engagement than fine-tune something that makes their product marginally worse and considerably harder to maintain.
Filed under