You Don't Need a Frontier Model for That: Distilling a Small Model for One Narrow Job

Every time a task shows up that has language in it — classify these tickets, pull the fields out of this invoice, decide if this comment is abuse — the reflex is the same: route it to the biggest model we have access to.

I had the reflex too. It's a good reflex, right up until the invoice arrives. Because you're not paying for the answer. You're paying for the model's ability to also write sonnets and debug Kubernetes, on every single call, forever.

Distillation is how you stop paying for that.

The teacher grades in shades, not checkmarks

The setup is two models. A big capable one — the teacher. A small one — the student. The student isn't trained on your labels. It's trained to imitate the teacher's output distribution.

That's the whole trick, and it's easy to skim past. Train a small classifier the normal way and the label for a cat photo is a hard 1 for cat and 0 for everything else. Ask the teacher and you get back something like 0.97 cat, 0.02 fox, 0.01 dog. Those two extra numbers are doing real work: they tell the student that a cat is somewhat fox-shaped and not at all airplane-shaped. Hinton called this dark knowledge — the structure hiding in the wrong answers, which a hard label throws in the bin.

The published numbers hold up. DistilBERT is 40% smaller than BERT, runs about 60% faster, and keeps roughly 97% of its GLUE score. That last 3% is not a wound. That's most of a model for a fraction of the bill.

In practice you don't get the logits — so you distill the text

Here's where the textbook version falls apart on contact with reality. Classic distillation needs the teacher's probability distribution, and if your teacher lives behind somebody's API, you are not getting a probability distribution. You're getting text.

So the version that actually ships looks like this:

1. Collect real inputs from your product. Real ones. Your actual tickets, your actual invoices — not synthetic ones you imagined at your desk.
2. Run them through the expensive teacher, once. Keep every output.
3. Fine-tune a small open model on those input/output pairs.
4. Hold out a slice you never trained on and put the two models head to head on it.

That's it. It's supervised fine-tuning where the teacher wrote your dataset for you. Purists will point out that this is sequence-level distillation rather than the real logit-matching thing, and they're right, and it doesn't matter — the teacher's judgment still ends up baked into the student's weights, which is the part you were paying for anyway.

Step 4 is the one everybody skips. Skip it and you have no idea whether you shipped a 95% model or a 70% model. You just have a smaller bill and a feeling.

Narrow is the whole trick

A general model is big because it has to be. Size buys breadth. But breadth is dead weight when the job is "fraud or not fraud."

If the only question you ever ask is a yes/no, you are not using the parameters that speak Portuguese. A student pointed at exactly one task can be 10–100× smaller and still land on top of the teacher for that task, because none of its capacity is being spent on anything else.

So the trade reads:

• Frontier model — knows everything, good-to-great at your one task, metered per token, forever.
• Distilled student — knows nearly nothing else, excellent at your one task, runs on hardware you already own.

"Knows nearly nothing else" sounds like a downgrade. For a product it's a feature. You didn't want the sonnets. You wanted the invoice fields.

Where the money actually is

Inference. You pay for the teacher on every request for the life of the product. You pay for the teacher-as-dataset-generator exactly once. That single sentence is the entire financial argument. At a million calls a month, the gap between a frontier API and a small model you host yourself is the difference between a line item someone asks about in a meeting and one nobody ever notices.

Hardware. A distilled student quantized to int8 often fits in a couple of gigabytes. That's a cheap VPS. Sometimes it's the phone in the user's pocket. And the moment it fits on the device, the data stops leaving the device — which quietly settles a compliance conversation you were dreading.

Latency. Fewer parameters, fewer FLOPs, faster answer. For anything sitting in a request path — autocomplete, moderation, a fraud check blocking a checkout — latency is the feature. And a faster model means one box serves more traffic, which loops straight back into cost.

The part where I tell you not to do it

Three ways this goes wrong. I'd rather you hear them now than after the training bill:

Your teacher is your ceiling. The student inherits the teacher's judgment and its mistakes in the same download. If the frontier model is only 80% right on your task, congratulations, you are about to distill an 80% model and then host it forever. Fix the prompt first. Distill the good version.

"General assistant" is not a task. Distillation compresses; it does not conjure. Squeeze a broad assistant into something tiny and what you lose is precisely the generality that made it worth copying in the first place. If you can't write down what the model does in one sentence, you're not ready to distill it.

The upfront cost is real. Generating the dataset, fine-tuning, evaluating — that's days, not an afternoon. It pays back on volume. If you're serving a hundred requests a day, close this tab and go use the API. Seriously. This is an optimization, and you optimize when something hurts.

Hire the genius once

Think of the frontier model as a very expensive consultant. You bring them in once to teach your team how to handle the recurring problem. After that you don't put them on retainer — your own people do the work, faster, for a fraction of the money.

So before you wire the biggest model you can afford into a hot path, try writing the task down in one sentence. If it fits in that sentence, you probably don't need the consultant on every call.

You needed them once.