Before You Distill, Measure: Racing DeepSeek Against Opus on 36 Office-Document Tasks
In the last post I argued that a small model pointed at one narrow job beats renting a generalist for every call. Fair enough in theory. Then I went to do it, and the first thing I learned is that the step before distillation is the one everybody skips.
Because the assumption sitting underneath "let's distill" is the cheap model can't do this yet. And almost nobody measures that. They just feel it.
So we measured it. Here's what came back, and why we ended up not distilling anything at all.
The task: drive Excel, Word, and PowerPoint from plain language
The agent's job is unglamorous and very real: take an instruction — "add a Tax column at the right of the table using the rate in H1", "set every body paragraph to 11pt", "delete slide 3 and reorder the rest" — and produce a correct .xlsx, .docx, or .pptx.
Thirty-six tasks across the three formats. And the expensive part of this project was never the models. It was building something that could tell me the truth about them.
The harness has to prove itself before it can judge anything
Here's the rule I'd put on a poster: an eval you haven't validated is a random number generator with a spreadsheet attached.
So before any model runs, the harness self-tests with two baselines per format:
- The reference solver — the task's own known-good answer — must score 100%. If it doesn't, the grader is rejecting correct work, and every model number after that is noise.
- The no-op solver — which literally copies the input file to the output — must score 0%. If it doesn't, some task's answer was already sitting in its input, and passing it proves nothing.
That second check sounds paranoid. It caught two real bugs during construction, both the same species: a task that graded something it never actually observed, so doing nothing scored as success. One Excel task sometimes generated inputs with no rows to delete. One Word task asked for bolded table cells while grading only paragraph text — and python-docx excludes table content from doc.paragraphs, so no check could ever see it.
Either bug would have silently inflated every score that came after. Nobody would have noticed. The numbers would just have been nice.
Two more properties that made the results mean something:
Three variants per task, all must pass. Each seed regenerates the input with different sizes and ordering — Excel shuffles column order, PowerPoint varies slide count. A solver that hardcodes a cell coordinate passes seed 0 and dies on the other two, and the runner labels that case specifically instead of reporting a generic failure. Locate by name, never by index.
BLOCKED is not FAIL. python-pptx cannot render, so it cannot detect text overflowing its box. Rather than guess, those checks report BLOCKED and drop out of the denominator. A correct answer the harness can't verify never counts against the model. Honest ignorance beats a confident wrong number.
The race
Same prompt builder. Same grader. Same 36 tasks. The only thing that changed was the engine behind the code generation.
| Format | Opus 4.8 (teacher) | DeepSeek-V4-Pro (student) |
|---|---|---|
| Excel | 94% (15/16) | 94% (15/16) |
| Word | 90% (9/10) | 90% (9/10) |
| PowerPoint | 100% (10/10) | 70% (7/10) |
Parity on two formats out of three. Not "close enough." Identical.
The 19% that wasn't real
The first DeepSeek Excel run came back at 19%, and I very nearly wrote that number down as the answer.
It wasn't a capability gap. The codegen prompt hands the model two lines — INPUT_PATH = ... and OUTPUT_PATH = ... — and DeepSeek routinely dropped them from its output while still referencing the variables. Every script died on a NameError before touching a single cell. I was measuring whether the model echoes boilerplate, not whether it can do spreadsheet logic.
Re-inject those two lines from the prompt, and 19% becomes 94%.
There was a second bug of the same species: the gateway rate-limits, and an un-retried HTTP 429 comes back as an empty response, which grades as a failure. Without exponential backoff, the "student score" is partly just the rate limit wearing a model's name.
If your first measurement says the cheap model is terrible, your first suspect is the harness. I'd guess a good share of the "we tested it, the small model can't do it" conclusions out there are one of these two bugs, and nobody went back to check.
The gap was a missing library API, not intelligence
DeepSeek's three PowerPoint misses cluster tightly: setting a title's font size, and slide delete/reorder. Both are exactly the operations where python-pptx has no clean API and you have to reach into the raw XML and manipulate sldIdLst by hand.
That's not a reasoning gap. That's "has anyone ever shown you this specific incantation." Which is precisely the kind of thing a couple of worked examples fixes — and precisely why the finding didn't lead where I expected it to.
So we didn't distill. We routed.
Excel and Word code generation run on the cheap engine at parity. PowerPoint keeps the expensive one. That's a config change — one line, no dataset, no training run, no GPU bill, reversible this afternoon.
Distillation is a real tool and I stand by the last post. But it's an optimization with days of setup behind it, and it earns its place only after the cheaper moves are exhausted. The ladder runs:
- Measure whether the cheap model is actually worse. Often it isn't.
- Route — cheap model by default, expensive one only for the cases that genuinely need it.
- Teach in-context — show the small model worked examples for its failing cases.
- Then distill, if the volume still justifies it.
We stopped at rung 2 and got most of the win. Then I went and ran rung 3, and it's worth its own section — because it did not behave the way "just add examples" implies.
Update: I ran rung 3, and it isn't a free patch
Same gateway, same grader, same ten PowerPoint tasks × three variants. The only change was a short python-pptx cookbook injected into the prompt ahead of the task.
Discipline first, because it decides whether the number means anything: the cookbook teaches library primitives only — that prs.slides is a collection and can't be sliced, that you reach a title through slide.shapes.title rather than guessing a shape name, that slide order lives in the _sldIdLst element, and that editing run-by-run preserves formatting. It demonstrates none of the graded tasks. No delete-by-title, no alphabetical sort, no uppercasing, no font size. The model still has to work out what to locate and what transform to apply. Anything stronger would measure copying, not teaching.
The headline: 7/10 → 8/10.
The headline is also misleading, which is the interesting part:
| Task | Control | + cookbook | |
|---|---|---|---|
uppercase_titles |
FAIL | PASS | fixed |
delete_risks_slide |
FAIL | PASS | fixed |
append_next_steps |
PASS | FAIL | regressed |
sort_content_slides |
FAIL | FAIL | unmoved |
Two fixed, one broken. The two fixes are exactly what I aimed at: uppercase_titles had been crashing on prs.slides[1:] plus a hardcoded shape.name == "Title 1", and delete_risks_slide needed the _sldIdLst primitive.
The regression is the finding. append_next_steps has nothing to do with anything I taught — it appends a slide — and it died on this:
AttributeError: 'SlidePlaceholder' object has no attribute 'has_placeholder'.
Did you mean: 'is_placeholder'?
A hallucinated API. My cookbook mentioned reaching the title "through the placeholder API," and that phrasing appears to have nudged the model into placeholder-hunting code on an unrelated task, where it invented an attribute that doesn't exist. The control run solved that same task cleanly and never went near placeholders.
Two caveats I have to own. First, the failure set is not stable between runs — the earlier baseline recorded title_font_size failing, while today's control passes it and fails uppercase_titles instead. Same 7/10 total, different tasks. At ten tasks, ±1 is noise, so read "+1" as suggestive, not established. Second, I only re-ran PowerPoint; Excel and Word were already at parity and had nothing to teach toward.
So the first lesson is not "in-context examples work." It's that context is not additive. Adding guidance fixed what it targeted and quietly damaged something it never mentioned — and the only reason I know that is the harness re-ran all ten tasks instead of just the three I cared about. Measure the whole suite after every prompt change, or you will ship a net regression and call it an improvement.
Which is the same lesson as the 19% earlier in this post, wearing a different hat.
Chasing it further: the gap was never reasoning
I kept going, because one task was still failing and I wanted to know why.
sort_content_slides turned out not to be a sorting bug at all. Seed 0 produced the exactly correct order. Seeds 1 and 2 crashed before saving anything:
TypeError: unhashable type: 'Slide'
python-pptx defines __eq__ on Slide, which sets __hash__ to None. The model had built a {slide: element} lookup — perfectly reasonable-looking code that can never run. So I added a sixth cookbook fact: slides are unhashable, track them by position.
It worked. sort_content_slides flipped to PASS. And the score didn't move, because add_chart_slide — which had passed twice — now failed on add_chart() called with the wrong number of arguments.
That's when I stopped adding facts and went back through every failure in all three runs:
'list' object has no attribute 'rId'
'Slide' object has no attribute 'rId'
'Presentation' object has no attribute 'presentation'
'PresentationPart' object has no attribute 'element'
unhashable type: 'Slide' (×3)
'SlidePlaceholder' has no attribute 'has_placeholder' (×2)
add_chart() missing 1 required positional argument
Every single failure was a crash on the library's API surface. Not one was faulty logic. The model always knew what to do; it kept guessing wrong about how to call it. And the cookbook reliably killed whichever error it named, whereupon the model reached for a different wrong call. Prompt facts were never going to converge on that — python-pptx has a bigger API than I have patience.
Look again at that traceback list, though. CPython is telling you the answer: "Did you mean: is_placeholder?" So instead of teaching more API, I ran the script, and on a crash handed the traceback back exactly once.
| Score | |
|---|---|
| Control | 7/10 |
| + cookbook | 8/10 |
| + cookbook, revised | 8/10 |
| + one repair retry | 9/10 |
Two repairs attempted. Both cleared on the first try.
The contract survives: the retry sees only the Python traceback. Never the grader, never the reference solver, never the expected output. It is fixing its own crash, not iterating toward the answer — which is precisely what a production agent harness does for a model, and precisely what a one-shot benchmark withholds.
And the last remaining failure is finally something worth calling a capability gap. append_next_steps on seed 2 runs cleanly and produces a structurally wrong deck:
answer : ['Next Steps', 'Confirm budget\nSchedule review']
produced : ['Confirm budget\nSchedule review', '']
Bullets written into the title placeholder, title left blank. Seeds 0 and 1 are perfect. No retry loop catches that one — nothing crashes. Only a grader finds it.
So the corrected picture of the "PowerPoint gap": 9/10 against the teacher's 10/10, and almost all of what looked like a capability difference was a cheap model guessing at an unfamiliar API with no chance to see the error message. Give it the error message and it fixes its own work.
If I'd distilled here, I'd have spent days teaching a student to imitate a teacher, when what it actually needed was permission to read its own stack trace.
The thing I'd take away isn't the scoreboard. It's that we almost skipped straight to training a model to fix a problem that turned out to be two missing lines of boilerplate and a retry loop — which, by the end, is exactly what it turned out to be, twice over.
Build the ruler first. It's cheaper than everything downstream of it, and it's the only part that tells you the truth.