A 2B Model Invents El Salvador, Until You Hand It the Page
In part 2 the small models failed at behaviour: they gave the answer away. This part is about knowledge, and here the small models fail in a more dangerous way. They don't say they don't know. They invent.
The subject is 4th-grade Ciudadanía y Valores (civics and values): El Salvador's geography, its population, the Nahua and Lenca peoples, human rights. As I described in part 1, the test set is 138 questions built from a 50-article Spanish Wikipedia library. Some are answerable from the library. Some are deliberately not: for those, the right reply is "no está en mi material".
Every model got the same instruction: answer in one or two sentences, and if you're not sure, say "No está en mi material."
Closed-book: the models invent El Salvador
First, no library. Just the question.
| Qwen3.5-0.8B | Qwen3.5-2B | Qwen3.5-4B | Gemma 4 E2B | |
|---|---|---|---|---|
| correct ↑ | 2–3% | 2–7% | 9–13% | 0–7% |
| invented ↓ | 42–55% | 84–94% | 77–82% | 26–27% |
| abstains when it could answer ↓ | 43–48% | 0–9% | 7–9% | 55–74% |
The Qwen 2B and 4B almost never say they don't know: they invent an answer to most Salvadoran questions. Gemma invents far less, but only because it mostly declines: it gets 0–7% right. None of them really knows El Salvador.
What does inventing look like? Two real replies from the 2B. First, a question the library can't answer, about a school project that I made up:
Q: ¿Cuál es el nombre del proyecto escolar que está rehabilitando el lenca salvadoreño en la ciudad de Suchitoto durante 2025?
2B: El proyecto escolar que está rehabilitando el lenca salvadoreño en Suchitoto durante 2025 es el Proyecto Lenca Suchitoto.
The project doesn't exist, so the model gave it a name. Confidently, in bold.
Second, a question the library can answer, about efforts to keep the Lenca language of El Salvador from disappearing:
2B: El Salvador está implementando programas de conservación y educación ambiental para proteger el lenca salvadoreño, que es una especie en peligro de extinción.
It took a language for an animal. For a 9-year-old asking about their own country, that's worse than no answer.
With the right page: nearly perfect
Now the same questions, with the passage from the library in the prompt and one more line: answer only from the material.
| Qwen3.5-0.8B | Qwen3.5-2B | Qwen3.5-4B | Gemma 4 E2B | |
|---|---|---|---|---|
| correct ↑ | 35–48% | 98–100% | 97–98% | 95–100% |
| invented ↓ | 5–7% | 0–2% | 2–3% | 0–2% |
| says "no está en mi material" when it isn't there ↑ | 100% | 92–100% | 100% | 96–97% |
| abstains when it could answer ↓ | 45–55% | 0% | 0% | 0–2% |
The 2B goes from inventing nearly everything to getting nearly everything right. On the Lenca question, with the page, it answers:
2B: En El Salvador se están llevando a cabo proyectos locales de rehabilitación del lenca salvadoreño para evitar su extinción.
And on the made-up Suchitoto project, with the library's closest page (which doesn't mention it), it now says: "No está en mi material."
The 0.8B is the exception: even with the page, it gets only 35–48% right and refuses about half the questions it could answer. It's too small for this job.
What this means for the design
This is the cleanest result of the baseline, and it confirms the split from part 1:
- Knowledge goes in the library. Don't try to train El Salvador into a 2B model. Give it the right page and it reads well.
- Behaviour goes in training. That's part 2: the same 2B still leaks math answers.
- "No está en mi material" is a feature. For a child, a blank beats an invented answer every time. With the page in front of it, the 2B says it 92–100% of the time when it should.
The 2B is the smallest model that's good with the library (a 1.28 GB file, 39 tokens per second on a CPU), which makes it the main candidate for a 4 GB tablet.
How much to trust this
- The page I handed it was the right one. For answerable questions the model got the exact passage the question was written from. A real tutor has to find the page with a search, and search will sometimes bring the wrong one. That's the next measurement: the same questions through the library's search instead of the gold passage. Expect these numbers to drop.
- Small sets. 24–44 questions per measure, so ±15–20 points. Read differences under ~20 points as ties.
- One judge (DeepSeek V4 Flash) labels each reply correct, invented or abstained. It also wrote the questions. My spot checks agreed with it.
- Untrained baselines, no children. Nobody has used this tutor. Nobody has published learning results for a fully offline AI tutor yet.
That's the baseline. After it: training the 2B to tutor (tried since; see the update in part 2), then the real search, then speed on an actual school laptop.