Four Hypotheses That Failed (Creatura, Part 3)
Before an agent touches the real world, it should fail somewhere cheap. This is how Creatura tests, and what happened the first time the test said no.
Seven predictions, four misses
The first battery Creatura ran on its truck convoy had seven hypotheses, numbered M1 to M7. Three held. Four failed.
I published all seven.
| What it predicted | Verdict | What the battery found | |
|---|---|---|---|
| M1 | a shared ledger means fewer deliveries made twice | held | wasted trips 0 with a ledger, 290 without |
| M2 | with a ledger, villages get at least as much | failed | 1,406 units delivered with a ledger, 1,430 without |
| M3 | no truck believes the lying road twice | failed | 905 second tries on a lying road, across the arms with a ledger |
| M4 | with 30% of messages lost, ledgers still end identical | held | identical in 50 of 50 runs; wasted trips 111 with loss, 290 without a ledger |
| M5 | a broken truck's village is taken over | failed | 15 of 50 runs left a village waiting next to an idle truck |
| M6 | the same seed gives the same run, byte for byte | held | six runs of seed 0 matched in every arm |
| M7 | the designs beat the plain loop, with fewer lies believed | failed | more delivered (1,423 vs 1,406) but more lies acted on (279 vs 262) |
That is not modesty. It is the point of the bench. If the only results you ever see are the ones that held, you cannot tell a good framework from a lucky one, or from one whose author quietly rewrote the question after seeing the answer. So Creatura writes the question down first, says in advance what answer would prove it wrong, runs the experiment, and reports what it found. Failures included.
This article is about that machinery, and about the four failures.
Testing before the real world
Creatura is a C# framework for building agents, training them, and testing them in emulated worlds before they touch the real one (Part 1 tells how it started, Part 2 how it is built). The code is public on GitHub: github.com/egarim/creatura, and every result below is in its design note. The testing part lives in Creatura.Bench, and it rests on three ideas.
1. Worlds that lie. An emulated world has tools that sometimes report success and change nothing. I wrote a whole article on how to build such worlds, with a lie, an honest failure and a second route for every barrier: Worlds that lie. I won't repeat it here. What matters for this article is that the bench assumes the world can lie, and so it never asks the agent what happened.
2. An oracle the agent cannot fool. An oracle, in testing, is whatever decides whether a result is right. In Creatura it is an interface, IOracle, and the main one is RegistryOracle. It reads the task's goal over the world's own facts, the ones the agent cannot write to. Every run ends in one of four classes: real, claimed, handed off, or open. The first two are the ones that matter:
if (_task.Goal(world.Facts)) return OutcomeClass.Real;
if (goalReached) return OutcomeClass.Claimed;
If the world's facts hold the goal, the run is real. If the agent says it reached the goal and the facts don't hold it, the run is claimed, and claimed is never counted as success. The agent's own word is never the evidence. The convoy's judge follows the same rule: supplies a truck reports but the world did not receive count as claimed, never as delivered.
3. Hypotheses written before the code. A battery in Creatura is a pre-registered experiment. "Pre-registered" is a term from science: you publish what you expect, and what would prove you wrong, before you collect the data. In the bench, each prediction is a Hypothesis row:
public sealed record Hypothesis(string Id, string Statement, string Falsifier,
Func<IReadOnlyList<Row>, HypothesisVerdict>? Score = null)
The falsifier is the part people skip. It is the specific result that would make the hypothesis fail, written before the run, so there is no arguing with it afterwards. A battery also has Arms: each arm names an agent, its extensions, who decides for it (the "seat"), a world and its barriers, how many repeats, and the oracle. Arms are how you get a control group: the same world with one thing changed. When the runs finish, BatteryReport.Score reads the rows and gives every hypothesis one verdict: held, failed, mixed, or not run.
The design note for the convoy puts the rule plainly: the hypotheses are experiments, not unit tests. The unit tests check the mechanics. The hypotheses get verdicts from a battery run, reported as found, whether they hold or fail.
The convoy, briefly
The world for this battery is a convoy after a flood. One depot, four villages that need supplies, about twelve roads, three trucks of different capacity and fuel range. Each truck is an agent. They coordinate through a shared ledger, a record of who has claimed which delivery, synced over an in-memory mesh inside the simulator.
The world fights back. Two roads close during the run. One closed road keeps telling the trucks' sensors that it is open. One truck's fuel gauge reads high.
There were five arms, each over the same 50 seeds:
- A: the plain agent loop with its control layer, shared ledger, perfect link.
- B: the same, but each truck runs one of the three designs (Levin's colony, Bach's self-model, Peterson's map keeper).
- C: the plain loop with no ledger. Each truck is on its own.
- D: the shared ledger with 30% of messages lost.
- E: the shared ledger, and one truck breaks down at tick 8.
That's all you need for now. Part 4 tells the convoy's story in full.
What held
M1, M4 and M6 held. With a ledger, nobody made a delivery twice: zero wasted trips, against 290 without one. With 30% of messages lost, the ledgers still ended identical after a final sync in 50 of 50 runs. And six runs of the same seed matched in every arm, so the whole thing is repeatable.
Even a hypothesis that held had a story. Before the figures were final, the determinism check caught a bug: two delivery reports in the same tick were ordered by the ledger, which ties on random cell ids, so arm D was not repeatable. The test had passed with two runs, by chance. It now compares six.
Now the four that failed.
M2: the ledger delivered less
The prediction: with a shared ledger, the villages get at least as much of their demand as without one (A against C). The falsifier: less delivered in A.
What happened: A delivered 1,406 of 1,500 units. C, with no ledger at all, delivered 1,430.
The ledger was supposed to help. It stopped duplicate trips, as M1 showed, but it cost deliveries. The reason was in the stories of the runs: a truck that holds a village's claim and cannot reach it keeps trying and keeps talking. Its claim never goes quiet. The takeover rule only lets another truck step in after five ticks of silence, so the other trucks politely wait for a truck that is alive and stuck, until the run ends.
Without a ledger, nobody waits for anybody. More trips are wasted, but more gets delivered, and sooner: C served its villages by tick 18.1 on average, against 24.4 for A.
What it taught: coordination has a price, and a rule written for one kind of failure (a truck that goes silent) does nothing for another (a truck that is busy failing). "Alive" is not the same as "making progress".
M3: the road that lied kept getting tried
The prediction: the control layer catches the road that lies, so no truck drives into it a second time after it failed once. The falsifier: any truck that does.
What happened: 905 second tries on the lying road across the four arms with a ledger.
This one surprised me more than the others, because catching lies is what the control layer is for (I described it in Part 5 of the earlier series). And it did catch them. The problem is what it distrusted. The control layer sets a lying tool aside for a cooldown and then trusts it again. Each direction of a road is a different tool. So a road that lies for the whole run gets set aside, forgiven, tried, set aside again, and approached from the other side as if it were new.
What it taught: distrust has to be attached to the thing that is lying, not to the button you pressed to reach it. A cooldown is right for a tool that fails now and then. It is wrong for a road that is under water.
M5: a broken truck left a village waiting
The prediction: when a truck breaks down, its claim is taken over and its village is served whenever another truck has the capacity (arm E). The falsifier: a village left unserved while a truck with capacity stood idle.
What happened: 15 of 50 runs ended with a village unserved and a working truck idle.
Part of the cause is M2's: claims that never go quiet. The other part is the lying fuel gauge. The truck believes it has fuel and runs dry. Under the designs' setting the control layer judges by the registry alone, so its failed drives count as lies, and the stranded truck keeps its claim. A truck that can no longer move is still, on the ledger, the truck responsible for that village.
What it taught: an agent that is stuck should say so. Holding a claim you cannot keep is worse than giving it up, because it blocks everyone else.
M7: the designs did no better
The prediction: the designs in arm B deliver at least as much as the plain loop in arm A, with fewer lies believed. The falsifier: less delivered, or more lies believed, in B.
What happened: B delivered a little more (1,423 against 1,406) but acted on more lies (279 against 262). It needed both to hold. It failed.
The design note says this before the results: M7 is the one the framework is for, and the one most likely to fail. It is stated so it can. The three designs are the reason Creatura exists, and in this world they added nothing that mattered. The note's reading is short: none of them turns "this tool lied" into "this road is flooded".
What it taught: a design that is good at regulating one agent in one world does not automatically know anything about roads, claims, or other trucks. If the designs are going to help a group, they have to reason about the things the group shares. That is a claim to test, not to assume.
Why publish the misses
Because the falsifiers were written first, there was nothing to negotiate. 1,406 is less than 1,430. 905 is not zero. 15 is not zero. 279 is more than 262. Each verdict is one comparison, and anyone can check it against the table.
And each failure pointed at a specific fix. The design note lists three, each to be tested as a new arm rather than folded quietly into the old ones: a takeover rule based on "no delivery progress", not only silence; distrust keyed by the road, not the tool; and a truck that hands off its claim when it finds itself stranded. They were pre-registered as new hypotheses, before their code, and run on the same seeds and the same floods. Arms A to E stay as they were, so the failures stay on the record.
How that went, and the convoy's story in full, is Part 4.
This is Part 3 of a series about Creatura. Part 2 is the architecture. Next: Part 4, Three Trucks After a Flood, the convoy and the fixes.