System One in C#: Testing Jev, and Turning Any Local Model Into One

A while back I wrote about Jev, the model that answers in types instead of writing JSON. It's a "System One" model: you hand it text and a schema of choices, and it returns one of your options with a probability — no JSON string to parse, nothing malformed possible. Neat idea. But two questions kept nagging at me as a .NET dev:

How do I actually test this from C#? It's a paid, early-access, network model. I'm not going to hit the real API in every unit test.
Do I even need Jev? The magic isn't really the model — it's the constraint. Could I get the same "pick one of my options, give me a probability" behaviour out of a local model I already run?

The answer to both turned out to be the same small thing: one interface. I put it in a repo — github.com/egarim/systemone-deciders — and this is the walk-through.

The one insight: System One is a harness, not a model

Here's the reframe that makes everything simple. "System One" is not a category of model you have to buy. It's a way of calling a model: never let it free-write; force the output to be one of your enumerated options; read the probability distribution over them. Jev is one implementation of that idea, built as a purpose-made model. A local model wrapped in a grammar is another. They can sit behind the exact same contract:

public interface IDecider
{
    Task<Decision> ChooseAsync(string state, string question,
        IReadOnlyList<string> options, double abstainBelow = 0.0, CancellationToken ct = default);
}

A Decision is the chosen Value, its Confidence, the full distribution over Options, and an Abstained flag — set when the top probability falls below your threshold, so you can escalate the uncertain cases to a human or a bigger model. That abstain path is the whole reason calibrated confidence matters: a cheap decider that knows when it doesn't know is worth far more than one that's always sure.

Once that interface exists, "test Jev" and "use a local model" stop being two problems and become two implementations.

Engine 1: Jev, and how to test it without hitting the network

The Jev implementation is a thin HTTP client: POST https://api.typesafe.ai/v1/systemone, model jev-latest, the question declared as a choice over your options. (There are community SDKs — SystemOneDotNet targets .NET Standard 2.0 so it runs in XAF/.NET Framework too — but a hand-rolled HttpClient keeps the dependency count at zero and the parsing under my control.)

The important part is the testing strategy, because a model behind a network is exactly the thing people test badly. Split it in two:

Unit tests — hermetic, no network, no key. The trick is to inject a stub HttpMessageHandler that returns canned JSON. Now you're testing the real JevDecider — its request-building and response-parsing — with zero network:

[Fact]
public async Task Maps_options_shape_response_to_typed_decision()
{
    var canned = """
    {"answers":{"decision":{"options":[
        {"option":"billing","probability":0.94},
        {"option":"technical","probability":0.05},
        {"option":"sales","probability":0.01}]}}}
    """;
    var jev = new JevDecider(new HttpClient(new StubHandler(canned)), apiKey: "test-key");

    var d = await jev.ChooseAsync("I was charged twice", "Which team?",
                                  ["billing", "technical", "sales"]);

    Assert.Equal("billing", d.Value);
    Assert.Equal(0.94, d.Confidence, 3);
    Assert.False(d.Abstained);
}

You can also assert the outgoing request is shaped right (model, state, the choice question, the Authorization header) by reading it back off the stub — so a refactor can't silently break the wire format. And test the abstain threshold, and that an API error becomes a typed exception. None of these need a key.

Live tests — real calls, gated off by default. Keep one or two that actually hit Jev, but return early unless TYPESAFE_LIVE=1 and a key are present, so CI stays green and fast:

[Fact]
public async Task Jev_classifies_a_clear_billing_ticket()
{
    if (Environment.GetEnvironmentVariable("TYPESAFE_LIVE") != "1") return;   // gated
    var jev = new JevDecider(new HttpClient());
    var d = await jev.ChooseAsync("I was double-charged $40, please refund.",
                                  "Which team should handle this?",
                                  ["billing", "technical", "sales"]);
    Assert.Equal("billing", d.Value);
    Assert.True(d.Confidence > 0.6);      // assert on the calibrated confidence, not just the label
}

Two rules for the live ones: assert on the confidence (that's Jev's actual selling point), and if you have a labeled set, assert batch accuracy rather than a single case — one-case model assertions are flaky by nature.

Engine 2: turning a local model into a System One decider

Here's where it gets fun. You don't need Jev to get bounded, never-malformed answers. You need constrained decoding: at every step, mask the model's next-token distribution so the only tokens it may emit spell one of your options. Illegal tokens get set to −∞ before sampling. The output space is your answer space.

The cheapest way to express that is a GBNF grammar — the grammar format llama.cpp speaks — built straight from your options:

root ::= "billing" | "technical" | "sales"

My LocalDecider builds exactly that grammar from the options list and sends it to a llama.cpp server along with the prompt, asking for token probabilities:

# any instruct GGUF you already have on the box
llama-server -m qwen2.5-3b-instruct-q4_k_m.gguf --port 8080

var local = new LocalDecider(new HttpClient { BaseAddress = new("http://localhost:8080") });
var d = await local.ChooseAsync("My card was charged twice this morning.",
                                "Which team should handle this?",
                                ["billing", "technical", "sales"]);
// d.Value is ALWAYS one of the three — the grammar makes anything else unrepresentable.

The model cannot return a paragraph, a "maybe", or a malformed value, because none of those are legal token sequences. Confidence comes from the first decision token's probability (n_probs). That's the guaranteed-valid path, which is what routing and guardrail decisions actually need. If you want a fully calibrated distribution over every option, score each option's sequence and softmax the log-likelihoods — a good next step, noted in the repo.

Same idea works through LM Studio, Ollama, or vLLM (json_schema / grammar / guided_choice); in-process, LLamaSharp exposes both grammars and raw logits so you can do it without a server at all. Whatever you pick, it hides behind IDecider and your calling code never knows.

The payoff: swap the engine, keep everything else

Because all three — JevDecider, LocalDecider, and a FakeDecider for tests — implement the same interface, the code that consumes a decision never changes:

IDecider decider = engine switch
{
    "jev"   => new JevDecider(new HttpClient()),
    "local" => new LocalDecider(new HttpClient { BaseAddress = new("http://localhost:8080") }),
    _       => FakeDecider.Always("technical", 0.82),   // unit tests, demos
};

var d = await decider.ChooseAsync(ticket, "Which team should handle this?", teams, abstainBelow: 0.6);
var route = d.Abstained ? "human-review" : d.Value;

That last line is the whole app. Your router's tests run against the FakeDecider — instant, hermetic, no model — and the same router runs in production against Jev or a local model. And you get a free experiment out of it: point it at Jev, then at a 3B model on your own box, and measure how close local-in-a-harness gets on your workflow. That comparison — is a purpose-built System One model actually better than an ordinary model with a grammar bolted on? — is the honest question about Jev, turned into a test you can run yourself.

The repo has all of it — the three engines, a sample CLI, and 14 tests that pass offline with no key and no GPU: github.com/egarim/systemone-deciders. Clone it, run dotnet test, then start a local server and watch a 3B model refuse to give you anything but one of your options.