Agents · Evaluation
Jev in an Agent Loop: the Decision Layer
Agents run in a loop. A model decides what to do, a tool executes, something evaluates the result, and the loop turns again. The first two steps get the attention. The third is where the engineering usually hurts.
Jev, released by TypeSafe AI on September 15, 2026, is built for that third step and nothing else. This is a look at where it fits, what it replaces, and what the first published test outside the vendor actually measured.
Two ways teams judge a step today
Code-based evaluation came first and is still the cheapest thing available. A function checks whether the agent called the tool, whether the JSON parsed, whether the number is in range. It is fast, deterministic and narrow: it needs fixed inputs, and agent behaviour is not fixed. Deciding whether the agent used a tool result to answer the user's question well is not something you can enumerate in a conditional.
So teams reach for an LLM judge, which accepts the unstructured trace and scores it with a prompt. That buys generality and pays for it three times: in latency, because the judgement is written one token at a time; in money, at every call; and in variance, because the same trace can score differently on two runs.
The third option is a model that treats evaluation as what it is, a decision task. Given a state and a typed question, return the typed answer with its probabilities.
Where a System One model goes
The slot is step three. It does not plan, it does not call tools, and it does not write the next instruction. It looks at the state the loop just produced and answers questions whose answers you declared in advance.
That position is also why its latency and price matter more than they look. Step three runs on every turn, and in an online setting it can run on a large share of production traces rather than a sample, which is exactly the decision cost usually forces teams to make.
Ask several small questions, not one big one
The idiomatic shape is not one prompt asking for a verdict. It is a handful of atomic questions, each with its answers declared, evaluated against the same state and combined in your own code.
TypeSafe's documentation says every question is evaluated in parallel and in isolation against the same state in one go, and that adding questions barely changes the response time. So the natural design is a fan of narrow questions: was the tool called, is the answer grounded in the retrieved evidence, how useful is it on a one-to-five rubric, which of these three search outcomes describes the run.
Each one comes back with probabilities, and the combining logic lives where you can read it, test it and change it without touching a prompt.
What one published test found
On September 20, 2026 LangChain published an experiment using Jev as a judge and comparing it with three language models on the same replayed agent run. On the binary pass-or-fail decision, Jev agreed with a human reviewer on all 500 repeated decisions; GPT-5.6 Terra agreed on 99.8 per cent, GPT-5.6 Luna on 96.4, and Claude Sonnet 4.6 on 80.0.
Repeatability was the sharper result. On the continuous quality score, Jev's mean per-case variance was 0.0000149, which LangChain reports as 92 to 913 times lower than the three language models. Accuracy tells you whether a judge agrees with a human; variance tells you whether it will say the same thing tomorrow. A testing apparatus needs both.
LangChain is careful about what this does not establish. They call it a narrow experiment built on five weather requests, they describe the variance finding as observational rather than causal, and they published the repository and pinned library versions so somebody else can run it. They also hosted a livestream with TypeSafe two days later, which is worth knowing when you weigh the write-up.
Posted 2026-09-18. A developer write-up of the same loop pattern: an LLM decides, a tool executes, a model evaluates.
The failure mode worth designing against
Cheap judgement changes behaviour. When a verdict costs a fraction of a cent, you stop sampling and start evaluating everything, which is the point. It also means a judge that is wrong in a consistent direction is now wrong everywhere, quietly and at scale.
LangChain says this outright: low cost can amplify mistakes, and a consistently wrong evaluator produces bad feedback faster. Low variance makes that worse before it makes it better, because a repeatable judge is a repeatable error until someone checks it.
The mitigation is the boring one. Keep a human-labelled oracle set, re-measure agreement when you change a question, and read a sample of the traces the judge rejected rather than only the ones it passed.
| Approach | Strength | Weakness |
|---|---|---|
| Code-based | Deterministic, near-free, auditable | Needs fixed inputs; cannot judge open-ended behaviour |
| LLM as judge | Accepts unstructured traces; general | Slower, dearer, and non-deterministic across runs |
| System One model | Typed answers with probabilities; repeatable; cheap enough to run on everything | No open-ended reasoning; a wrong rubric is now wrong consistently |
What to write down before you wire one in
Three things, and none of them is a prompt. The state: what exactly the model sees on each call, and in what shape. The questions: each one atomic, with its options or its rubric spelled out, because those are the type. The thresholds: what your code does at a confidence of 0.9, and what it does at 0.55.
That last one is where this style of model moves work rather than removing it. A language model hides the threshold inside its prose; a typed decision hands you the number and makes you decide. Teams that already keep a labelled evaluation set will find this easy. Teams that do not will discover they needed one anyway.
Agent evaluation questions
Does a decision model replace an LLM judge entirely?
Only for questions whose answers you can declare in advance. Anything that needs an explanation, a rewrite or an open-ended critique still needs a generating model.
Does asking more questions cost more time?
TypeSafe's documentation says questions are evaluated in parallel and in isolation against the same state, and that adding questions barely changes the response time.
Is 100 per cent agreement a benchmark result?
No. It is one narrow experiment on five replayed weather requests, published by a company that ran a livestream with the vendor two days later. Treat it as a first data point.
Keep reading
Ciyo writes about the models behind creative and agent tooling, and tests the ones it can run.