Models · System One
Can You Route on Jev's Confidence Score?
A decision model that returns an answer is useful. A decision model that returns an answer plus a trustworthy number describing how sure it is can be wired into a system that knows when to stop and ask a person.
That is the entire promise of a calibrated confidence value, and it is also the claim most worth checking, because a confidence number that does not track reality is worse than no number at all: it tells your code to proceed. Two independent evaluations published their data in the week of September 17, 2026, and between them they answer the question better than any vendor page could. Ciyo does not run Jev; this is a read of published evidence.
Confidence is not the probability
A Choice answer comes back with two different numbers, and conflating them is the first mistake. The probabilities describe the distribution across your declared options: how the model's belief is spread. The confidence value is a separate scalar describing how sure the model is about the answer at all.
TypeSafe's own documentation frames it as a second axis: the answer tells you what, and confidence tells you whether to act. That framing is the reason it is worth a whole article. You can build a system where the answer chooses the branch and the confidence chooses whether a human sees it first.
The documented pattern ties the threshold to the blast radius
TypeSafe publishes a confidence-routing pattern, and the useful thing about it is that it does not give one number. It gives a ladder tied to how bad it is to be wrong.
Below 0.6, hand the case to a person. Above 0.6, act — but only if the action is cheap. For an expensive or irreversible action, ask the user to confirm between 0.6 and 0.85 and act automatically only above 0.85. In their worked example, showing an account balance clears at 0.6 while approving a transfer does not clear until 0.85.
This is a better default than any single global threshold, because it makes you write down what the action costs. Most teams discover, doing this, that they have been running several very different actions off one branch.
What two independent runs measured
The first is small and pointed. On September 17, 2026 a reproducible benchmark classified 60 hand-labelled agent tool calls into four risk categories, split between clear, ambiguous and adversarial cases. Overall accuracy was 91.7 per cent — 100 per cent on the clear cases, 71.4 per cent on the ambiguous ones. The finding that matters here: every incorrect answer arrived hedged. Misses carried confidence between 0.130 and 0.785, the model never returned maximum confidence while wrong, and the 0.9-to-1.0 band was 98 per cent accurate.
The second is larger and pre-registered. On September 20, 2026 an evaluation published 5,721 calls across 21 experiments, with 50 predictions timestamped before any data was collected. Of those predictions, 26 were confirmed and 21 were falsified, 18 of them because the authors had expected failure where none appeared. On calibration it found accuracy essentially flat between 0.50 and 0.95, then reaching 100 per cent at a 0.99 threshold that still covered 60.2 per cent of traffic.
Read together, they say something more precise than “the confidence is good”. High confidence was reliable in both. The middle of the range carried very little information in the larger one.
What that means for your threshold
If accuracy is flat from 0.5 to 0.95, then a threshold of 0.7 and a threshold of 0.9 are buying you roughly the same thing on that task, and the extra rejections at 0.9 are mostly wasted human time.
The interesting band is the top. A 0.99 gate that still auto-handles 60 per cent of traffic at full accuracy is a genuinely useful operating point: three fifths of the work runs untouched and the remainder reaches a person. That is a different system design from “route the bottom 5 per cent to a human”.
But this is one task on one day, and it is not yours. The number to act on is the one you measure on your own labelled set.
| What was found | What it supports | What it does not support |
|---|---|---|
| Every wrong answer was hedged, n = 60 | Treating a maximum-confidence answer as safe on that task | A guarantee that it never fails confidently on yours |
| Accuracy flat from 0.50 to 0.95 | Suspecting that a mid-range threshold buys little | A claim that confidence is useless below 0.99 |
| 100 per cent at 0.99, covering 60.2 per cent | Designing a high-gate, high-coverage escalation policy | The same coverage on a different task or distribution |
Measuring your own curve, in an afternoon
You need a labelled set, and if you do not have one this is the moment you find out you needed it anyway. Two hundred rows you have already judged is enough to be interesting.
Send each row, keep the answer and the confidence, then bucket by confidence and compute accuracy per bucket. That table is your calibration curve. Pick the lowest threshold whose bucket is accurate enough for the cost of that action, and route everything below it to a person.
Then re-run it when you change the question wording, because you have changed the model's task, and the curve you measured belonged to the old one.
Posted 2026-09-21, quoting a post by the prediction-system company Yarrow. The poster's definition of calibration — when a model says 70 per cent, the event should happen 70 per cent of the time — is the standard one and is why a calibration curve is worth measuring. Note the commercial context: the quoted account sells a forecasting product and is drawing a parallel to its own.
The failure this design invites
A cheap, repeatable judge changes behaviour. When a decision costs a fraction of a cent, teams stop sampling and start evaluating everything, which is the point of the thing.
It also means a judge that is wrong in a consistent direction is now wrong everywhere, quietly, at full confidence, on every row. Low variance makes that worse before it makes it better, because a repeatable mistake is repeated exactly.
The mitigation is unglamorous: keep a human-labelled oracle set, re-measure agreement whenever a question changes, and read a sample of what the judge approved rather than only what it rejected.
Confidence questions
Is confidence the same as the top probability?
No. The probabilities describe the spread across your declared options. Confidence is a separate value about how sure the model is at all, and it is the one the documented routing pattern uses.
What threshold should I use?
The published pattern ties it to the cost of the action: below 0.6 to a person, 0.6 for cheap actions, 0.6 to 0.85 with user confirmation for expensive ones, and above 0.85 automatically. Then measure your own curve.
Did anyone find it confidently wrong?
Not in the 60-case benchmark, where every miss was hedged and the model never answered wrongly at maximum confidence. One practitioner reported confidently wrong probabilities on unknowable cases, without publishing data.
Does Ciyo use confidence routing?
Not with Jev, which Ciyo does not run. The pattern itself — a typed judgement plus a threshold that decides whether a person looks — applies to any evaluator, including a language model acting as a reviewer.
Keep reading
Ciyo writes about the models behind creative and agent tooling, and tests the ones it can run.