Models · System One
The Options You Declare Are the Whole Model
With a language model, a bad prompt produces a bad sentence and you can see it. With a typed decision model, a bad question produces a perfectly well-formed answer of exactly the right type, and you cannot see anything wrong at all.
That is the trap worth understanding before you wire one into anything. Two pieces of evidence from the week of September 17, 2026 — one a pre-registered evaluation, one a practitioner's run — point at the same conclusion from different directions: the accuracy you get is mostly a property of how you wrote the questions. Ciyo does not run Jev; it makes no image and no video and is in neither of our model registries.
One question, or five
The most direct evidence published this week came from a practitioner running phishing triage. Asked as a single question, the task scored 62.6 per cent. Split into five separate questions, evaluated against the same state and combined in code, it scored 95 per cent.
That is one person's run on their own labels with no published dataset, so it is a reason to try the split rather than a number to quote. But it matches what the vendor's own documentation recommends — decompose into atomic, single-purpose questions rather than asking a multi-factor judgement in one — and it matches the shape of the API, where every question in a call is evaluated in parallel against the same state.
The economics make the advice easy to follow. The cost that dominates a call is sending the state, not answering the questions. Asking five is close to free once the state is on the wire.
Posted 2026-09-21. The poster reports both numbers from their own phishing-triage run, adds that the probabilities were confidently wrong where the answer was genuinely unknowable, and ends with the right advice: shadow-test it on your own labels first. No dataset was published, so the figures are the poster's own.
How to split a question that resists splitting
“Is this email a phishing attempt” is a conclusion, not a question. The split is to ask about the evidence that would make it true, one piece at a time, and let your own code do the concluding.
Does the sender domain match the organisation it claims to be? Does the message ask for a credential or a payment? Is there time pressure in the wording? Do the visible link texts match their destinations? Is the greeting generic where this sender normally uses a name?
Five narrow questions, five answers with their own confidence values, and a rule in your code that you can read, test and change without touching a prompt. That last part is the real prize: the combining logic becomes ordinary software.
questions: {
"sender_mismatch": { "type": "noul", "instructions": "The sender domain does not match the organisation the message claims to be from." },
"asks_for_secret": { "type": "noul", "instructions": "The message asks the reader for a password, a code, or a payment." },
"time_pressure": { "type": "noul", "instructions": "The message pressures the reader to act quickly." },
"link_mismatch": { "type": "noul", "instructions": "A visible link text differs from where the link actually points." },
"risk": { "type": "score", "instructions": "How risky is it to deliver this message to an inbox?",
"criteria": ["harmless", "suspicious", "likely phishing", "clearly malicious"] }
}An option name is a label; the criteria are the definition
A Choice takes a `criteria` map from option names to descriptions, and a Score takes an ordered array of level descriptions. Those strings are not documentation for your colleagues. They are the only definition the model has.
“billing” tells it almost nothing. “The sender is asking about an invoice, a charge they do not recognise, a refund, or changing their payment method” tells it what to look for. The second version is also the one you can argue about with a colleague before anything is automated, which is the point where disagreements are cheap.
Write the criteria the way you would write instructions for a new person on their first shift: what belongs here, what looks similar but belongs elsewhere, and what to do when it is neither.
The answer it cannot give you
Now the sharper finding, and the one that changes a design. In the pre-registered evaluation published on September 20, 2026, the authors tested whether the model would signal that a state fell outside the declared options. In 30 out-of-scope cases where no escape option existed, it flagged none of them.
That is not a defect. It is the type system working exactly as advertised: you declared the possible answers, and a value of the declared type came back. But it means an out-of-scope input does not fail loudly. It succeeds quietly, with the closest option and a confidence number attached to it.
The fix is one line of question design. Declare the escape option yourself — `none_of_these`, `needs_a_human`, `not_applicable` — and write criteria for it as carefully as for the real options. Then route it in code.
What the same evaluation found it is good at
It is worth reporting the other half, because the folklore about these models is often wrong in the reassuring direction and wrong in the pessimistic direction too.
The same run found number comparison and date ordering holding at 99.6 per cent across 13 different question designs, and negation handled correctly in 100 per cent of the cases tested. A practitioner guide published three days earlier had warned that negations can misfire and that the model is poor at arithmetic and dates. The larger, pre-registered run did not reproduce that.
Which is a useful reminder about this whole subject in September 2026: a great deal of confident advice is circulating about a model that has been public for less than a week. Prefer the evidence with a method and a repository attached.
| Claim | Source | What the pre-registered run found |
|---|---|---|
| Poor at arithmetic and date calculations | A practitioner guide, 2026-09-17 | Number comparison and date ordering held at 99.6 per cent across 13 designs |
| Negations can misfire | The same guide | Negation handled correctly in 100 per cent of the cases tested |
| Context rot: accuracy falls with irrelevant state | The same guide | Not contradicted here; worth testing on your own state |
| Reads instructions literally | The same guide | Consistent with the out-of-scope finding: it answers the question you wrote |
A checklist before a question goes to production
Is it atomic? If the sentence contains “and” or “unless”, it is probably two questions.
Are the options exhaustive, including an escape? If a plausible input has nowhere to land, it will land somewhere wrong.
Does each option have criteria? An option name is a label; the criteria are the definition, and the model only has what you wrote.
Is the rubric ordered? A Score expects two to ten levels that mean something in sequence, and the returned score can land between them.
Have you measured it against your own labels? Every number in this article came from somebody else's task.
Question-design questions
Why is splitting a question better?
Each question is evaluated in isolation against the same state, so a narrow question is a narrow task. The combining logic then lives in your code, where it is testable. One practitioner run went from 62.6 to 95 per cent on their own labels after splitting.
Does asking more questions cost more?
Very little. The dominant cost is the state you send; the questions are evaluated in parallel against it.
What happens to an input none of my options fit?
It gets the closest option anyway. A pre-registered evaluation found 0 of 30 out-of-scope cases flagged when no escape option was declared. Declare one.
Is it bad at dates and negation?
A practitioner guide said so on September 17, 2026. A larger pre-registered run three days later measured 99.6 per cent on number and date ordering and 100 per cent on negation. Test it on your own data rather than trusting either.
Keep reading
Ciyo writes about the models behind creative and agent tooling, and tests the ones it can run.