Where it pays off: five patterns from real projects
System One Models for leaders, part 2 of 4
In the first post I argued that most AI inside a product is deciding, not writing, and that a new class of model is built for exactly those decisions. That is a useful idea, but ideas are cheap. The better test is what people have actually built with it.
Over the last few months a handful of open projects have put System One models to work, mostly TypeSafe's Jev, on real problems: searching long documents, driving a browser, grading AI output, supervising coding agents. I read each one closely, including the parts the authors were honest enough to flag as unproven. Five patterns keep repeating. None of them needs you to understand the engineering, and each one maps to a process your business probably already runs.
The short answer
System One models pay off when you can turn a messy task into a series of small choices over options you control. The projects below do that in five ways: quote the evidence instead of rewriting it, narrow a huge choice step by step, let an agent act only from a menu, grade AI output with a consistent judge, and put a cheap supervisor beside an expensive worker. In each case ordinary software stays in charge and the model only makes the judgment.
Pattern 1: quote the evidence, don't rewrite it
The business problem. Someone needs one fact from a long document: a clause in a contract, a limit in a policy, a figure in an annual report. The usual AI answer is to have a language model read the document and write a reply. You get a fluent paragraph, and then someone has to check it line by line against the source, because a paraphrase can quietly drift from what the document says.
What the projects do instead. jevfetch splits a long page into passages and asks, for each one, "does this help answer the question?" The passages that pass come back word for word. A demo by an engineer at Reducto goes a step further: you ask a question about a PDF and the answering passages light up on the page itself, in real time.
Why it matters. The output is the evidence, not a summary of it. In the author's own tests, jevfetch quoted the source exactly in 81 out of 81 cases against 25 out of 81 for the standard tool, which follows from the design: it copies text and never rewrites it. A wrong answer is still possible, but it shows up as the wrong passage highlighted, which a reviewer spots in seconds. Anyone who has sat through a compliance, legal or audit review knows that "show me where it says that" matters more there than a well-written answer does.
Where you would use it: contract review, policy lookups, regulatory filings, support knowledge bases, audit evidence.
Pattern 2: narrow a huge choice step by step
The business problem. Some decisions have too many options to consider at once. Which of 318 pages answers the question? Which of thousands of product categories does this item belong to? Which of fifty teams owns this ticket?
What the projects do instead. jev-doc-search finds the right page in a long financial report without any of the usual search infrastructure. It first asks "which section?", then "which page inside that section?", then checks each finalist with a yes or no. neo4jev does the same across a company's knowledge graph, choosing one real connection at a time. Because the model only ever picks from options that actually exist, every answer points at something real.
Why it matters. This is how a good filing system or a support escalation tree already works: you never choose from everything at once, you choose a drawer, then a folder. The model makes each of those small choices quickly and with a confidence attached, so the software can keep two or three promising paths open instead of betting on the first guess. The evidence here is thin, two worked examples in one project and a demo in the other, so treat it as a proof that the idea works rather than a measure of how well.
Where you would use it: product catalogues, ticket taxonomies, document archives, org charts, any nested category system.
Pattern 3: let an agent act, but only from a menu
The business problem. AI agents that click through software on your behalf are slow, expensive and hard to trust. If an agent reports "the test passed" or "the order is placed", how do you know?
What the projects do instead. QuickE2E tests websites from plain-English instructions. At every step, ordinary software lists the buttons and links that actually exist on the page, and the model picks one. The model never types anything, never sees passwords, and cannot invent an action, because only options the software offered can run. Software, not the model, decides whether the test passed.
Why it matters. In the author's own measurements on one task, run five times each, QuickE2E took about 3 to 4 seconds and cost about $0.0004 per run on the hosted model, against about 25 seconds and $0.09 for a general-purpose coding agent doing the same job (project README). Those are small, directional numbers. The design point matters more to a leader than the speed: the list of things the AI could possibly do was written by your software before it ran. That is the same principle we already apply to people and systems under least-privilege access, now applied to an AI.
Where you would use it: software testing, operations runbooks, back-office workflow bots, any agent that takes actions in real systems.
Pattern 4: grade AI output with a judge that gives the same answer twice
The business problem. Every team that puts AI into production ends up grading its output: was this reply accurate, grounded, useful? The common approach is to have a second language model do the grading. It works, but it is expensive at volume and, worse, inconsistent. Grade the same answer twice and you can get two different scores, so you cannot tell whether your AI improved or the grader changed its mind.
What the projects found. LangChain and two authors tested exactly this. Four AI judges graded the same recorded agent runs 100 times each, and a human's verdicts served as the answer key. The System One model matched the human on every pass or fail call, cost about $0.00035 per grading against $0.028 for Claude Sonnet, answered in under half a second, and gave almost exactly the same score on every repeat.
Why it matters. The consistency is the headline, not the cost. A grader that varies from run to run makes every quality dashboard you build on it unreliable. The authors are careful to say the sample was tiny (five runs, one human reviewer) and that consistent is not the same as correct. They propose a measure worth borrowing: a judge's value is how often it agrees with people multiplied by how repeatable it is. A judge needs both.
Where you would use it: quality assurance on AI replies, content moderation review, support ticket QA, document review checklists.
Pattern 5: a cheap supervisor beside an expensive worker
The business problem. Long-running AI agents, the kind that write code or work through a case for an hour, fail in recognisable ways. They loop, drift off task, skip checks, or declare victory too early. Putting a second expensive model on watch doubles the cost.
What the projects do instead. Foreman watches a coding agent and, every cycle, asks about ten narrow yes-or-no questions: Is it stuck? Is it off track? Is it ignoring the house rules? Does a human need to look? Plain software then turns those answers into one action, such as continue, steer, stop or escalate, using thresholds written in a settings file.
Why it matters. Two ideas here are worth taking into any AI governance conversation. First, the questions a supervisor asks are narrow, so a fast, cheap model can ask them constantly instead of occasionally. Second, how cautious the supervisor is becomes a number someone can read and change, not wording buried in a prompt. Your risk team can review a threshold. They cannot meaningfully review a paragraph of instructions to a language model. The author is candid that the accuracy of these checks is not yet proven, which is exactly why the thresholds need to be visible.
Where you would use it: supervising any long-running AI process, live customer conversations, content pipelines, automated case handling.
What the five patterns have in common
Read them again and the same structure shows through. Software lists the real options, the model makes a small judgment with a confidence attached, and software decides what to do with that confidence. The expensive, creative work, where there is any, stays with a language model or a person.
That gives you a practical way to scan your own operations. You are looking for places where a person or an AI is choosing among known options, at volume, and where you would benefit from the answer being fast, cheap, repeatable or traceable to its source. You are not looking for places where something new has to be written.
Questions to ask your team
- Where do we ask AI to summarise a document when what we really need is the passage that answers the question?
- Which of our AI agents can take actions, and is the list of actions it may take written down in our software or left to the model?
- How do we grade our AI's output today, and would the grader give the same score twice?
- For our longest-running AI processes, what watches them, and who can change how cautious that watcher is?
- Which of our decisions involve choosing among hundreds of options, and how do we narrow them today?
Claims to treat with care
Almost every number in this post comes from the people who built the project, measured on their own tasks, often with five or fewer runs. I have reported them because they show the shape of the opportunity, not because they predict what you will see. One result is worth remembering for the opposite reason: on one larger set of questions, the jevfetch author found their tool was more accurate and faster but about 20% more expensive overall. Cheaper per call does not always mean cheaper end to end, and an honest builder told us so.
The next post is about the other side: where these models do not fit, and which vendor claims to discount. Before then, pick one process your business runs every day and ask yourself whether it is really a writing task, or a sequence of choices that has been dressed up as one.
← Part 1 · Next: Part 3, Where it doesn't fit → · Glossary · All courses
Member discussion