Where it doesn't fit, and the claims to discount

System One Models for leaders, part 3 of 4

Every new class of technology arrives with a vendor slide that says it does everything, and System One models are no exception. The previous post showed five patterns where they earn their place. This one is about the boundary: the work they cannot do, the claims you will hear that need translating, and the questions that separate a real result from a good demo.

I find this the more useful half for a leader. Knowing where something pays off tells you where to look. Knowing where it fails tells you what to say no to, and that is usually where the money is saved.

The short answer

A System One model cannot create anything. It cannot write a reply, summarise a meeting, draft a plan or reason through a problem in steps. It can only choose among options you give it. Its headline claims, "cannot hallucinate", "100 milliseconds", "beats GPT on our benchmark", are each true in a narrow sense and misleading in a wide one. Ask for results on your own data, measured by someone other than the vendor, before you believe any of them.

Where it does not fit

Anything that has to be written. Replies to customers, summaries, reports, translations, marketing copy. jevfetch, the evidence-quoting tool from the last post, is honest about this in its own documentation: it is bad at summaries and translation because it can only select, never write.

Anything that has to be invented. PR Lens is a tool that turns a software change into an architecture diagram. It uses an ordinary language model, and correctly so, because drawing a diagram means inventing components, naming them and deciding how they connect. None of those options exist until the model creates them, so there is nothing for a System One model to choose from.

Multi-step reasoning. Working out why a shipment is late, negotiating terms, diagnosing an unfamiliar failure. These need a chain of thought where each step depends on the last. A System One model gives you one quick judgment, not a line of argument.

Decisions where the options are not known in advance. If the answer could be anything, there is no list to choose from. You may be able to create one, which is how the search projects in the last post work, but if you cannot, this is not the tool.

The more useful view: a judgment shell around a generative core

The interesting cases are not either-or. PR Lens is a good illustration. Its core job, drawing the diagram, is generative and should stay on a language model. But around that core sit several one-second judgments that are System One work. Is this change big enough to need a diagram at all? How risky is it, from cosmetic to touching payments? Is each arrow in the generated diagram actually supported by the change, or did the model invent it?

That gives you a design pattern worth asking your teams about. Keep the creative step on a language model or a person, and wrap it with fast judgments: one before it, deciding whether it is needed and how carefully to do it, and one after it, checking the output against the source. The expensive step runs less often, and its mistakes are caught more often.

The same lens applies to systems you already own. activegraph, a framework for long-running AI agents, does not use a System One model today. Look inside it, though, and many of its steps are judgments rather than writing: does this new evidence support or contradict what we believe? Is this new task a duplicate? Did the result satisfy the task? Workflow engines, CRMs and ticketing systems are full of the same moments. You do not need a new platform to find them, only someone willing to ask of each automated step whether it is writing something or judging something.

Claims to discount, and how to translate them

"It cannot hallucinate." This is the claim you will hear most. It means the format cannot go wrong: you will always get one of the options you defined, never a made-up category or a broken answer. It does not mean the choice is right. The model can pick the wrong option and be confident about it. The honest version of the claim is "it cannot give you an answer outside your list", which is valuable but much narrower.

"About 100 milliseconds." Speed depends on where the model runs. Convai's open model Laya reports about 33 milliseconds on a modest graphics processor, and between 190 and 460 milliseconds on an ordinary server processor (model card). Both are fast compared with a language model's seconds, but a tenfold range is the difference between fitting inside a live interaction and not. Ask what hardware the number came from.

"Beats leading LLMs on our benchmark." Check who ran the benchmark. Jev's headline accuracy figures come from TypeSafe's own benchmark, and I have not found an independent study comparing these models with language models on the same tasks. That does not make the numbers wrong. It makes them a vendor's claim, which you would treat the same way for any other software purchase.

"Works out of the box." Some do, some do not. Laya's own documentation says its base version performs close to chance on new tasks until it is tuned on your data. An open model you can run inside your own walls may need a tuning step a hosted one does not. Budget for it.

"Consistent results." Consistency is real and valuable, as the grading experiment in the last post showed. But consistent is not correct. A model that is consistently wrong will be wrong at scale, quickly and cheaply. You want both: agreement with your best people, and the same answer twice.

"Cheaper per call." Usually true, and sometimes beside the point. The jevfetch author found that on one larger test their tool was faster and more accurate but about 20% more expensive overall, because of how many calls it made. Ask for cost per completed task, not cost per call.

Small samples. Several of the most-quoted results in this space come from five test runs and one human reviewer. They show that an idea works. They do not show how well it works on your problem.

Two risks that are easy to miss

Your data leaves the building. A hosted model means the text you are judging is sent to the vendor. For many tasks that is fine. For customer records, health data, or anything under data residency rules, it may not be, and the answer may be an open model you run yourself, with the tuning cost that comes with it. Settle this before the pilot, not after it looks promising.

The model changes underneath you. Hosted models are updated. If your team does not pin a specific version, the same question can start returning different answers after an update, and cached results from the old version can linger. A judgment you rely on for compliance needs a version number, the same as any other system of record.

Questions to ask a vendor

1. Who ran the benchmark behind your headline numbers, and can we see results on tasks like ours?
2. What hardware does your speed figure assume, and what will we see in our environment?
3. Does our data leave our environment, and where is it processed and stored?
4. Can we pin a model version, and how will you tell us before it changes?
5. How do we see when the model is unsure, and what share of cases do customers typically send to a person?

Questions to ask your team

1. For each AI step we run, is it writing something or judging something?
2. Where a language model writes something important, what checks its output against the source?
3. When we compare AI options, are we comparing cost per call or cost per finished task?

The last post in this course turns all of this into a 30-day pilot your team can run on one real decision. Before then, think about the AI claim you heard most recently from a vendor or in a board pack. If you translated it the way this post does, what would it actually be promising?

← Part 2 · Next: Part 4, A 30-day pilot → · Glossary · All courses