How to start: a 30-day pilot your team can run
System One Models for leaders, part 4 of 4
The first three posts in this course covered what a System One model is, where it has paid off, and where it does not fit. This last one is practical. If you think there is something here for your business, how do you find out without starting a programme?
My answer is a 30-day pilot on one decision, run in the background beside your current process, ending in a clear yes or no. It is deliberately small. Most AI pilots fail not because the technology does not work but because nobody agreed in advance what "worked" would mean. This plan fixes that first.
The short answer
Pick one high-volume decision your business already makes, where you have past answers you trust. Agree on how you will measure success before anyone builds anything. Test the model against your past answers, then run it silently beside the live process for a week. At the end, decide on three numbers: how often it agrees with your best people, how often it gives the same answer twice, and what share of cases it can handle confidently enough that nobody needs to look.
Week 1: choose the decision, and settle the data question
The choice of decision matters more than anything that follows. A good first candidate has five properties.
It happens often. Hundreds or thousands of times a week. The savings in a System One model are per decision, so a rare decision will never show them.
The answer is one of a known set. Route to one of eight teams, approve or refer, low, medium or high priority. If the answer has to be written, go back to the last post.
You have past answers you trust. Decisions your people already made, ideally a few hundred, that you are confident were right. These are what you will test against, and without them you have no way of knowing whether the model is good.
A wrong answer is cheap to catch. Choose a decision where a mistake is noticed and corrected, such as a misrouted ticket, rather than one where it causes harm before anyone sees it. Leave credit decisions and anything touching safety for a later round.
It is costing you something today. Either people spend time on it, or a language model is already making it at language-model prices and speeds. That is the cost the pilot is trying to remove.
In the same week, settle where the data can go. If the text you would send to the model includes customer records or anything covered by data residency rules, a hosted service may be ruled out, and you will be looking at an open model you run yourself. That changes who you need on the team and adds a tuning step, so it is far better to know on day three than on day twenty-five.
What you have at the end of week 1: one named decision, an owner from the business side, a few hundred past examples with trusted answers, and a yes or no on whether the data can leave your environment.
Week 2: write the questions and test against the past
Someone who knows the process, not only an engineer, writes the questions the model will answer. This is the step people underestimate. "Is this a complaint?" sounds clear until you meet the message that is half complaint and half question. Writing the definitions forces your team to say what the rule actually is, which is often useful in its own right.
Then run the model over your past examples and compare. You are looking for two things: how often it matches your trusted answers, and whether it gives the same answer when asked twice. LangChain's grading experiment, from part 2, suggests a simple combined measure: agreement with people multiplied by repeatability. A model needs both.
Where it disagrees with your people, look at the cases. Some will be model errors. Some will turn out to be cases where your own team disagreed with each other, or where the definition was unclear. One open project, jev-align, makes this a routine: it finds the cases the model is least sure about, asks a person to label just those, and proposes a better wording of the question, which a person then accepts or rejects. You do not need that tool, but you should borrow the habit. Spend your people's time on the hard cases, and treat the question's wording like a policy document that gets reviewed and versioned.
What you have at the end of week 2: an agreement rate and a repeatability rate on your own past data, and a short list of the cases where the model, or your definition, needs work.
Week 3: run it in the shadows
Past data can flatter a model. The real test is live traffic. For one week, run the model beside the existing process without letting it act. Every time a person or the current system makes the decision, the model makes it too, silently, and both answers are logged.
This is called running in shadow mode, and it is the single most important safety step in the plan. Nothing changes for customers or staff. At the end of the week you have a side-by-side record of what the model would have done against what actually happened, on real cases, at real volume.
Use the week to look at confidence as well as accuracy. A System One model tells you how sure it is. Plot its agreement with your people against its confidence and you will usually see that when it is very sure, it is very often right, and when it is unsure, it is not. That relationship is what makes the next step possible.
What you have at the end of week 3: a week of real decisions with the model's answer, your answer and the model's confidence side by side.
Week 4: set the line, and decide
Now choose a confidence line. Above it, the model's answer would be used directly. Below it, the case goes to a person or a language model, as it does today. Moving the line trades automation against risk: a higher line means fewer automated decisions but fewer mistakes among them.
This is a business decision, not a technical one, and it belongs with the process owner and whoever owns the risk. It is also one of the real advantages of this approach. The line is a single number someone can read, argue about and change, rather than a paragraph of instructions inside an AI prompt.
Then decide, against the measures you agreed in week 1.
Agreement: how often the model matched your trusted answers above the line.
Repeatability: whether it gives the same answer twice.
Coverage: what share of cases fell above the line, and so would need no person.
Cost and speed per decision, compared with how you do it today, measured per finished case and not per call.
Escalation load: how many cases still go to people, and whether that is fewer than today.
If the numbers clear the bar you set, the next step is to let the model act on cases above the line, keep the shadow log running, and review it monthly. If they do not, you have spent a month and learned something specific about a decision you care about. Both are good outcomes for a pilot.
Who you need
A process owner from the business, who chooses the decision and owns the confidence line. An engineer for part of their time, to connect the model and keep the log. One or two experienced people who can label the hard cases in week 2. If your data cannot leave your environment, add someone who can run and tune an open model. That is the whole team.
Where it fits, and where it does not
This plan suits a decision that is frequent, has a known set of answers and has a history you trust. It does not suit a decision you make a few times a month, one where the right answer is genuinely contested inside your organisation, or one where a single mistake is costly enough that no confidence line would satisfy your risk team. For those, a System One model might still check a person's work after the fact, but it should not make the call.
Questions to ask your team
- Which decision would we pick first, and who owns it on the business side?
- Do we have a few hundred past answers to that decision that we trust?
- Can the data involved leave our environment?
- What agreement rate and coverage would we need to see to switch it on?
- Who decides where the confidence line sits, and how often do we review it?
Claims to treat with care
A vendor may offer to run your pilot on their own benchmark data, or quote accuracy from other customers. Thank them, and use your own past decisions instead. The only result that should change your mind is the one on your data, in your process, measured against your best people. And remember that the shadow week is the evidence, not the demo.
That is the end of the course. If you take one habit from it, I hope it is this: when you look at an AI step in your business, ask whether it is writing something or deciding something. If it is deciding, what would it take for your team to tell you, with a number, how often it gets that decision right?
← Part 3 · Back to the start: Part 1 · Glossary · All courses
Member discussion