Innovation pilots: how many cases must yours see?
How many cases does a pilot need to see? Every case the operation produces often enough to appear within a year, including month-end close, cancellation, correction and the exception nobody documented. Duration does not measure that. A three week pilot can see a great deal or almost nothing, depending on which cases went into it.
Why did the pilot work when the system did not?
Because the two faced different populations. The pilot got whatever cases fit inside the window, and the window was chosen by the project calendar. The system gets the whole operation, including month-end close, the client who cancels midway, the order that comes back and the invoice issued wrong.
That is not a failure of pilot execution. It follows from how the pilot was sized. The question that usually opens the planning meeting is how long the test should run. The question that decides the outcome is which cases it has to contain.
We have covered how to frame the experiment in rapid prototyping: validate before investing, and what is missing after approval in prototype approved: what's missing to make it a system. This piece sits before both: what went into the test.
How many paths does a real process have?
More than a process diagram usually shows. You can answer that with data, because some companies have started donating their event records to research.
One of the best documented public cases is a Dutch multinational in coatings and paints, which in 2019 released for research the 2018 purchase orders of some of its 60 subsidiaries. That is 76,349 purchase documents, 251,734 items and 1,595,923 events recorded across 42 activity types, performed by 607 people and 20 automated processes (official log description).
Two students at RWTH Aachen, in the student category of the challenge, analyzed that log and counted 11,973 distinct activity sequences (Hüser and Heisenberger). A single case holds between 1 and 990 events.
The number that matters most to anyone designing a pilot is a different one. Six activities account for roughly 86% of all events, and the other 36 split what is left. A pilot sized by what appears most often sees the process through six words.
That shape is not unique to this log, and the opposite extreme exists too. A group at TU München cataloged 98 public logs of real processes across 37 attributes, among them the coverage of the five most frequent variants, and recorded variant counts ranging from 1 to 22,632 (Costa, Eroglu, Andree and Pufahl). Where your operation falls in that range is an empirical question, not an assumption.
What the pilot finds that the diagram did not know
The useful part of a test is rarely the confirmation. It is the gap between the process a company believes it has and the one it executes.
In the purchasing log, the company believes one specific group of items always goes through a cross check between goods receipt and invoice receipt. Within the dominant category of the log, the analysts found around 14,000 cases, across 747 different vendors, where the two records did not appear together. The control existed in the document and not in the execution. Write the pilot from the document and you inherit that entire gap.
In the same category, 53,092 cases required a payment block to be lifted before the invoice could be cleared, and 7,800 ended with the purchase item deleted, an outcome that adds 72 days on average to the case. None of those three is the path anyone draws on a whiteboard.
A second log, from another sector, shows the same kind of drift at smaller scale. Following 1,050 patients with suspected sepsis at a Dutch regional hospital over a year and a half, Felix Mannhardt and Daan Blinde measured, in a study published in 2017, adherence to two clinical time rules: antibiotics within one hour of triage and lactate within three hours (BPMDS report).
The first rule was violated in 58.5% of cases. The second, in 0.7%.
The authors themselves do not read that 58.5% as a verdict on care. They offer two competing explanations: the one hour rule is acknowledged to be strict and not every patient in the sample had severe sepsis; and the antibiotics timestamp is typed in by hand, with records 24 hours after triage and others preceding it. Their conclusion is that the measurement alone does not support action.
For anyone sizing a pilot, that is the lesson. The distance between 58.5% and 0.7% can be a rule that does not fit the operation or a record that does not capture what happened, and the two hypotheses separate only by measuring.
How to size coverage instead of the calendar
Start with the list, not the schedule. Before picking a start date, write down the cases the pilot has to contain, with a name and a minimum count each:
The common case, to check the basic design works. The exception the team resolves outside the system today, with a spreadsheet, a message or a phone call. The correction case, when someone makes a mistake and has to undo it. The cancellation midway through the flow. The periodic case, which exists only at month or quarter close.
That last one is the one that escapes most often. In the multinational's purchasing process, a rent item creates one order and twelve goods receipts across the year, one per month, each with an invoice for a twelfth of the value. Logistics items reach hundreds of goods receipts on a single line. A three week pilot sees none of that, and not because it is short: because it does not contain the case.
When a case on the list cannot be produced within the period, you have two honest options. Inject the case deliberately, by reprocessing a real record from last year inside the pilot. Or declare the case uncovered, in writing, and carry it as an open risk into the promotion decision.
There is a third option, which is assuming it will work because the others did. That is the one that tends to produce the system that breaks at the first month-end close.
How this works in practice
The scenario below is a working hypothesis, not a client measurement. It is here to show the difference between the two ways of sizing.
An equipment maintenance company billing around USD 300,000 a month tests a work order system. The pilot runs three weeks with two technicians and closes 60 orders without incident. The recommendation is to promote it.
Rewritten by coverage, the same pilot asks for something else. Thirty ordinary orders. Five jobs with a part under warranty, which the technician settles today by phone with the supplier. Five orders opened and then canceled by the client after the technician had already traveled. Three rework cases, where the equipment comes back with the same fault inside 30 days. And the monthly contract close, which has to be reprocessed with data from a past month because it does not fit the window.
The thirty ordinary orders probably pass under either design. The other thirteen decide whether there is a system or a good looking screen with a parallel operation behind it.
The case the pilot missed needs an address
This is the architectural part, and it holds even for a well sized pilot, because complete coverage does not exist.
The unforeseen case eventually happens. The only real choice is whether it happens inside the system or outside it. With no defined place, it leaves. It becomes a message on somebody's phone, and then it becomes knowledge only that person holds. That is how the operation became what it is today.
Three decisions settle this, and all of them fit before go-live. Where the off-pattern case gets recorded, under the same identifier as the ordinary case. Who has the authority to resolve it, and within what deadline. What state the record is left in afterwards, so next month's report knows how to count it.
Without those three, the system carries a declared coverage the operation does not respect, and the numbers coming out of it stop matching finance. That is the same mechanism we described in management dashboards: why the numbers don't match.
The hospital study leaves one more lesson, about method. The process model generated automatically by the algorithms was too imprecise to discuss with doctors and nurses. The model drawn by hand, in rounds with the people doing the work, covered 98.3% of the log. Reading the data correctly still depends on the people who operate.
Frequently asked questions
Will the implementation take too much of my team's time?
Sizing by coverage usually costs less time than sizing by calendar, because it trades weeks of passive observation for chosen cases. A pilot that injects five exceptions in week one learns more than one that waits three weeks to see whether any turn up. The real cost is reprocessing old records, and that tends to be hours of work rather than sprints.
I don't see how to measure the pilot's result.
Measure coverage before you measure performance. The list of cases written at the start becomes the scoreboard: how many ran inside the system, how many escaped to a spreadsheet or a message, how many nobody managed to produce at all. That scoreboard is verifiable and does not depend on user sentiment, unlike the impression that the test went well.
I'm worried the system won't actually clear the bottlenecks.
That worry is well founded, and coverage is what addresses it. Bottlenecks rarely live in the common case, which is already fast because it repeats. They live in the exception, the rework and the wait for a decision. A pilot that only saw the common case has no way to demonstrate anything about the bottleneck, and approving on that basis is deciding without evidence.
Next step
If your pilot is about to be promoted and the list of cases it actually saw was never written down, a 30 minute conversation is worth having: the Operational Architecture Diagnostic maps which paths your operation produces before the decision gets made.
Related case studies
Did you recognize your operation in this article?
Book the Operational Architecture Diagnostic: 30 minutes to map where your operation's bottleneck is. No proposal in the first conversation.
Book a diagnostic