Measuring innovation results: before and after is not enough
Before and after does not prove the project worked, because it mixes the project's effect with everything that would have changed on its own. To prove it, compare the unit that changed with a similar one that has not changed yet, measured the same way over the same window. You can do it without a lab, by staging the rollout.
Why does before and after mislead even when the data is right?
Because the "after" number carries the project's effect plus three other things: seasonality, whatever changed in the market or the team over the same period, and regression to the mean. The data can be correct and the conclusion wrong.
Whoever approves a project usually sees two snapshots. In March the indicator was 28, in September it is 21, so the project cut it by 25%. Nothing in those two snapshots separates what the project did from what would have happened anyway. The question the board should ask is "compared with what?", and before and after has no answer to it.
What is regression to the mean, and why does it flatter your project?
It is the phenomenon in which an extreme measurement tends to be followed by one closer to the average. Adrian Barnett, Jolieke van der Pols and Annette Dobson describe the effect, in a 2005 paper in the International Journal of Epidemiology, as common in repeated data and say it should always be considered a possible cause of an observed change. According to the abstract, the effect is stronger when there is more measurement error and when follow-up looks only at a subgroup selected by a baseline value.
The application to management is ours, not the paper's. An innovation project rarely starts in a good month. It starts after a quarter of delays or a spike in returns. If the indicator was at an extreme when you chose the unit, part of the improvement that follows may come only from a return to the normal level. The same goes for the unit chosen because "it suffers the most": it was selected by its baseline value, which is the situation where the effect weighs most.
What do controlled experiments show about ideas that seem to work?
They show that seeming to work is a weak criterion. Ron Kohavi and Stefan Thomke report in the 2017 Harvard Business Review that at Google and Bing only 10% to 20% of experiments produce positive results, and that at Microsoft as a whole one third are effective, one third are neutral and one third are negative. Every experiment had a comparison version running at the same time, which is why the company knows which third each idea landed in.
The same article has a Yahoo case. The observational part of the study estimated that ads shown on Yahoo sites had raised brand searches by 871% to 1,198%. When the company ran a controlled experiment, the increase was 5.4%. Other variables had changed over the observed period, and the observational analysis could not separate them.
There is a limit. These are online experiments at technology companies, with a volume of users that an operation with four warehouses does not have. What carries over is the method, not the hit rate: without a comparison version, the idea that worked and the idea that did not look the same in the report.
How do you create a comparison unit without building a lab?
By staging the rollout. Instead of switching the new process on in every unit on the same day, switch it on in some of them and leave the others waiting, with the same indicator measured in both.
It works in four steps. First, choose units that are similar in size and type of work. Second, define the indicator and the measurement window before you start, not after you see the numbers.
Third, do not decide who goes first by enthusiasm. If the most interested manager goes first, her unit was probably already the best one. In the experiment at Ctrip, a Chinese travel company, employees volunteered and a lottery decided who would work from home for nine months. Performance rose 13%, about 9 points from more minutes worked and 4 from more calls per minute, according to Nicholas Bloom, James Liang, John Roberts and Zhichun Jenny Ying (NBER, 2013). The subject was different. The design is what matters: volunteers, and an order set by lottery.
Fourth, measure the units through the same system. If one records in the new system and the other in a spreadsheet, the difference may be one of recording, not of result. That point is the subject of Measuring automation ROI without a system of record.
When there is only one unit, the comparison gets weaker but does not disappear. Use a long series of the indicator before the project, to see the size of normal fluctuation, and track a second indicator the project should not move. That is our own practice, not one of the studies cited. If both rise together, the cause is probably not the project.
A practical example: order checking across four warehouses
The scenario is hypothetical, and the numbers are illustrative, not measurements. A distributor with four regional warehouses and 220 people wants a checking system at dispatch. The indicator is the number of orders returned for picking errors per 1,000 orders. In March, all four warehouses are at 28.
The innovation manager rolls the system out in warehouses A and B and leaves C and D waiting six months, with the same indicator measured in all four. In September, A and B are at 21 and C and D are at 25.
Looking only at A and B, before and after shows a drop of 7, or 25%. With the comparison, the drop attributable to the project is 4: A and B improved by 7, and C and D improved by 3 without the system. More than 40% of the apparent improvement would have happened anyway, through seasonality or regression to the mean, among other causes. The argument to the board moves from "we dropped 25%" to "the project cut 4 returns per 1,000 orders, about 14% of the baseline". That number holds up against the question "compared with what?".
The cost is real: C and D waited six months for a system that appears to work. That is why the window and the stopping criterion are agreed in advance. When to measure an innovation project covers how to choose that window without measuring inside the transition period.
This kind of comparison only exists if the warehouses record their work in a structured system. In the SuperVida case, contract management moved from an improvised Airtable base to a custom-built system, the kind of record that lets you measure two units the same way. That is why proving a result is usually an architecture problem, not a statistics problem: without an equal record across units, no comparison holds.
Frequently asked questions
I do not see how to measure the result. Can you prove it without a comparison unit?
You can estimate it, with less force. With a single unit, use a long series of the indicator before the project and track a second indicator the project should not move. The conclusion is more fragile, and the report should say so.
I cannot justify the investment right now. Isn't leaving a unit waiting a waste?
It is a cost, and it shows up in the request. In return, the board gets a number that withstands the question "compared with what?", and approval for the other units rests on data. The unit that waits gets the system later, with a rollout already adjusted by what the first ones showed.
I fear the project will not really remove the bottlenecks. How do I know if it worked or if it was luck?
Compare the difference between units with the indicator's normal fluctuation before the project. If the difference fits inside the usual fluctuation, the evidence does not support the conclusion that the project worked. The right decision may be to wait another window or widen the comparison.
The next step
If your next project is going to be judged by a before and after, the Operational Architecture Diagnostic, a 30-minute conversation, is an invitation to design the comparison unit, the indicator and the window before you start: talk to Yowpi.
Related case studies
Did you recognize your operation in this article?
Book the Operational Architecture Diagnostic: 30 minutes to map where your operation's bottleneck is. No proposal in the first conversation.
Book a diagnostic