# When to measure the result of an innovation project > Innovation projects get worse before they get better. When to open the measurement window, and what to record so the right project does not get killed. - Author: Marlon Trettin - Published: 2026-10-18 ยท Updated: 2026-10-18 - Language: en - Canonical: https://yowpi.com/en/blog/when-to-measure-an-innovation-project --- You measure after the operation settles, not when the system goes live. An innovation project usually gets worse before it gets better: the team relearns the work while the process rearranges itself, and the numbers fall in that interval. Measuring inside the dip produces a verdict about the transition, not about the project. ## TL;DR - A US Census Bureau study covering tens of thousands of American manufacturing establishments found causal evidence of J-curve returns: short-run performance losses before the gain. In the correlational measure, controlling for size, age, capital and IT infrastructure, a one-standard-deviation increase in the AI index comes with a 1.33 percentage-point drop in productivity ([McElheran, Yang, Kroff and Brynjolfsson, 2025](https://www2.census.gov/library/working-papers/2025/adrm/ces/CES-WP-25-27.pdf)). - The loss concentrates in older firms, for a specific reason: they stop sustaining structured management practices such as monitoring KPIs and production targets. That pullback accounts for about one third of their productivity loss. - The gain shows up later. Between 2017 and 2021, firms already using AI in 2017 grew more in revenue, labor productivity and employment. The authors record that result conditional on survival, which is the point of this article: a project killed in the dip never makes it into the gain column. - Only 25% of companies worldwide have fully deployed 40% or more of their AI experiments, and 54% of global leaders expect to move at least 40% into operation within three to six months ([Deloitte, State of AI in the Enterprise 2026](https://www.deloitte.com/us/en/about/press-room/state-of-ai-report-2026.html)). ## Why does the number get worse right after go-live? Because the old routine stopped working and the new one is not automatic yet. In that interval the operation pays two costs at once: the process being dismantled and the process that does not yet pay off. Economists call this the J-curve. The Census Bureau research led by Kristina McElheran, Mu-Jeung Yang, Zachary Kroff and Erik Brynjolfsson linked two census surveys from 2017 and 2021 and found the pattern with causal evidence: fall first, growth later. [MIT Sloan](https://mitsloan.mit.edu/ideas-made-to-matter/productivity-paradox-ai-adoption-manufacturing-firms) summarizes McElheran's reading that the technology is not plug-and-play and demands systemic change, which creates friction, mostly at established firms. The study is about industrial AI in factories. The mechanics are not: they hold for an ERP replacement, a systems integration or a workflow redesign, because what produces the dip is the team relearning the work, not the particular technology that arrived. The finding most useful to anyone running innovation is a different one. At older firms, what degrades in the short run is the management, not the machine. They stop monitoring indicators and production targets during the adjustment, and that retreat accounts for about a third of their productivity loss. The instrumentation loosens in exactly the month you need it most. ## What is the right measurement window? The window opens when the series settles, not on a calendar date. Before that you have two different populations of events mixed into the same period: those that went through the transition and those already running in the new regime. The monthly average adds them together and returns a number that describes neither. If the transition took six weeks and you measure the whole quarter, half the sample is learning. The good project looks mediocre in the report. The bad project looks salvageable, because the blame fits inside the learning. The signal of stability is in the dispersion, not the average. While the team is still learning, the same type of case takes very different amounts of time to move. When the spread between similar cases stops shrinking month over month, the new regime has formed and the average starts to mean something. That plateau alone does not tell you the project worked. It tells you that you can start measuring. The answer about the result comes from the level at which the dispersion settled, compared with the level before go-live. Notice the mismatch between that requirement and the deadline companies give themselves. In the Deloitte survey of 3,235 business and IT leaders across 24 countries, 54% of global leaders expect to move at least 40% of their experiments into operation within three to six months. That is a scaling expectation placed inside the interval where the curve is still falling. ## What does your system need to record to separate the two phases? Event-level records, with timestamps. Without them you cannot say which week the curve turned, and without that date there is no measurement window, only the monthly close. In practice it is three things, and none of them is a new dashboard: 1. When each case moved. Not the completion date, but the time of every stage transition. That is what lets you reconstruct the series afterwards, including retroactively. 2. Which path it took. New flow, old flow, or the exception someone opened outside the system. If all three land in the same bucket, the new flow's result gets diluted by volume that never passed through it. 3. Which version of the process was in force. A simple regime marker, updated when the process changes. Without it, the comparison with the past compares different things. This is an architecture decision, made before go-live. Once the quarter closes without those fields, no analysis recovers the information: the data was never written. It is the same problem covered in [management dashboards: why the numbers do not match](/en/blog/management-dashboards-why-numbers-dont-match), seen from the project-measurement side. Two subjects get confused here. In [how to measure innovation ROI and prove it](/en/blog/measure-innovation-roi-and-prove-it) the topic is the baseline and attribution of the gain, meaning what you compare against. Here the topic is when you compare. You can get the baseline right and still read the number in the wrong week. ## An example, and it is a hypothesis Stated plainly: the case below is constructed to illustrate the reasoning. The numbers are illustrative and did not come from any Yowpi measurement. A distributor replaces manual order checking with a flow that validates automatically. In the first month, average release time goes up, because the team double-checks what the system decided and still redoes part of it in the old spreadsheet. In the second, the average returns to its previous level and the board concludes the investment changed nothing. What nobody looked at: between orders of the same type, the time difference had halved. Simple cases were already passing straight through, and the complex ones were concentrating the delay. That was the new flow working on the part it set out to solve, hidden inside an average that still carried the rework of the transition. Without a timestamp per stage and without a marker for which path each order took, that reading does not exist. Only the average exists, and the average said the project was useless. ## What to agree on with the board before you start Write down four things before the first line of code, somewhere the board can consult later: - Which business indicator the initiative moves, chosen from the ones the board already tracks. - How long the transition is expected to last, with the date the measurement window opens. - Which signal indicates the regime has settled, so the date is revisable on evidence rather than convenience. - What the stop condition is, meaning what has to happen for the project to end before the window opens. The last one protects the other three. Without a written stop condition, any bad number mid-transition becomes an argument to end it, and any project without results becomes an argument to wait another quarter. Agreeing on the criterion in advance takes both conversations out of improvisation. The shutdown criterion itself is covered in [when to shut down an innovation project that does not scale](/en/blog/when-to-shut-down-an-innovation-project). What changes here is the order: the stop condition has to exist before the measurement window opens, otherwise it gets written after the bad number, by whoever the bad number bothers. This agreement also changes the conversation about adoption. A system barely used in month two is expected; in month six it is a symptom. The difference is in [measuring real usage after go-live](/en/blog/system-adoption-why-go-live-is-not-the-end), not in asking the team whether they liked it. ## Frequently asked questions **I cannot see how to measure the result. Where do I start?** With the field, not the dashboard. Pick an indicator the board already uses and check whether the system records the time of each stage transition in that process. If it does not, that is the first piece of work. A dashboard built on monthly aggregates cannot answer when the result appeared, and that question decides the verdict. **I cannot justify the investment right now. Does this change anything?** It changes what you promise. A business case promising a gain in the first quarter delivers a drop and loses credibility for the next request. Declare the transition, say how long it should last and mark the date measurement begins. The result then holds up when it arrives. The J-curve evidence supports presenting it that way, and it comes from public research, not from a vendor. **I worry it will not remove the bottlenecks. How do I find out sooner?** Look at the dispersion instead of waiting for the close. If the team already knows the flow and the dispersion settled at the same level as before the project, the bottleneck is where it always was and only the interface changed. That signal appears weeks before the quarter ends, with time to correct scope while the project is still running. ## The next step If your operation today only produces a monthly average, the Operational Architecture Diagnostic is a 30-minute conversation about what your systems record and what is missing to measure a project at the right moment: [book one when it makes sense](/en/contact).