Internal system prototypes: how much fidelity is enough?

Less than it seems. In the user studies cited here, rough and polished prototypes found almost the same number of usability problems. Polish changes something else: how the audience reads the project. Before choosing a level of fidelity, decide which question the prototype answers: its role in the routine, the experience of using it, or the implementation.
Does a rough prototype find fewer problems?
In the user tests we consulted, no. The starting point is the study by Virzi, Sokolov and Karis, of GTE Laboratories, presented at CHI 1996. It had two experiments, each with 20 participants, ten per condition.
In the first, the product was a portable electronic book. Half the participants used the real device. The other half used more than 100 index cards, swapped by hand by a researcher. The device revealed 34 of the 38 usability problems identified. Paper revealed 32. The study does not report how many appeared in both conditions.
In the second, the product was an interactive voice response phone service. The working version took over two months to develop. The rough version was one of the authors reading the system prompts aloud. The result: 19 of 21 problems with the working version and 20 of 21 with the reading.
The difference was not statistically significant in either experiment. The authors add the caveats the data calls for: the sample is small and made up of college students, problem severity was not measured, and the absence of a difference does not prove the two methods are equivalent.
A 2010 study reached a similar result with a different kind of product. Sauer, Seibel and Rüttinger tested a walk-behind floor scrubber with 48 users, half of them experienced, in three versions: a paper and cardboard drawing, a partially operational mock-up, and the working machine. Fidelity changed neither the number of problems reported nor the usability rating.
What does a rough prototype fail to measure?
It measures poorly whatever depends on how the real system behaves. In the floor scrubber study, people using the reduced versions overestimated the water flow and brush pressure settings. Performance could only be measured on the working machine. The authors also recall a known weakness of the method: prototypes tend to be rated more generously than the final product.
Virzi and colleagues record an objection of the same kind, raised by peers: the result might not hold for systems that are extremely response-time sensitive. The authors say they are not so sure, but acknowledge that the exact bounds of their results await more research.
A 2017 review by Jeff Sauro of MeasuringU, a usability research firm, covers 11 studies and concludes that a rough prototype is a reasonable but imperfect substitute for finding problems. Task time is poorly predicted, and aesthetic ratings are less reliable. We use the review as a map of the literature. The numbers in this article come from the original studies.
In practice: if your question is "can the operator complete the task?", paper and a clickable screen usually answer it. If it is "how long does the task take with a real day's volume?", only the system running on real data answers it.
What does polish tell the audience?
It says the project is further along than it is. Stephanie Houde and Charles Hill, then at Apple, wrote in 1997 that finished-looking prototypes are often taken as a sign that the design is near completion. They add that inexperienced audiences tend to believe a prototype is more functional than it is simply because it is shown on a computer screen.
The paper has an example of a misread prototype. A digital movie editor prototype was built to test the implementation, and the discussion drifted to the menus. In the project leader's words, the interface was neither good enough nor bad enough to avoid misunderstandings.
So polish is a message. For someone who runs innovation, that matters in the board meeting. The polished screen helps get the budget approved and, by the reading Houde and Hill describe, can create the expectation of a short deadline in the same moment. They sum up the way out in one sentence: a key part of prototyping is making clear which aspects of the prototype correspond to the final system and which do not.
Which question does your prototype answer?
Houde and Hill propose replacing the fidelity scale with three questions. Role: what function does the system serve in the routine of the person using it? Look and feel: what is it like to use, on screen and in the hand? Implementation: through which techniques and components will it work?
For them, a prototype is any representation of a design idea, in any medium. One of their examples is a pizza box weighted to match a planned portable computer, carried by an architect on a site visit. The box showed that the rectangular shape was awkward to carry. It was a cardboard box, and it answered a question no screen would have answered.
From here on, the reading is ours. In an internal system, the three questions translate directly:
- Role: does the new step fit the routine, or does it create work in parallel to what the team already does?
- Look and feel: does the operator finish the task without help? We covered this when discussing who should test the prototype.
- Implementation: does the data exist, does it sit in a system that can be read, and does it arrive when the decision needs it?
The third is the one a good-looking screen does not touch, because a demo screen usually runs on made-up data. It is also where our thesis lives: the problem is rarely the technology of the screen. It is the architecture, the path the data takes between systems. An implementation prototype may have no interface at all. It can be a routine that fetches the real data for a week and shows what came back.
What does this look like in a company?
The scenario below is hypothetical, and its numbers are not measurements. An equipment rental company wants a portal where customers renew contracts on their own. The innovation manager commissions a navigable prototype in the company's visual identity and presents it to the board. The board approves and asks whether it can launch in 30 days.
The prototype answered the look-and-feel question. Nobody asked where equipment availability comes from. It comes from the ERP, which exports once a day. The portal would show as free a machine that was rented out that morning. That discovery arrives after the budget is approved and the deadline promised.
The other path uses three small prototypes, each with one question. A paper flow, walked through with two service agents, shows whether renewing through the portal removes the phone call or only postpones it. A simple clickable screen, tried by five customers, shows where they get stuck. A routine with no screen reads availability from the ERP for a week and compares it with what actually happened in the yard. The board gets the three answers, and not only the screen.
Two of our cases show constraints a screen does not reveal. At Reatop, a hospital waste control platform, unstable connectivity in basements and disposal areas ruled out traditional web solutions in the field. A constraint like that does not show up in a designed screen. The answer was a native mobile app that works offline. UniTrust, a US insurance IMO, had two weeks to put a system live in August 2022. With a deadline like that, deciding what stays unpolished is the design decision itself. And once the prototype convinces people, you still need to know what separates an approved prototype from a system.
Frequently asked questions
Will the board take an unpolished prototype seriously?
It depends on how it is presented. Houde and Hill note that less expert audiences usually cannot be expected to understand approximate representations, which is why the polished prototype has a place in the approval meeting. The care lies in saying out loud what is real and what is scenery. Bringing the result of the data test gives the board something the screen alone does not.
Won't building three prototypes take too much of the team's time?
Each one tends to be smaller than a single prototype that tries to answer everything. In the 1996 study, the working version of the voice service took over two months, and the version read aloud found 20 of the 21 problems, against 19. The team's time goes into the test sessions. In that study, it was building the working version that took the months.
How do you measure the result of a prototype?
Write down the question and the criterion before building. For look and feel, count the problems operators ran into and how many finished the task. For implementation, check whether the real data arrived complete and on time. Do not use a rough prototype to estimate task time, because Sauro's review indicates that this measure is poorly predicted.
The next step
If your prototype was approved on the strength of the screen and nobody tested where the data comes from, the Operational Architecture Diagnostic is a 30-minute conversation that starts with that question. Talk to an architect.
Read next
Related case studies

Did you recognize your operation in this article?
Book the Operational Architecture Diagnostic: 30 minutes to map where your operation's bottleneck is. No proposal in the first conversation.
Book a diagnostic