Who should test the prototype of an internal system?
Test the prototype with the people who will run the system every day, not only with the people who approve the budget. A demo for the board measures whether the idea is convincing. Five short sessions with real operators, on real tasks, show where the system stalls, and that is what holds up the decision to invest.
Why is a demo for the board not enough?
Because it tests persuasion, not use. In a demo, someone who knows the prototype drives the clicks, uses clean data and follows the happy path. The board sees a screen that works. The operator, on a Tuesday at four in the afternoon, will have an incomplete record, a customer who is not on the list and an urgent case that does not fit the designed flow.
This is not a fault of the approver. Approving is legitimate work, and the board needs to see the whole. The problem is treating the approval as if it were the test. They are two different questions: the board asks "does this solve the business problem?", and the operator asks "does this fit my day without slowing me down?". Only the second decides whether the system gets used.
If you run innovation, there is an extra complication. The person who signs is usually the one you can reach most easily. The people who operate usually sit in a function that does not report to you. Testing with the group that is within reach is the natural path, but it tells you least about real use.
What does the research say about how many users to test?
It says that a few people of the same profile already reveal a good share of the problems, and that the profile matters more than the number. The data has limits, and they belong in the same breath.
Jakob Nielsen, in a 2000 article for the Nielsen Norman Group, uses the formula N(1 - (1 - L)^n), where L is the share of problems a single user finds. In the projects he and Thomas Landauer studied, L averaged 31%. With that value, five users find about 85% of the problems, and after the fifth the findings start to repeat. Nielsen recommends splitting the test budget into rounds: test with five, fix, test again with another five, because a fix may not work or may create a new problem.
The same article carries the caveat that matters most here. The formula holds for comparable users. When a system serves very different groups, each group has to be tested, with 3 to 4 users per group when there are two.
Laura Faulkner, in a 2003 study in Behavior Research Methods, Instruments, & Computers, tested 60 users and drew random sets from them. Some sets of five found 99% of the problems, others only 55%. With ten users, the worst set rose to 80%, and with twenty, to 95%. The practical reading: five is a cheap starting point, but the variation is large, and a mixed operational group calls for more people.
Both studies measure usability problems found in a test. They do not measure whether the system fits the process or whether a business result follows. Nielsen's figure dates from 2000 and comes from the projects he studied, not from a specific company's internal systems. Use it as an order of magnitude for planning the round, not as a promise.
And the evidence on involving users in the design?
It is favorable but limited. Bano and Zowghi published a systematic review in 2013 covering 87 empirical studies published between 1980 and 2012. Only 13 of them looked specifically at user involvement in requirements engineering. Of those, 9 confirmed benefits and 4 were inconclusive.
That result does not isolate the prototype, and the review is about requirements, not testing. Our reading, which does not come from the study: the cost of hearing five operators before you build is small next to the cost of finding the same problem after go-live, with the system paid for and the team drifting back to the spreadsheet.
Who needs to be in the room, and in what role?
Three roles, with different functions. The operator tests: the person who will do the task every day and who knows what happens when the case leaves the standard path. The line manager tests the control view, the report and the screen used to follow results. The approver observes, preferably without speaking, because watching the operator stall tends to persuade more than a presentation argument does.
Three rules change the quality of the test. Use real data, anonymized if needed, because a prototype with sample data never meets the messy record. Ask for tasks taken from the operator's real week, not a script the prototype already knows how to pass. And do not help: every time someone explains what to do, the test loses the point where the system, alone, fails to explain itself.
Each session should leave a short record: the task, whether it was completed without help, where the person stopped and which old route they tried to use instead. That record is what you take to the board, and it is stronger than an opinion about the prototype.
A practical example: the clinic front desk
The scenario below is hypothetical. The numbers and results are illustrative assumptions, not measurements. A network of six clinics prototypes a scheduling tool with a payer pre-authorization check. In the demo, the director sees the patient booked, the authorization queried and the approval document attached in a few clicks. She approves.
Now imagine five front-desk staff from different clinics, each with the real list for the day. Two stop at the same point: the patient arrives without the authorization number, and the prototype requires the field to save. Their way out is to jot it on paper and enter it later, which is exactly the route the system was meant to replace. A third notices that the screen shows the authorization but not the window the payer gave for using it.
None of these problems appeared in the demo, because nobody driving the clicks ever arrived without the number. In this scenario, fixing the required field in the prototype is a screen change. Discovering it in production is a sheet of paper on the counter and a parallel spreadsheet. The decision to invest now rests on what the operator managed to do, not on what the board thought of the screen.
How do you organize a test round in one week?
Start with the definition, before any session. List five to eight real tasks from the operator's week, choose the success criterion for each (completed without help, time to first stall, old route used) and record the starting point of the current process. Without that reference number, the test result becomes opinion.
Then invite five operators of the same profile and, if there is more than one profile, repeat the group. Sessions of 45 minutes, one person at a time, with the approver and the line manager observing. That time estimate is our design recommendation, not a research figure. At the end of the week, fix what stalled more than one person and run a second round with other operators, as Nielsen suggests.
The test answers the usability question. The architecture question, where the true data comes from and what happens on the exception, is a different one, covered in rapid prototyping: how to validate before investing and in prototype approved: what makes it a system. In the Reatop case, the native mobile app with offline use was designed for people who operate in the field, tied to an integrated back office, and that is the kind of system where testing with the front line matters most.
Frequently asked questions
My team resists change. Does putting operators in the test invite more resistance?
In our reading, the effect tends to be the opposite. People who test see their own change request turn into a change in the system, and that builds ownership. The resistance that remains shows up early, when fixing is cheap, and not after go-live. Pick operators their peers respect, not only the most enthusiastic ones.
Implementation will take too much of the team's time. Can we test without stopping operations?
Yes, if the round is short. Five sessions of 45 minutes, scheduled with the line manager, take each operator for less than an hour. That sizing is our design recommendation. The time the test costs is less than the time of maintaining two routes after go-live.
I do not see how to measure the result of a prototype test. What should I record?
Record per task: whether it was completed without help, where the person stopped and which old route they tried to use. Measure the current process beforehand, so you have a comparison. The record does not measure the project's financial return, but it shows whether the system fits the routine, which is the condition for any return to appear.
The next step
If you have a prototype your board has approved and want to understand what the operator has not yet seen in it, the Operational Architecture Diagnostic, a 30-minute conversation, is an invitation to start: talk to Yowpi.
Related case studies
Did you recognize your operation in this article?
Book the Operational Architecture Diagnostic: 30 minutes to map where your operation's bottleneck is. No proposal in the first conversation.
Book a diagnostic