# Priority scores: why two people score differently > Two people give one initiative different scores because the criterion is ambiguous. Test your queue's disagreement in one afternoon before trusting it. - Author: Marlon Trettin - Published: 2026-10-31 ยท Updated: 2026-10-31 - Language: en - Canonical: https://yowpi.com/en/blog/priority-scores-why-two-people-score-differently --- Two people give the same initiative different scores because the criterion, usually "impact", does not say what to count. That variation between scorers has a name, noise, and you can measure it before you trust the queue. Ask three people to score the same five initiatives, each alone, and compare the scores. ## TL;DR - In a noise audit at an insurance company, the median price difference between two underwriters for the same policy was 55% ([strategy+business](https://www.strategy-business.com/article/How-noisy-is-your-company)). The company's executives expected about 10% ([Insurance Thought Leadership](https://www.insurancethoughtleadership.com/our-big-problem-with-noise/)). - A priority score is a numbered opinion. The average hides the disagreement, and the disagreement shows which criteria nobody defined. - The test fits in an afternoon: the same five initiatives, three scorers, independent scores before any discussion. - To reduce the disagreement: written anchors for each score, criteria split into smaller parts, and a record of each scorer's score, not just the average. ## Why does a priority score look more objective than it is? Because it is a number. A score from 1 to 5 on "impact" looks like a measurement, goes into a spreadsheet, produces an average and yields a ranking. What it actually stores is the judgment of whoever scored, and each person read the word "impact" in their own way. Finance reads cash and IT reads risk avoided. Operations reads hours handed back to the team. All three are right, and the average of the three measures none of those things. When the meeting ends with a ranked queue, what got recorded is the result of a calculation, not the three readings that fed it. This is an architecture problem, not a people problem. The criterion was written as a word, not as a question with a single answer. ## How far apart do two people land on the same case? Further than you would expect, according to what has been measured in other settings. In the book *Noise*, Daniel Kahneman, Olivier Sibony and Cass Sunstein describe an audit at a large insurer in which [48 underwriters assessed the same cases](https://www.insurancethoughtleadership.com/our-big-problem-with-noise/). The company's executives expected about 10% variation between prices. The median difference found was 55%. The [strategy+business](https://www.strategy-business.com/article/How-noisy-is-your-company) review of the book records the same pattern at a larger scale: when 828 senior executives were asked how much variation they expected in expert judgments, the median answer was 10%. According to the review, the actual variation can be four to five times that. These numbers have a limit. An insurer is not an innovation queue, and we found no published measurement of the disagreement in priority scores for technology initiatives. The mechanism is the same, though: competent people reading the same case with a criterion each of them interprets in their own way. Whether your queue shows disagreement of that size is a hypothesis, and the next section shows how to test it. ## How do you measure the disagreement in your own queue? With a noise audit in miniature. The method is described in [Harvard Business Review](https://hbr.org/2016/10/noise): members of one unit assess a common set of cases, and the degree to which their assessments vary measures the noise. In practice, four moves. Pick five initiatives already in the queue, ideally from different areas. Pick three people who usually take part in prioritization. Ask each to score all five alone, on the same scale and with the same written criterion. Only then put the scores together and calculate, for each initiative, the distance between the highest and the lowest. On a 1 to 5 scale, a 3-point distance on one initiative means the people were not assessing the same thing. There is no use in arguing about who is right. The useful question is which reading of the criterion each of them made. ## What reduces the disagreement without making the decision rigid? Four practices, all of the same kind: take out of the criterion whatever depends on interpretation. The strategy+business review sums up the set the authors call decision hygiene: do not let judgment reach a conclusion too early, aggregate independent judgments, use shared scales and break complex judgments into smaller parts. The first is writing anchors. Instead of "impact from 1 to 5", define what a 5 is and what a 1 is: for example (illustrative values), a 5 is more than 200 hours a month handed back to one area, measured against the baseline, and a 1 is fewer than 20. The exact number is your decision. What matters is that two people, given the same data, reach the same score. The second is splitting the criterion into parts. "Impact" becomes hours handed back, money avoided and risk reduced, each with its own scale. The weighted sum comes later, and the weight of each part is an explicit board decision, not an accident of the conversation. The third is scoring before discussing. Whoever speaks first anchors the group. With written, independent scores, the discussion starts from the disagreements, which are the informative part, not from the consensus. The fourth is record architecture. The system where the queue lives has to keep each scorer's score, with the date and the anchor used, and not just the average. When only the average survives, the disagreement disappears from the history, and nobody can tell later whether an initiative moved up the queue on merit or because of who was in the room. It is the same logic as the [initiative inventory](/en/blog/initiative-inventory-you-prioritize-what-you-can-see): the queue is only reliable if what feeds it has an owner and a record. ## A practical example: five initiatives, three directors The scenario below is hypothetical and the numbers are illustrative assumptions, not measurements. A clinic network with twelve sites has five initiatives in its queue. The operations, finance and IT directors score each one's impact from 1 to 5, alone. Initiative A gets 5, 4 and 4. B gets 3, 3 and 3. D gets 2, 2 and 3. On these three the scores move together and the average is reliable. Initiative C gets 5, 2 and 3, and E gets 4, 5 and 2. The averages of C and E (3.3 and 3.7) put them in the middle of the queue, and the meeting would treat them as middling initiatives. The 3-point distance tells a different story. On C, operations saw hours handed back to reception desks and gave a 5, while finance found no cash and gave a 2. On E, IT gave a 2 because the initiative creates one more integration to maintain. Neither is middling. Each has a clear benefit for one area and a clear cost for another, and the average erased that information. The right conversation is about what they cost and to whom, before any position in the queue. This is the kind of reading that [prioritization by dependency and bottleneck](/en/blog/prioritize-innovation-and-get-past-the-pilot) asks for: knowing what each initiative unlocks and what it pushes onto another area. And if the decision is going to the board, [showing a cost range instead of a single number](/en/blog/tech-project-estimates-give-the-board-a-range) follows the same principle of exposing uncertainty instead of hiding it in an average. ## Frequently asked questions **My team resists change. Won't scoring initiatives with written criteria look like bureaucracy?** It can, if the criterion arrives finished and unexplained. Start with the one-afternoon test, without creating any new rule: show the people who scored how far apart their scores were. Someone who sees their own 5 next to a 2 usually asks for the criterion instead of receiving it as an imposition. **I don't see how to measure the result of prioritization. How do I know the queue got better?** Measure the disagreement before and after writing the anchors, using the same five initiatives or equivalent ones. If the average distance between scorers drops, the criterion got clearer. That does not prove the queue is more accurate, but it is a step you can measure, and a ranking can only be better if it is reproducible to begin with. **Will implementation take too much of my team's time?** The test takes one afternoon of three people and a spreadsheet. Writing the anchors takes one more short conversation. The large cost comes later, if the queue keeps no record of who scored what: every review goes back to arguing from zero about what has already been decided. ## The next step If your initiative queue today is an average nobody can open, the Operational Architecture Diagnostic is a 30-minute conversation to look at how the scores are generated and recorded and what that changes in your order of priorities: [book it through our contact page](/en/contact).