Chapter Five

Clarify what matters – Explore criteria

So far, in Part 2 you have explored alternatives. This chapter shows you how to explore criteria that you can use to evaluate these alternatives. You will learn to identify what matters to you and to your stakeholders so that you can deliberately weigh off an alternative’s benefits and drawbacks. You’ll also learn how to prioritise criteria so that they best represent the group’s viewpoint.

Picture this: You face a difficult decision that requires agreeing with one of your colleagues, and it looks like you won’t agree. You have scheduled a meeting where you hope to reach common ground. Before the meeting, you remind yourself to be open minded and you commit to seeing the situation from her viewpoint. But once the meeting starts, your colleague states right away what he wants to do and why. You counter with your preferred alternative, revealing an unbridgeable gap. Both of you dig in your heels, emotions dial up to 11, and you get nowhere. It’s a stalemate.

All too often we quickly jump into defending the alternative we prefer. This is what Center for Applied Rationality’s Julia Galef calls a soldier mindset.1 But our intuitive judgment improves much if we delay choosing until the end of a structured process.2 Therefore, it is advisable to start with an open mind or, as Galef calls it, a scout mindset. Part of this open-mindedness is developing various alternatives, which we did in the previous chapter. Another part entails expressing what matters to us as we get our treasure – our decision criteria.

Making strategic decisions requires pursuing competing objectives and, thus, inevitably involves making trade-offs.3 For instance, you might be developing a go-to-market strategy for a new product that must be effective and low-cost and aligned with your organisational values. As a result, you will need to give away something that you value for something that you value more. In other words, whichever alternative you select will have a cost. Realising that there is no free lunch is a good reminder that if something seems too easy in the decision process, you’re probably missing something.

Just like we have a tendency to settle too early for a limited set of alternatives, we naturally think of only a portion of the evaluation criteria. We think we know what we want, but a closer examination often shows us wrong. Now, to be fair to ourselves, criteria don’t just pop up in neat lists, and complex decisions often require deep soul searching and iterating between the question, the alternatives, and the criteria. Furthermore, what matters differs from one stakeholder to another, so there is often not one objectively superior alternative. This is worth repeating: Complex problems do not have one right answer, only better (and worse) ones.4 We are not solving for the right answer, we’re solving for an excellent alternative or, at least, an acceptable one.

So, let’s look at what makes a good set of criteria and how we can develop it. The good news is that we can leverage some of the tools from previous chapters, as a good set has criteria that are reasonably collectively exhaustive, mutually exclusive, and insightful.5

ASSEMBLE A REASONABLY COLLECTIVELY EXHAUSTIVE SET OF CRITERIA

A good list has criteria that are distinct from one another, so that you don’t double count anything. It is also reasonably collectively exhaustive, so that you don’t forget anything important.

A basic universal set of criteria might be feasibility + desirability – as you will want to choose an alternative that you can implement (feasibility) and that you want to implement more than the others (desirability). Although this criteria set applies to virtually all decisions, it remains generic – and it is usually worth clarifying what feasibility and desirability mean for your specific decision.

Criteria are not just beneficial to help you choose among alternatives but can also serve as a launchpad for creating new alternatives. Think of the house purchase example: If we had been more explicit about the quietness consideration from the outset, it might have led us to look at other neighbourhoods or even other villages.

Note, however, that you want to keep the size of the set of criteria manageable. It should balance being collectively exhaustive with being parsimonious, which might mean excluding secondary criteria.6

SELECT MUTUALLY EXCLUSIVE CRITERIA

Often, criteria overlap. Consider for instance the trip from NYC to London. You might care about leg space, privacy, and comfort. Although all of them are potentially important, you are mixing up conceptual levels. Leg space, for instance, might be a sub-category of comfort. If you consider them as separate criteria on the same conceptual level, you double count some aspects of alternatives, which introduces problems in our weighted sum approach.

Now, in the case of comfort, it is reasonably easy to identify that some criteria overlap. However, with more abstract criteria, such as organisational fit, alignment with cultural values and other similarly intangible notions, it can be more challenging to explore criteria that are conceptually distinct from one another. Nonetheless, it’s critical that you create this clarity both for the sake of your problem-solving efforts and for communicating your conclusions later on. If your audience gets confused because you have not been clear in your thinking, they are less likely to trust your judgment and be convinced by your arguments.

How do you go about doing so? By always asking why something matters. For instance, you might say, leg space is important to you. Why? Because it allows me to stretch out my feet. Why does that matter? Because I can avoid cramps. Why is that important? Because it helps me be more comfortable during the journey. Here you might stop and say that this is enough specificity and that you don’t need to justify why it’s important to be comfortable. The act of asking why a few times has helped you move from instrumental criteria to fundamental criteria. It’s those criteria that you ultimately want to use to evaluate your alternatives. As a rule of thumb, if two criteria score similarly across alternatives, you might want to explore if one might be a sub-criterion of the other. If that’s the case, clean up the set!

SELECT INSIGHTFUL CRITERIA

Good criteria shed light on what matters in our solution. Two key characteristics of insightful criteria are that they are unambiguous and measurable. Unambiguous means that the criterion is explicitly defined. Consider ‘compensation’ as a criterion for deciding among job opportunities. Are you just considering base pay, or are you also including annual bonuses, health insurance, retirement benefits, disability insurance, and other ancillary benefits? If you do not explicitly define what comes under ‘compensation’, various people will have various interpretations, promoting disagreements.

Relatedly, you should also specify how each criterion will be measured. With financial compensation and other readily quantifiable criteria such as price, weight, time, distance and so on, this can be reasonably straightforward.

However, not all criteria lend themselves to this approach. Qualitative criteria, such as cultural fit, happiness, perceived risk and others, require you and your stakeholders to make judgment calls for how to score them.7 To be clear, there is nothing wrong with making judgment calls, but when presenting your conclusions, you need to be able to explain why alternatives scored high or low in a convincing manner to a potentially critical audience. In these settings, conclusions formed on weaker-quality evidence – say anecdotes, stories, or analogies – will be weaker than those based on ‘hard data’.

On a practical note, it is useful to make your criteria all vary in a consistent direction. For instance, where a higher score is consistently better. Such criteria are called benefit criteria. Imagine that what matters to us travelling from New York City to London is price, speed, comfort, and greenness (absence of emissions). The latter three – speed, comfort, and greenness – are benefit criteria: for each, an alternative with a higher score is more desirable than one with a low score. But price isn’t. With price, a higher price creates a worse alternative. There’s an easy fix, though: all we have to do is replace ‘price’ by ‘affordability’. Although seemingly trivial, making all criteria benefit criteria will save you headaches when calculating overall scores, interpreting results, and discussing viewpoints with stakeholders.

Finally, specify an appropriate range for each criterion. A good way to do this is to develop a 0–100 scale for each criterion. So, if we continue with our NYC-to-London example, an alternative scoring 0 in affordability would be the most expensive one. But don’t leave it as such a qualitative description. Instead, define what that means for you. Maybe it’s $1000, maybe it’s $100k, depending on your viewpoint. What matters is that you be as unambiguous as possible, using quantitative measures in lieu of qualitative ones wherever possible.

HOW TO IDENTIFY GOOD CRITERIA

If you’re still making fun of Albrecht for omitting to consider quietness when choosing his house, thinking that developing a good list of criteria can’t be that hard, be careful. In empirical studies, researchers at Duke and Georgia Tech have demonstrated that decision makers can omit nearly half of the criteria they would later consider relevant. Not only that, the participating decision makers perceived the omitted criteria as not only relevant, but almost as important as the criteria they had initially considered!8

So, identifying criteria might not be as simple as it looks; here are five ideas to help.9

• •Take several cracks at it, starting on your own. Bond and co-authors suggest that you start by yourself, without external help, to get your thinking on paper before it gets influenced by others’ opinions. Furthermore, in a series of experiments, they concluded that the best way to improve this general list was to go through the exercise several times (decision makers typically identify only 30% to 50% of their criteria in their first go).10
• •Use scenarios. You may start with listing all relevant and not so relevant concerns that come to mind (you can prune the list later). To make the list more lively, you can turn it into a wish list, a future perfect scenario of what would happen if everything went ideally with the problem-solving effort. Why would you be happy with the outcome? Then turn your reasons for happiness into specific criteria. To contrast this positive scenario, also perform a premortem: Assume that your project went wrong and ask yourself why.11 Maybe your project ended up being too expensive, maybe its outcome was low quality, maybe the implementation was too slow. The goal of these project-yourself-into-the-future techniques is to vividly imagine outcomes prior to the decision to gain further perspective.
• •Leverage frameworks. Another avenue is to look for frameworks that already provide pre-populated lists of criteria. Take for instance a ­business-unit manager who is considering where to expand internationally with a new product line. She has identified alternatives: Europe, Latin America or South-East Asia. As she is comparing the three markets, she thinks of their differences and similarities. She considers the levels of competition and how easy or difficult it is to get into the market. As she continues, she remembers that, while in business school, she had learned a framework by Michael Porter, called the five forces framework, that might provide a pre-configured list of criteria she could use.12 She looks at the list and realises: (1) level of competition, (2) power of buyers, (3) power of suppliers, (4) barriers to entry and (5) threat of substitutes are all relevant criteria that she needs to consider in her market comparison. Digging deeper, she notices that these five criteria cover all of the considerations she had been pondering – in other words, the framework is collectively exhaustive for what she is trying to achieve. Now, could she have developed a similar list by herself? Maybe, but it can be surprisingly difficult to cover all relevant aspects of a challenge just by relying on your own thinking and structuring. So juxtaposing it with other structures, such as those provided by frameworks, can be useful.
• •Contrast alternatives. Developing criteria in a vacuum might result in a set that misses key trade-offs. In fact, senior executives often confess that even though they go through the effort of developing a matrix, it does not capture the key aspects of their challenges. One main reason seems to be that they often develop their criteria without considering a concrete alternative. As a result, although the criteria look good on paper, they don’t connect to what the managers like and dislike. To avoid this trap, start your criteria search by considering two starkly different alternatives. That will help you identify what you value. First, you might just write down what you like and dislike about each alternative. Next, you can articulate more precisely the underlying criteria.
• •Enlist others. Finally, just as with alternative generation, you may want to ask for help. You can ask the stakeholders in the decision for their values, interest, and concerns.13 You may also ask people who have been in your position elsewhere. We are currently working with an organisation that is looking for an IT provider to help them drive their digitisation. Out of five potential IT providers, the organisation has to choose one. Yet, instead of just setting criteria within the top team, they have reached out to other organisations in Switzerland that are already a step ahead in their digitisation efforts, not to find out which IT provider were chosen, but above all, to understand how they made these decisions.

PRIORITISE YOUr CRITERIA

Finally, not all criteria are equally important, so you need to clarify their relative priority. Say, there is a trade-off to be made between customer satisfaction and cost. How much do you value one over the other? This is a complex enough task, but it gets even more difficult in group decision making, where each stakeholder has an opinion.

Over the past decades, decision analysis has yielded many ways to assign weights to criteria to match the decision makers’ preferences. Out of this debate, the community hasn’t converged on a single, widely accepted approach.18 To keep things simple, we propose to follow a simple direct-rating approach where you assign a weight from 1 (weakest) to 5 (strongest) to each criterion.

Weighing is usually an iterative process as your understanding of the problem evolves throughout the process. In particular, you might suffer from equalising bias, the tendency to allocate similar weights to all criteria. To counter this tendency, one debiasing technique consists of ranking the criteria first in numerical order and only then assigning them weights.19

VALIDATE THAT YOUR CRITERIA CAPTURE THE KEY AREAS OF CONCERN

Having come this far, use your list to evaluate alternatives, asking yourself if you would be comfortable living with the resulting decision. If not, you may have overlooked or misstated some criteria: Test whether your criteria would help you explain a prospective decision to someone else. If not, spend more time refining them: What’s unclear? What’s missing?

CHAPTER TAKEAWAYS

Summarise what matters in a set of criteria that is reasonably MECE and insightful.

For criteria, ‘insightful’ means that you’re balancing being collectively exhaustive and parsimonious; you’re giving more weight to the important criteria; you make all your criteria vary in a consistent direction; and you choose an appropriate range of performance for each criterion.

Identifying a good list of criteria can be surprisingly challenging. To help you, consider taking several cracks at it; using scenarios; leveraging frameworks; contrasting alternatives; and enlisting others.

CHAPTER 5 NOTES

1.   1Galef, J. (2021). The scout mindset: Why some people see things clearly and others don’t, Penguin.
2.   2Kahneman, D., D. Lovallo and O. Sibony (2019). ’A structured approach to strategic decisions.’ MIT Sloan Management Review Spring 2019.
3.   3There are many decision methods out there. There are many decision methods to help people choose when pursuing multiple objectives. Of those, the AHP for analytic hierarchy process (Saaty, T. L. (1990). ’How to make a decision: The analytic hierarchy process.’ European Journal of Operational Research 48(1): 9–26) is often cited as one of the most popular. For a review, see Mardani, A., A. Jusoh, K. Nor, Z. Khalifah, N. Zakwan and A. Valipour (2015). ‘Multiple criteria decision-making techniques and their applications: A review of the literature from 2000 to 2014.’ Economic Research-Ekonomska Istraživanja 28(1): 516–571.
4.   4Complex problems do not have objectively superior solutions. This is a consequence of their ill-definition. See, for instance, p. 280 of Hayes, J. R. (1989). The complete problem solver. New York, Routledge.
5.   5This aligns reasonably well with others’ lists. For instance, 117–118 and p. 121 of Keeney, R. L. (2007). Developing objectives and attributes. Advances in decision analysis: From foundations to applications. W. Edwards, R. F. Miles and D. von Winterfeldt, Cambridge University Press: 104–128. For other examples, see p. 50 of; p. 82 of Keeney, R. L. (1992). Value-focused thinking: A path to creative decision making. Cambridge, Massachusetts, Harvard University Press.
6.   6See, for instance, p. 328 of Edwards, W. (1977). ’How to use multiattribute utility measurement for social decision making.’ IEEE Transactions on Systems, Man, and Cybernetics 7(5): 326–340.
7.   7For a more in-depth description of how to do this, see, for instance, pp. 40–42 of Goodwin, P. and G. Wright (2014). Decision analysis for management judgment, John Wiley & Sons.
8.   8Bond, S. D., K. A. Carlson and R. L. Keeney (2008). ’Generating objectives: Can decision makers articulate what they want?’ Management Science 54(1): 56–70.
9.   9For more on this topic, see, for instance, pp. 110–113 of Keeney, R. L. (2007). Developing objectives and attributes. Advances in decision analysis: From foundations to applications. W. Edwards, R. F. Miles and D. von Winterfeldt, Cambridge University Press: 104–128.
10. 10Bond, S. D., K. A. Carlson and R. L. Keeney (2010). ’Improving the generation of decision objectives.’ Decision Analysis 7(3): 238–255.
11. 11Klein, G. (2007). ’Performing a project premortem.’ Harvard Business Review 85(9): 18–19. Soll, J. B., K. L. Milkman and J. W. Payne (2015). ­’Outsmart your own biases.’ Ibid. 93(5): 64–71.
12. 12Porter, M. E. (1979). ’How competitive forces shape strategy.’ Harvard Business Review.
13. 13See p. 106 of Keeney, R. L. (2007). Developing objectives and attributes. Advances in decision analysis: From foundations to applications. W. Edwards, R. F. Miles and D. von Winterfeldt, Cambridge University Press: 104–128.
14. 14See p. 98 of Hofstede, G. (2001). Culture’s consequences: Comparing values, behaviors, institutions, and organizations across nations, Sage Publications.
15. 15Helmreich, R. L., J. A. Wilhelm, J. R. Klinect and A. C. Merritt (2001). ­’Culture, error, and crew resource management.’, ibid.
16. 16See p. 16 of House, R. J., P. W. Dorfman, M. Javidan, P. J. Hanges and M. F. S. de Luque (2013). Strategic leadership across cultures: GLOBE study of CEO leadership behavior and effectiveness in 24 countries, Sage Publications.
17. 17Huang, X., E. Van De Vliert and G. Van der Vegt (2005). Breaking the silence culture: Stimulation of participation and employee opinion withholding cross-nationally.’ Management and Organization Review 1(3): 459–482.
18. 18For a review, see Riabacke, M., M. Danielson and L. Ekenberg (2012). ’State-of-the-art prescriptive criteria weight elicitation.’ Advances in Decision Sciences 2012. For a good primer on weight assignation, see pp. 44–47 of Goodwin, P. and G. Wright (2014). Decision analysis for ­management judgment, John Wiley & Sons, ibid.
19. 19Montibeller, G. and D. Von Winterfeldt (2015). ’Cognitive and motivational biases in decision and risk analysis.’ Risk Analysis 35(7): 1230–1251.