What Data Was Collected?

The first question you want to ask yourself before conducting any analysis is what are the variables and metrics that are available to you? Was the data collected the data you really need? Do you have the right variables to help you make the decisions? As discussed in Chapter 3, decision‐makers sometimes prefer to be precisely wrong rather than vaguely right. They may feel more comfortable looking at available data sets that are precise but less relevant to the problem, than less accurate data that has the key metrics they need to make the decision.

Consider the case study of a new chocolate chip cookie brand. Prior to the national launch of the product, the brand followed the common practice of introducing the product in a single representative test market. The test market was used to examine the product's performance and evaluate different pricing strategies, advertising tactics, and distribution. During the test, the company collected information on sales, market share, and advertising expenditure—both their own and that of competing brands—as well as survey data. Three months into the test of the product, the hard data looks great. The brand has backed the pilot launch with strong marketing, and sales of the product are above expectations and comparable to, or even better than, previous new product launches. The product has captured substantially more than its expected market share in the category.

The decision at hand is whether the brand should stop the test market and launch the product nationally, stop the test market and not launch the product, or continue the test market for a few more months. The third option, continuing the test market, risks an opportunity cost of delaying the national launch and sparking possible competitors' reactions. The data seems to support the first option—stop the test and launch the brand nationally—but at this point, you want to ask yourself what data was collected and what are your IWIKs™. Do the pieces of evidence collected help you answer your IWIKs? Working backward from the decision, do you have the inputs needed to call the shot and make an informed and confident decision about the launch?

Thinking about the drivers of the sales data and the forward‐looking nature of a decision to introduce the product nationally, you probably want to know the breakdown of the overall sales data between trial and repeat. What are customers' repeat purchase intentions? Indeed, when we looked at the trial and repeat purchase data for the new chocolate chip cookie brand, we see that the strong marketing behind the launch led to trial sales that exceeded expectations. However, both the actual repeat purchase data and the post‐purchase survey results—which asked customers if they planned to buy the product again—were well below expectations. The excellent product trial had helped to compensate for the low repeat sales figure, leading to rosy overall sales and market share figures. However, asking the right questions and going beyond the sales and market share data revealed a different and less favorable outlook for the brand. A manager who had not thought in advance about collecting the trial and repeat data would have missed this important insight and would probably have made the wrong decision in this case.

Many famous failed decisions are attributed to the wrong data being collected. One notable example is Coca‐Cola's decision in 1985 to discontinue the Coke formula—loved by consumers for almost 100 years—and introduce New Coke. The decision came in response to their major competitor, Pepsi, showing that in blind taste tests, Pepsi the drink has been called Pepsi since 1961 was preferred to Coke. Coca‐Cola management conducted large‐scale and expensive market research. Almost 200,000 prospective customers were blindfolded and instructed to taste different iterations of a sugary concoction, some of which had been manufactured by Coca‐Cola, and some by rivals. Coca‐Cola's extensive research led to the creation of a new formula that, in blindfold taste tests, was preferred to the original Coke formula and to Pepsi. Coca‐Cola then decided to replace the original Coke formula with a New Coke flavor. That decision met strong resistance, demonstrations, and, in some cases, even riots among Coca‐Cola's loyal consumers. That furious consumer backlash caught Coca‐Cola's management by surprise and eventually led them to revert back and reintroduce Classic Coke, keeping the loved original formula. Such dramatic U‐turn decisions by companies are rare, but when they occur, they're often damaging.

What went wrong with the New Coke decision‐making process? After all, Coca‐Cola invested millions of dollars and thousands of hours in marketing research prior to making the decision. Let's ask the question of what data was collected. The primary metric or key performance indicator (KPI) used to decide on the optimal cola flavor was a measure based on blindfolded consumer taste tests of various cola formulas. Why was this the wrong KPI to make that decision? The answer seems, in retrospect, simple: Consumers don't go blindfolded to buy their soft drinks in the supermarket. The question Coca‐Cola failed to ask, considering the brand and taste that has shaped American preferences for decades, was this: Would supermarket consumers, with their eyes wide open to labels, names, and packaging, prefer a new flavor of Coke over the original? The answer to that question was, apparently, no.

How can managers ask the right questions and make sure the right KPIs are being collected? Let's examine the two preceding examples: the test market for the chocolate chip cookie brand and the introduction of New Coke. What was missed in both examples were the fundamentals: understanding and exploring the drivers of consumer behavior. Had the leaders of Coca‐Cola conducted an effective IWIK session, as discussed in Chapter 2, or a careful backward approach from the decision to the information needed to make that decision, as discussed in Chapter 3, they would have been much more likely to collect the right information and make the right decision. These tools would have helped the decision‐makers identify the right metrics to describe the customer purchase decision, like trial and repeat purchases in the test market case, or the preference—even love—for a classic brand, which couldn't be measured by blindfolded testing.

Another red flag to look for when it comes to assessing data and major KPIs are averages. Because one can almost never report all the data points, it is common to report averages across different groups. However, averages can be misleading. Consider a survey in which customers reported their preference for pulp in the orange juice they drink on a –2 to +2 point scale. Having two segments in the population, one that likes pulpy orange juice and another that dislikes the pieces of fruit in their drink may lead to an average preference for pulp of around 0. Seeing only the average preference for pulpiness, the researcher may conclude that consumers do not care about pulp. This would mask the truth, as all consumers actually care about the level of pulp in their orange juice, but half of them truly like it and the other half dislike it.

Averages may not only mask the truth, but they may also lead to an entirely wrong conclusion. Imagine you have run an advertising experiment where some of the customers were targeted with an online display ad, and another randomly selected group was not. The campaign seems to have worked. As the last row in Table 4.1 shows, 8.3% of the customers who were shown the ad ended up buying the product and only 7.8% of the customers who were not shown the ad bought the product. Because customers were randomly either shown or were not shown the ad, the analyst concluded that the campaign was successful and recommended showing the ad to all customers. Another analyst who looked at the campaign results was interested in investigating how effective the campaign was for new versus repeated customers (rows 2 and 3 in Table 4.1). To her surprise, the analyst found that whether she looked at repeat customers or new customers, showing the ad led to a lower purchase rate than not showing the ad. But how could that be? The overall results are simply the sum of the two groups? How could each group (new and repeat customers) have a higher purchase rate when they do not see the ad, but overall, the campaign showed a positive effect for showing the ad? This situation is known as the Simpson's Paradox.¹ Looking across groups leads to the opposite result than looking at each of the groups separately.

What should we do? How do we know which slice the data we should look at? The analyst happened to slice the data by new versus repeat customers, but what would have happened if instead, she sliced the data by geography, age of the customer, the income of the customer, and customers who saw the ad on a Mac versus a PC? Would that lead to different results? As with the previous examples, it all comes down to what KPIs are relevant for the decision. The splits of the data that matter should be the split that you can and may want to act on.

Because many companies fail to understand the risk of looking at averages, such companies often create the average product that sits in the middle of consumers' preferences but may serve no one. Putting it in simple terms, if we were to serve customers the average tea that lies between the preferences of some customers for iced tea and the others for hot tea, we would have served all customers a lukewarm tea that none of our customers would want. Be aware of averages and make sure you don't serve your customers or colleagues lukewarm tea.

When considering data or analyses, managers need to be fierce interrogators. To begin with, ask yourself some key questions: Are these the KPIs I would expect to see? Are these the KPIs I need to know to be able to make the decision? What else do I wish I knew to be able to make the decision?

What Data Am I Not Seeing?

The data points you observe are almost never the entire data. There are almost always some aspects of the data that you are not seeing. Even the census data collected by the government, which, by design, is aimed at reaching every citizen, tends to miss people who have no permanent residence, like homeless people or students, or people who elect not to respond. There are two critical questions the fierce interrogator must ask: What data am I not seeing? And is the data I am missing similar to the data I do observe? It is often assumed that the unobserved data are similar to the observed data, but if this assumption is incorrect, it could lead to erroneous predictions and faulty decision‐making.

A famous example illustrating the issue of missing data dates back to World War II. U.S. Army engineers were examining the airplanes that came back from combat to assess where to add more armor for critical protection. As you can see in Figure 4.1, most of the hits and damage to the airplanes (the dots in the figure) were, as one would expect, in the large surface areas like the wings and the main body of the airplane. Based on this figure, the engineers decided to reinforce these high‐risk areas. Abraham Wald, who was a statistics professor at Columbia University at the time and was consulting to the Army, looked at the figure and recommended reinforcing the armor around the cockpit, the engines, and the gas tank surfaces. The army engineer asked whether Wald was looking at the same figures they looked at. In fact, Wald was thinking of the missing data. He made his observation realizing that the Army engineers hadn't accounted for what is known as survival bias—an error that arises when sampling is limited to people or things that have passed some sort of selection process. Tragically, the planes that were hit in the cockpit, the gas tank, or the engine, never made it back from combat.

In this example, the data observed (damage to airplanes that came back) was diametrically opposed to the unobserved data (damage to airplanes that did not make it back). Asking two questions—what data am I not observing? and are the observed data points similar to the unobserved ones?—helped Abraham Wald detect the problem with the observed data.

Two other famous cases of missing data involved incorrect predictions of many polls in the 2016 U.S. presidential election and the 2016 United Kingdom European Membership Referendum (better known as Brexit). Many people refused to answer these polls, and these people's opinions differed sharply from the opinions of those who did respond. Pollsters failed to adjust correctly for the opinions of nonrespondents. This is called nonresponse bias. It reflects a specific missing data problem that becomes a bias only if nonrespondents differ in a meaningful way from respondents. One possible solution to the missing data problem of nonresponse is to create one degree of separation between the respondent and the response. For example, in the 2016 U.S. presidential election, the few polls that did well in predicting the outcome were those that asked people to predict how their neighbor would vote. This approach helps adjust for nonresponse bias as well as biases arising from dishonest responses.

In the preceding three cases—the WWII airplanes' analysis, and the failures of the U.S. election and the Brexit polls—there was no intent on the part of the data provider to hide part of the data from the decision‐makers, and yet missing data led to erroneous predictions or conclusions. Now consider the situation common in business or public policy where data providers may be motivated to show only the part of the data that supports their agenda. Such cases might include deliberately withholding or obscuring data.

The missing data problems previously described are just a glimpse into the world of unobserved data errors. How does a fierce interrogator of data mitigate the risk of missing data? As always within the QI framework, the answer is to apply a set of questions: What are the data points I am not seeing? Does the data include nonresponse? Is the data I am not seeing similar to the data I am seeing? Who provided the data? Do they have a reason not to show me the entire data, and, if so, what are they likely to hide?

Can I Trust the Data and Analyses?

One of the most common pitfalls of data‐driven decision‐making is failing to see that the data you're basing decisions on is flawed. Assuming the data collected included the right KPIs, you should explore whether that data is accurate and reliable. It is equally important to understand the assumptions underlying the generation and analysis of the data.

There are many famous examples of wrong decisions based on inaccurate data. Consider, for example, Christopher Columbus crossing the Atlantic Ocean in 1492. Hoping to reach Asia from Portugal, Columbus based his idea of the journey's distance on the calculations of a medieval Persian geographer. Not only were those calculations erroneous, but Columbus also assumed the geographer was referring to Roman miles when, in fact, the measures were in Arabic miles. That mistake amounted to a difference of almost 5,592 miles (9,000 km) more than Columbus's estimate. Centuries later, the National Aeronautics and Space Administration (NASA) made a similar mistake with the Mars Climate Orbiter (MCO). Relying on calculations provided by Lockheed Martin Corporation, NASA assumed that the crucial acceleration figure was presented in the metric system. In fact, Lockheed Martin had used the English system of feet, inches, and pounds. This error led to the loss of the MCO, at a cost of $125 million.

Another example of the dangers of faulty data, and one that was more costly by far, is the financial crisis of 2008. The crisis has been attributed to bad and missing data about the risk inherent in subprime mortgages. In the case of the 2008 financial crisis, some of the faulty data may have arisen from ill intent on the part of those who were selling these loans who may have presented these using vanity metrics. Vanity metrics look impressive on the surface, but don't truly reflect the underlying situation and are not useful to understanding performance or informing decisions. Typical vanity metrics in business include figures like the number of social media likes or shares or sales due to promotions without accounting for the cost. Another common vanity metric is for a business with a decreasing trend of annual revenue to report the cumulative revenue, which is, by definition, always increasing.

So how do you identify such data reliability issues? The first thing you want to do when you receive data or analysis is to examine simple summary statistics like the average and the range of the values of each relevant variable in the data. This should help you quickly identify the more apparent problems in the data. If you find that the average age of the customers in the data is 150 years, or the average tenure of employees in the company is 45 years, you know these variables are faulty. Similarly, looking at the minimum or maximum values can help identify outliers. If you are looking at the survey responses to a 10‐point scale Net Promoter score survey question, and you find a minimum value of 1 and a maximum value of 34, you know there is a problem in the data. The maximum cannot be larger than 10 in a 10‐point scale.

Another way to quickly interrogate the data is to plot it in whatever way allows you to readily explore its reliability. Simple plots are a great way to reduce large or complex data to a visual. We cannot count the number of times we have identified issues with data simply by asking our analysts to report the average of each variable or to visually plotting the data. A simple plot can help you explore important patterns in the data, as in the case of Anscombe's Quartet (see “The Anscombe's Quartet” call‐out box).

Keep in mind that data can get distorted when it is imported from one software or one system to another, as in the preceding NASA example. Your analyst, being well versed in analytic methods and eager to jump straight into the analysis and deliver results, may be tempted to skip the step of looking at summary statistics of the data plotting it. Make sure they don't!

Assessing the Data—Key Learning

To examine the reliability of the data, the fierce interrogator asks a series of questions. We encourage you to use this list to guide your data interrogations.

1. What is the source of the data? Data and analyses rarely arrive at your desk at random. There are often intentions (good ones, but also possibly bad ones) behind how and why the data was collected and how and why the analyses are presented to you. Depending on the source and the intention, there could be possible agendas behind the data delivered to you. A good data interrogator asks these questions: Does the data provider have a reason not to show me the entire data? If so, what are they likely to hide? For example, is it the marketing team that provides evidence about the success of the advertising campaign? Understanding the source of data, and the intent or possible agenda, can inform me about possible issues in the data that you want to pay closer attention to.
2. Are the metrics provided the ones you expected to see? If not, why not? Are your data providers showing you the right KPIs? Are you being presented with vanity metrics that make the data provider look good?
3. How were the metrics calculated? Many metrics have no clear definition. For example, when a company reports having 10 million customers, you want to ask yourself how customers are defined. Are customers everyone who ever visited the company's website (even if they never bought anything), people who last bought from the company five years ago, or only active customers who purchased in the past year? Depending on the agenda behind the data, the data provider may choose different metrics. Make sure that you understand the metrics, particularly those that are critical to your decision‐making.
4. When and where was data collected? Are the time period, location, and context relevant to the decision at hand? Should we make decisions about mobile wallet adoption in 2022 in Hungary based on mobile adoption data in Austria in 2017? We happen to have accurate and reliable data from Austria, but no readily available good data from Hungary. Am I better off with accurate but less relevant and possibly outdated data from Austria or less precise but more current data from Hungary?
5. Are the comparisons being made to relevant and comparable alternatives? Almost every company can look good if compared to the right competitor. If comparisons are made, are the metrics comparable across alternatives? Different companies may measure the same KPI (e.g., the number of customers) in different ways.
6. What is missing? Are there other data points that may be relevant? Do you have this data over time so you can explore possible trends?
7. Is the data I am not seeing similar to the data I am seeing? What data didn't you capture? Who was left out? Could you have fallen prey to a nonresponse or a survival bias?
8. Are there outliers? Was there any data the provider could not explain (outliers) and therefore did not show? Is there a pattern in the outliers that may prove valuable?

Asking yourself these questions can help you scrutinize the data and analysis presented to you. Note that to be able to ask or answer these questions, you do not need to be a math whiz or a skilled data scientist, in fact, as the example in the Anscombe’s Quartet call‐out box demonstrates, quite the opposite. You simply need curiosity, critical thinking, and a good understanding of the context. You need to combine quantitative information with a great deal of intuition and business acumen. This is why the I in QI is a strong way to lead early in the process: good intuition is the key component of great data interrogation.