12

EXPERIMENT
AND ITERATE

WE’VE ALL HEARD THE saying “No one ever got fired for buying IBM.” This expression is indicative of a troubling mindset in business: Safe decisions mean job security. The problem is that safe decisions are not necessarily conducive to innovation. Luckily, more and more businesses are encouraging employees to experiment and embrace smart risks.

For example, in 2014 PepsiCo—the parent company for over a dozen well-known chip brands—decided to launch a high-stakes experiment with its Ruffles brand, opting to spend nearly 100 percent of its ad budget on digital.¹ In 2013, less than 30 percent of the brand’s ad spending had been on digital, with most spending going toward TV ads. But PepsiCo realized that its spending, which was based on traditional advertising wisdom, didn’t match modern consumer behavior, especially for the company’s target demographic. After seeing impressive year-over-year sales growth, the Ruffles experiment helped give PepsiCo the confidence it needed to shift marketing spending away from TV into social and digital in 2015 across its brands.²

IN THIS CHAPTER, YOU WILL LEARN:

How to use experiments to reduce risk and address uncertainties.

What techniques can be used to assess likely consumer interest in a new offering.

How customer insights can be worked into the design process.

USING EXPERIMENTS TO RESOLVE UNCERTAINTIES

Experiments sound risky. By their very definition, you don’t know what the result will be. Dan Ariely, a renowned professor of behavioral economics at Duke University, explains that businesses often avoid experiments for a very simple reason. As he describes the issue, “We tend to value answers over questions because answers allow us to take action, while questions mean that we need to keep thinking. Never mind that asking good questions and gathering evidence usually guides us to better answers.”³ Used effectively, experiments represent a low-cost way of filling in details related to key unknowns. Will you be able to overcome key technical challenges? Is the idea feasible at scale?

Just because you’re using experiments to explore big questions, it doesn’t mean that the experiments themselves need to be costly or complicated. For instance, when McDonald’s considered adding a shrimp salad to its menu a few years back, a simple thought experiment around scale quickly put an end to the idea—buying shrimp for thousands of its restaurants would have had a significant impact on the overall shrimp supply, driving up the cost of shrimp and making the economics of a shrimp salad infeasible.⁴

Good experiments aren’t about grandeur. They’re about timing and reliability. With respect to timing, we mean that experiments should test the biggest risks early on. Although these risks may not be the most interesting or the easiest to test, don’t get caught in the weeds before you’ve tested the risks that are outcome determinative.

As to reliability, we mean that the experiment needs to actually test what you think you’re testing (see Figure 12-1). While many of us may have thought that high school was the last time we would need to think about the scientific method, its lessons are indispensable. Your experiment should be structured to address an articulable hypothesis, and there should be strict controls to ensure that only one variable is being tested at a time. Look to the experience of executives at the department store Kohl’s in 2013, when they were struggling to decrease operating costs. While there was a proposal to open stores an hour later, executives were split on whether this would cause a significant drop in sales.⁵ The team designed an experiment involving 100 of the company’s stores to test the prediction that opening an hour later Monday through Saturday would not result in a meaningful decline in sales. The test revealed that the impact on sales would be insignificant.

ELEMENTS OF A RELIABLE BUSINESS EXPERIMENT

Figure 12-1

More important than the results of the test, though, was the process. Kohl’s made a clear and precise prediction that identified specific dependent and independent variables (as opposed to something like “Changing store hours will not affect our financial performance”). The company also set a meaningful sample size and chose a metric (sales figures) that could be easily compared to on-hand quantitative data. In doing so, Kohl’s was able to quickly and inexpensively find an easy way to substantially decrease its operating costs.

GAUGING INTEREST IN A CONCEPT

One of the biggest uncertainties surrounding any new offering is whether customers will even consider buying it. One of the first—and easiest—experiments to consider is simply asking. Especially in the B2B space, there’s little harm in calling a few existing customers to get a little feedback on your concept, including whether they find the idea appealing, how it might be improved, who would need to sign off on the purchase, and where it might fall in the organization’s existing list of budget priorities. Over time—as we’ll explore—you’ll want to conduct concept tests that can better estimate your penetration rate in the market, but a few early phone calls can go a long way in gauging interest and shaping the design of your new offering.

Regardless of how you conduct your concept tests, it’s important to have a story—something that customers can readily understand. Food and beverage executive Christine Dahm explained to us how one industry powerhouse was exceptionally good at coming up with new product ideas, yet struggled to get reliable consumer feedback prior to launch.

When we tested the product ideas with consumers, we couldn’t re-create the emotional states that the products were responding to. Our tests ended up getting rational responses rather than the responses consumers would have given if they were actually living the experience. We partnered with a TV network to write concepts, and the difference was obvious. They’re storytellers, so naturally they were very good at turning our concepts into stories that resonated with the consumer.

As Dahm suggests, it’s important that your concept tests help re-create the context in which a decision to purchase or use a product would be made. If you ask consumers about the most important parts of a car, you might well hear from many people that the brakes are most important. If you then ask whether they would be more likely to buy a car with better brakes, the answer would likely be a resounding yes. But when people actually go to buy a car, almost no one is actually making decisions based on brakes. In order for your concept tests to be meaningful, it’s essential to create the story that puts customers “in the moment.”

This continues to be true as you conduct more rigorous concept tests. As you try to get a better idea of how widely your new offering might be adopted, basic website development tools and unique link generators (such as Goo.gl) can provide an easy and inexpensive way to quantify interest in your concept. Once you’ve set up a basic web page that tells the story of your new offering, you can take to the streets to hand out cards or brochures with a link to the site, or you can email a list of potential customers and encourage them to learn more by visiting your site. You can track your overall response rate, and you can use unique links to the site to see whether one campaign had a better response rate than another. Beyond seeing who’s interested in learning more, your site can use a buy-now button to see how many customers are willing to actually make a purchase, even if (at this stage) the button only takes consumers to a page thanking them for their interest and giving them a chance to learn more when your offering is actually available.

In conducting these broader concept tests, it’s still important to pay attention to your experimental design. Remember to think about the number of variables you’re changing, the biases that may be in play, and the sample size for your test.

BRINGING CUSTOMERS INTO THE DESIGN PROCESS

The final way experiments should prove useful is in gathering customer feedback. When you look at a new product, the version you see on the shelf is often far removed from the earliest prototypes. As a point of reference, it typically takes Nestlé 100 recipes to arrive at a single new product.⁶ Similarly, one of the earliest iPhone prototypes had a 5- × 7-inch screen and was roughly 2 inches thick.⁷

Getting something tangible in front of customers early on in the process can be a great way to gather feedback before you’ve spent too much money going down a bad path. Early prototypes can be rough, unfinished, and very different from the final product. The point is to understand what the customer likes or dislikes and to capture the questions that are raised (see Figure 12-2). With this feedback, you might make slight refinements to your design, or you may end up going in a completely different direction.

Beyond giving customers something to react to, we often find it helpful to let them help “build” the new offering. In focus groups, for instance, we may have participants design packaging or a marketing poster for a concept that we’ve come up with during the session. They’ll be asked to pitch their finished designs back to us, which gives us a chance to see what features and messaging they focus on, as well as what associations they may make with existing offerings. Similarly, we’ll sometimes give participants a set amount of fake money to spend on product features and offer them a limited set of features to buy. The features will have different price points that reflect the relative costs or difficulty associated with actually building in those features. This type of data helps us determine whether the product attributes that we plan on investing in are valuable enough to the consumer to justify the expense. It makes things more genuine than simply asking them what matters, when they may speak rationally but unrealistically.

FEEDBACK GRID

Figure 12-2

THIS CHAPTER IN PRACTICE—CALLING CUSTOMERS

We recently worked with a global B2B company that was looking for new services to offer its customers. We talked to experts, engaged in primary research with potential customers, and collaborated with internal team members to come up with a list of ideas for new offerings. After whittling down the list with senior executives—getting rid of those that were too far afield or too costly to implement—we still had a short list that needed to be prioritized. Our first step? Calling some customers.

The information we got from our calls was illuminating. Across all of our calls, every customer we talked to prioritized the same concept in the short term, and every customer indicated that they just weren’t ready for one of our more sophisticated concepts. Ultimately, we used this information to design a broader suite of offerings that customers could grow into over time. We were also able to get useful estimates on how we could price the new offerings, how long the decision-making process would take, and where the company’s brand was an asset or a hindrance in the sales process. The value we got from our calls far exceeded the value of the hours we spent on the phone.

CHAPTER SUMMARY

Companies—and the individuals running them—tend to be somewhat risk averse. People value answers over questions, so they shy away from experiments, which will necessarily have uncertain results. But the quality of information you can get from targeted experiments can be unparalleled. By running fast, inexpensive experiments that answer key questions, companies can make better informed decisions, substantially reducing the amounts of time and money that are typically wasted on failed innovation efforts. In particular, customer-facing experiments can provide invaluable insights into how customers make decisions, what they’re looking for in a new offering, and whether they’re likely to consider buying whatever it is you’re trying to sell.

Experiments can be an inexpensive way to resolve major unknowns before a project is too far along. Experiments should follow a careful design to ensure that the results are both valid and useful.

Concept tests can be a useful way to determine whether customers are interested in your new offering, but each concept needs a story that helps re-create the experience in which a customer would actually be making a decision to purchase or use the product.

Customers can help shape the design of your final product by reacting to prototypes or participating in activities designed to highlight what’s really important to them. Be sure to capture what customers like and dislike, as well as the questions they raise and any new ideas they may propose.