RESEARCH METHOD
01 A/B Testing
Use A/B testing to compare two versions of the same design to see which one performs statistically better against a predetermined goal.1
A/B testing is an optimization technique that allows you to compare two different versions of a design to see which one gets you closer to a business objective.2 The tests are run by randomly assigning different people down two paths—the “A” test and the “B” test—until a statistically relevant sample size is reached. At the end of the test, you will be able to determine which design gets you closer to your goals.
Take, for instance, the challenge of increasing the number of people who sign up for a free trial of your online service. There could be many explanations why people aren’t registering: Is the sign-up form too long? Are people worried about their privacy and what you will do with their data? Do they want to know about pricing information before they register? You can find out the answer to each of these questions by making small modifications to the interface, and then run an A/B test to see which version prompts more people to register. For instance, given the scenario above, you can design and run several tests that compare:
• different treatments of the page microcopy—the text that guides and reassures the user—regarding the terms of the service (tone, length, font size);
• the form elements (how many, layout, which are required); and
• different treatments of the button or call to action (page placement, size, color, labeling).
Even though there is a benefit to being able to measure which design generates better results, A/B testing won’t help you understand why the design was preferred over the alternate. A/B testing is not a replacement for qualitative methods that can assess your customers’ desires, attitudes, and needs, nor can it uncover larger problems like whether customers feel that they can trust your site or that it is credible.3 To that end, A/B testing should always supplement qualitative methods that help you gain a deeper understanding of what really motivates your customers and what they really want.
1. A/B tests are adapted from the classic direct mail practice in which two different versions of the same mailing are sent out to different people in order to see which one gets the better response rates.
2. Nielsen, Jakob. “Putting A/B Testing in Its Place,” 2005, http://www.useit.com
3. Kahavi, Ron, Randal M. Henne, and Dan Sommerfield. “Practical Guide to Controlled Experiments on the Web: Listen to Your Customer Not to the HiPPO.” Proceedings of the 13th ACM SIGKDD, 2007.