• We take a comprehensive look at UX metrics. No other books review so many different metrics. We provide details on collecting, analyzing, and presenting a diverse range of UX metrics.

• This book takes a practical approach. We assume you’re interested in applying UX metrics as part of your job. We don’t waste your time with unnecessary details. We want you to be able to use these metrics easily every day.

• We provide help in making the right decisions about UX metrics. One of the most difficult aspects of a UX professional’s job is deciding whether to collect metrics and, if so, which ones to use. We guide you through the decision process so that you find the right metrics for your situation.

• We provide many examples of how UX metrics have been applied within different organizations and how they have been used to answer specific research questions. We also provide in-depth case studies to help you determine how best to use the information revealed by the UX metrics.

• We present UX metrics that can be used with many different products or technologies. We take a broad view so that these metrics can be helpful throughout your career even as technology evolves and products change.

• Chapter 1 provides an overview of user experience and metrics. We define user experience, discuss the value of measuring the user experience, share some of the emerging trends, dispel some of the common myths about UX metrics, and introduce some of the newest concepts in UX measurement.

• Chapter 2 includes background information on UX data and some basic statistical concepts. We also provide a guide for performing common statistical procedures related to different UX methods.

• Chapter 3 focuses on planning a study involving metrics, including defining participant goals and study goals and choosing the right metrics for a wide variety of situations.

• Chapter 4 covers various types of performance metrics, including task success, time on task, errors, efficiency, and ease of learning. These metrics are grouped under an “umbrella” of performance because they measure different aspects of the user’s behavior.

• Chapter 5 looks at measuring usability issues. Usability issues can be quantified easily by measuring the frequency, severity, and type of issue. We also discuss some of the debates about appropriate sample sizes and how to capture usability issues reliably.

• Chapter 6 focuses on self-reported metrics, such as satisfaction, expectations, ease-of-use ratings, confidence, usefulness, and awareness. Self-reported metrics are based on what users share about their experiences, not what the UX professional measures about their actual behaviors.

• Chapter 7 is devoted to behavioral and physiological metrics. These metrics include eye tracking, emotional engagement, facial expressions, and various measures of stress. All of these metrics capture something about how the body behaves as a result of the experience of interacting with a user interface.

• Chapter 8 discusses how to combine different types of metrics and derive new metrics. Sometimes it’s helpful to get an overall assessment of the user experience of any product. This global assessment is achieved by combining different types of metrics into a single UX score, summarizing them in a UX scorecard, or comparing them to expert performance.

• Chapter 9 presents special topics that we believe are important but that don’t fit squarely into one of the five general categories. These include A/B testing on a live website, card-sorting data, accessibility data, and return on investment (ROI).

• Chapter 10 presents five case studies. Each case study reviews how different types of UX metrics were used, how data were collected and analyzed, and the results. These case studies were drawn from UX professionals in various types of organizations, including consulting, government, industry, and not-for-profit/education.

• Chapter 11 provides 10 steps to help you move forward in using metrics within your organization. We discuss how UX metrics can fit within different types of organizations, practical tips for making metrics work within your organization, and recipes for success.

1.1 What is User Experience

Before we try to measure user experience, we should know what it is and what it isn’t. While many UX professionals have their own ideas of what constitutes a “user experience,” we believe the user experience includes three main defining characteristics:

In the absence of a user doing something, we might just be measuring attitudes and preferences, such as in a political poll or survey about your favorite flavor of ice cream. There has to be behavior, or at least potential behavior, to be considered user experience. For example, we might show a screenshot of a website and ask participants what they would click if it were interactive.

You might also note that we never defined any characteristics of the product or system. We believe that any system or product can be evaluated from a user experience perspective, as long as there is some type of interface between the system or product and the user. We are hard-pressed to think of any examples of a product that don’t have some type of human interface. We think that’s a good thing, as it means that we can study almost any product or system from a UX perspective.

Some people distinguish between the terms usability and user experience. Usability is usually considered the ability of the user to use the thing to carry out a task successfully, whereas user experience takes a broader view, looking at the individual’s entire interaction with the thing, as well as the thoughts, feelings, and perceptions that result from that interaction.

In any casual conversation about usability, most people would agree that it’s good to have something that works well and isn’t confusing to use. On the other side of the coin, some companies may intentionally design products to be confusing or frustrating. Fortunately, this is a rare occurrence. For the purposes of this book, we will be somewhat idealistic and make the assumption that both users and designers want products to be easy to use, efficient, and engaging.

User experience can sometimes mean the difference between life and death. For example, the health industry is not immune to poor usability. Usability issues abound in medical devices, procedures, and even diagnostic tools. Jakob Nielsen (2005) cites one study that found 22 separate usability issues that contributed to patients receiving the wrong medicine. Even more troubling is that, on average, 98,000 Americans die every year due to medical error (Kohn et al., 2000). While there are no doubt many factors behind this, some speculate that usability and human factors are at least partially to blame.

In some very compelling research, Anthony Andre looked at the design of automatic external defribulators (AEDs)(2003). An AED is a device used to resuscitate an individual experiencing cardiac arrest. AEDs are found in many public spaces, such as shopping malls, airports, and sporting events. An AED is intended to be used by the general public with no background or experience in life-saving techniques such as CPR. The design of an AED is critical, as most individuals who are actually using an AED are experiencing it for the first time, under a tremendous amount of stress. An AED must have simple and clear instructions, and deliver them in a way that is time sensitive and also mitigates user errors. Andre’s research compared four different AED manufacturers. He was interested in how each of them performed in terms of users being able to deliver a shock successfully within a specified time limit. He was also interested in identifying specific usability issues that were impacting user performance with each of the machines.

In his 2003 study, he assigned 64 participants to one of four different machines. Participants were asked to enter a room and save a victim (a mannequin lying on the floor) with the AED they were assigned. The results he found were shocking (no pun intended!). While two machines performed as expected (0% errors from a sample of 16 participants for each machine), two other machines did not fare so well. For example, 25% of the participants who used one of the AEDs were not able to deliver a shock to the victim successfully. There were many reasons for this outcome. For example, participants were confused by the instructions on how to remove the packaging for the pads that adhere to the bare chest. Also, the instructions on where to place the electrodes were somewhat confusing.

After Andre shared his research findings with his client, they agreed to address these issues as part of a product redesign effort.

Similar situations can arise on a regular basis in the workplace or in the home. Just think of the written instructions for such actions as lighting the pilot light on a furnace, installing a new lighting fixture, or trying to figure out a tax form. An instruction that’s misunderstood or misread can easily result in property damage, personal injury, or even death. User experience plays a much wider role in our lives than most people realize. It’s not just about using the latest technology. User experience impacts everyone, every day. It cuts across cultures, age, gender, and economic class. It also makes for some very funny stories!

Saving lives is, of course, not the only motivation for a good user experience. Championing user experience in a business setting is often geared toward increasing revenues and/or decreasing costs. Stories abound of companies that lost money because of the poor user experience of a new product. Other companies have made ease of use a key differentiator as part of their brand message.

The Bentley University Design and Usability Center had the opportunity to work with a not-for-profit organization on the redesign of their charitable-giving website. They were concerned that visitors to their website would have difficulty finding and making donations to the charitable foundation. Specifically, they were interested in increasing the number of recurring donations, as it was an excellent way to build a more continuous relationship with the donor. Our research included a comprehensive usability evaluation with current and potential donors. We learned a great deal about how to not only improve navigation, but simplify the donation form and highlight the benefits of recurring donations. Soon after the launch of the new website, we learned that the redesign effort was a success. Overall donations had increased by 50%, and recurring donations increased from 2, up to 19 (a 6,715% increase!). This was a true usability success story, and one that also benefits a great cause.

User experience takes on an ever-increasing role in our lives as products become more complex. As technologies evolve and mature, they tend to be used by an increasingly diverse set of users. But this kind of increasing complexity and evolution of technology doesn’t necessarily mean that the technologies are becoming easier to use. In fact, just the opposite is likely to happen unless we pay close attention to the user experience. As the complexity of technology grows, we believe that user experience must be given more attention and importance, and UX metrics will become a critical part of the development process to provide complex technology that’s efficient, easy to use, and engaging.

1.2 What are User Experience Metrics?

A metric is a way of measuring or evaluating a particular phenomenon or thing. We can say something is longer, taller, or faster because we are able to measure or quantify some attribute of it, such as distance, height, or speed. The process requires agreement on how to measure these things, as well as a consistent and reliable way of doing it. An inch is the same length regardless of who is measuring it, and a second lasts for the same amount of time no matter what the time-keeping device is. Standards for such measures are defined by a society as a whole and are based on standard definitions of each measure.

Metrics exist in many areas of our lives. We’re familiar with many metrics, such as time, distance, weight, height, speed, temperature, and volume. Every industry, activity, and culture has its own set of metrics. For example, the auto industry is interested in the horsepower of a car, its gas mileage, and the cost of materials. The computer industry is concerned with processor speed, memory size, and power requirements. At home, we’re interested in similar measurements: how our weight changes when we step on the bathroom scale, where to set our thermostat in the evening, and how to interpret our water bill every month.

The user experience field is no different. We have a set of metrics specific to our profession: task success, user satisfaction, and errors, among others. This book gathers all the UX metrics in one place and explains how to use these metrics to provide maximum benefit to you and your organization.

So what is a UX metric and how does it compare to other types of metrics? Like all other metrics, UX metrics are based on a reliable system of measurement: Using the same set of measurements each time something is measured should result in comparable outcomes. All UX metrics must be observable in some way, either directly or indirectly. This observation might be simply noting that a task was completed successfully or noting the time required to complete the task. All UX metrics must be quantifiable—they have to be turned into a number or counted in some way. All UX metrics also require that the thing being measured represents some aspect of the user experience, presented in a numeric format. For example, a UX metric might reveal that 90% of the users are able to complete a set of tasks in less than 1 minute or 50% of users failed to notice a key element on the interface.

What makes a UX metric different from other metrics? UX metrics reveal something about the user experience—about the personal experience of the human being using a product or system. A UX metric reveals something about the interaction between the user and the product: some aspect of effectiveness (being able to complete a task), efficiency (the amount of effort required to complete the task), or satisfaction (the degree to which the user was happy with his or her experience while performing the task).

Another difference between UX metrics and other metrics is that they measure something about people and their behavior or attitudes. Because people are amazingly diverse and adaptable, we sometimes encounter challenges in our UX metrics. For this reason, we discuss confidence intervals with most of the UX metrics discussed in order to reflect the variability in the data. We will also discuss what metrics we consider relevant (and less relevant) in a UX context.

Certain things are not considered UX metrics, such as overall preferences and attitudes not tied to an actual experience of using something. Think of some standard metrics such as the Presidential Approval Ratings, the Consumer Price Index, or the frequency of purchasing specific products. Although these metrics are all quantifiable and may reflect some type of behavior, they are not based on actually using something in order to reflect the variability in the data.

UX metrics are not an end unto themselves; rather, they are a means to help you reach an informed decision. UX metrics provide answers to questions that are critical to your organization and that can’t be answered by other means. For example, UX metrics can answer these critical questions:

1.3 The Value of UX Metrics

We think UX metrics are pretty amazing. Measuring the user experience offers so much more than just simple observation. Metrics add structure to the design and evaluation process, give insight into the findings, and provide information to the decision makers. Without the insight provided by metrics, important business decisions may be made based on incorrect assumptions, “gut feelings,” or hunches. As a result, some of these decisions are not the best ones.

During a typical usability evaluation, it’s fairly easy to spot some of the more obvious usability issues. But it’s much harder to estimate the size or magnitude of the issues. For example, if all eight participants in a study have the same exact problem, you can be quite certain it is a common problem. But what if only two or three of the eight participants encounter the problem? What does that mean for the larger population of users? UX metrics offer a way to estimate the number of users likely to experience this problem. Knowing the magnitude of the problem could mean the difference between delaying a major product launch and simply adding an additional item to the bug list with a low priority. Without UX metrics, the magnitude of the problem is just a guess.

User experience metrics show whether you’re actually improving the user experience from one product to the next. An astute manager will want to know as close to certain as possible that the new product will actually be better than the current product. UX metrics are the only way to really know if the desired improvements have been realized. By measuring and comparing the current with a new, “improved” product and evaluating the potential improvement, you create a win–win situation. There are three possible outcomes:

User experience metrics are a key ingredient in calculating a ROI. As part of a business plan, you may be asked to determine how much money is saved or how revenue increases as a result of a new product design. Without UX metrics, this task is impossible. With UX metrics, you might determine that a simple change in a data input field on an internal website could reduce data entry errors by 75%, reduce the time required to complete the customer service task, increase the number of transactions processed each day, reduce the backlog in customer orders, cut the delay in customer shipments, and increase both customer satisfaction and customer orders, resulting in an overall rise in revenue for the company.

User experience metrics can help reveal patterns that are difficult or even impossible to see. Evaluating a product with a very small sample size (without collecting any metrics) usually reveals the most obvious problems. However, many more subtle problems require the power of metrics. For example, sometimes it’s difficult to see small inefficiencies, such as the need to reenter user data whenever a transaction displays a new screen. Users may be able to complete their tasks—and maybe even say they like it—but many small inefficiencies can eventually build up to impact the user experience and slow down the process. UX metrics help you gain new insights and lead toward a better understanding of user behavior.

1.4 Metrics for Everyone

We’ve been teaching a class on UX metrics, in one form or another, for almost a decade. During this time, we have met many UX and non-UX professionals who have little-to-no background in statistics, and even a few who were terrified of anything that looks like a number. Despite this, we have continually been impressed and inspired by how these folks are able to learn the basics on how to collect, analyze, and present UX metrics quickly and easily. UX metrics are a very powerful tool, but also easily accessible to almost anyone. The key is simply to try, and learn from your mistakes. The more metrics you collect and analyze, the better you will get! In fact, we even see some individuals who use this book simply as a guide to what types of UX metrics make the most sense for their organization or project and then go off and ask someone else to actually do the dirty work and collect/analyze the data. So, even if you don’t want to get your hands dirty, there isn’t an excuse for incorporating UX metrics into your work.

We’ve written this book to be easy and approachable to the broadest possible audience. In fact, we probably favor simplification rather than a deep dive into heavy statistical analysis. We feel this will help attract as many UX and non-UX people as possible. Of course, we strongly encourage everyone to go beyond this book by creating new metrics tailored to your organization, product, or research practice.

1.5 New Technologies in UX Metrics

Earlier we stated that UX metrics apply to a vast array of products, designs, and technologies. In fact, even with new tchnologies emerging every day, UX metrics still remain highly relevant. However, what does change (and quite rapidly) are the technologies themselves that better allow us to collect and analyze UX data. Throughout the book you will get a sense of some of the newest technologies that might make your job a little easier, and certainly more interesting. We wanted just to highlight a few of the latest technologies that have emerged in the last few years.

There are some exciting new advances in the world of eye tracking. For decades, eye tracking was restricted to the lab. This is no longer the case. Within the last couple of years, two major vendors in eye tracking (Tobii and SMI) have released goggles that can be used to track eye movements in the field. So, as your participant is walking down the aisle at the supermarket, you can gather data on what he/she is looking at and for how long. Of course, it is a little tricky when different objects occur in approximately the same location and at different depths. But, no doubt they are improving these goggles with each new release.

Eye tracking is even moving beyond hardware. For example, EyeTrackShop has developed technology that collects eye movement data through the participant’s webcam. So, no longer are you restricted to being in the same location as your participants, now you can literally collect eye-tracking data with anyone in the world, assuming they have an Internet connection and a webcam. This is a very exciting development, and it is certainly going to open up the market for eye-tracking data to many UX professionals who did not have, or could not afford the hardware.

Another exciting new technology is in the area of affective computing. For decades, UX professionals have gained insight into a user’s emotional state by listening to and observing the participant, and of course asking all the right questions. These qualitative data have been, and will always be, extremly valuable. However, advances in affective computing have added a new dimension to measuring emotional engagement. Companies such as Affectiva combine data from sensors that measure skin conductance, along with facial recognition software that anayzes different facial expressions. Together, these two pieces of data tell the researcher something about not only the level of arousal, but the valence (whether it is a positive or negative emotion).

There are a host of new unmoderated usability testing tools that make data collection very easy and affordable. Some tools such as UserZoom and Loop11 are powerful and affordable for collecting a lot of usability data very efficiently. Other tools such as Usabilla and Userlytics do a very nice job of integrating both qualitative and quantitative data for a reasonable price. Other tools, such as UsabilityTesting.com, allow you to essentially run qualitative-based, self-guided usabilty studies very easily and quickly. And, of course, there are some very specialized tools that help track clicks or mouse movements. It is very exciting that there are so many new technologies that the UX researcher can add to his/her suite of tools.

Analyzing open-ended responses has always been very luborious and inprecise. It is all too common for researchers to disregard verbatim comments or just randonly select a small sample for quotes. In the last few years verbatim analysis software has improved greatly to the point that researchers now have the ability to analyze open-ended responses.

1.6 Ten Myths about UX Metrics

There are many common myths about UX metrics. Some of these myths may come from of a lack of experience with using metrics. Perhaps these myths arose from a negative experience (such as someone from marketing screaming about your sample size) or even other UX professionals complaining about the hassles and costs associated with using metrics. Ultimately the source of these myths doesn’t matter. What matters is to separate fact from fiction. We’ve listed 10 of the most common myths surrounding UX metrics and a few examples that dispel these myths.