8.1.1 Combining Metrics Based on Target Goals

Perhaps the easiest way to combine different metrics is to compare each data point to a target goal and represent one single metric based on the percentage of users who achieved a combined set of goals. For example, assume that the goal is for users to complete at least 80% of their tasks successfully in no more than 70 seconds each on the average. Given that goal, consider the data in Table 8.1, which shows task completion rate and average time per task for each of eight participants in a usability test.

Table 8.1 shows some interesting results. The average values for task completion (82%) and task time (67 seconds) would seem to indicate that the goals for this test were met. Even if you look at the number of users who met the task completion goal (six participants, or 75%) or the task time goal (five participants, or 62%), you would still find the results reasonably encouraging. However, the most appropriate way to look at the results is to see if each individual participant met the stated goal (i.e., the combination of completing at least 80% of the tasks in no more than 70 seconds each). It turns out, as shown in the last column of Table 8.1, that only three, or 38%, of the participants actually met the goal. This demonstrates the importance of looking at individual participant data rather than just looking at averages. This can be particularly true when dealing with relatively small numbers of participants.

This method of combining metrics based on target goals can be used with any set of metrics. The only real decision is what target goals to use. Target goals can be based on business goals and/or comparison to ideal performance. The math is easy (each person just gets a 1 or 0), and the interpretation is easy to explain (the percentage of users who had an experience that met the stated goal during the test).

8.1.2 Combining Metrics Based on Percentages

Although we’re well aware that we should have measurable target goals for our usability tests, in practice many of us don’t have them. So what can you do to combine different metrics when you don’t have target goals? One simple technique for combining scores on different scales is to convert each score to a percentage and then average them. For example, consider the data in Table 8.2, which shows results of a usability test with 10 participants.

One way to get an overall sense of the results from this study is to first convert each of these metrics to a percentage. In the case of the number of tasks completed and the subjective rating, it’s easy because we know the maximum (“best”) possible value for each of those scores: there were 15 tasks, and the maximum possible subjective rating on the scale was 4. So we just divide the score obtained for each participant by the corresponding maximum to get the percentage.

In the case of time data, it’s a little trickier, as there’s not a predefined “best” or “worst” time—the ends of the scale are not known beforehand. One way of handling this would be to have several experts do the task and treat the average of their times as the “best” time. Another way is to treat the fastest time obtained from the participants in the study as the “best” (25 seconds, in this example), the slowest time as the “worst” (70 seconds, in this example), and then express other times in relation to those. Specifically, you divide the difference between the longest time and the observed time by the difference between longest and shortest times. This way, the shortest time becomes 100% and the longest becomes 0%. Using that method of transforming the data, you get the percentages shown in Table 8.3.

Table 8.3 also shows the average of these percentages for each of the participants. If any one participant had completed all the tasks successfully in the shortest average time and had given the product a perfect score on the subjective rating scales, that person’s average would have been 100%. However, if any one participant had failed to complete any of the tasks, had taken the longest time per task, and had given the product the lowest possible score on the subjective rating scales, that person’s average would have been 0%. Of course, rarely do you see either of those extremes. Like the sample data in Table 8.3, most participants fall between those two extremes. In this case, averages range from a low of 28% (Participant 4) to a high of 85% (Participant 9), with an overall average of 58%.

So if you had to give an “overall score” to the product whose test results are shown in Tables 8.2 and 8.3, you could say it got 58% overall. Most people wouldn’t be too happy with 58%. Many years of grades from school have probably conditioned most of us to think of a percentage that low as a “failing grade.” But you should also consider how accurate that percentage is. Because it’s an average based on individual scores from 10 different participants, you can construct a confidence interval for that average, as explained in Chapter 2. The 90% confidence interval in this case is ±11%, meaning that the confidence interval extends from 47 to 69%. Running more participants would probably give you a more accurate estimate of this value, whereas running fewer would probably have made it less accurate.

One thing to be aware of is that when we averaged the three percentages together (from task completion data, task time data, and subjective ratings), we gave equal weight to each of those measures. In many cases, that’s a perfectly reasonable thing to do, but sometimes the business goals of the product may indicate a different weighting. In this example, we’re combining two performance measures (task completion and task time) with one self-reported measure (rating). By giving equal weight to each, we’re actually giving twice as much weight to performance as to the self-reported measure. That can be adjusted by using weights in calculating the averages, as shown in Table 8.4.

In Table 8.4, the subjective rating is given a weight of 2, and each of the two performance measures is given a weight of 1. The net effect is that the subjective rating gets as much weight in the calculation of the average as the two performance measures together. The result is that these weighted averages for each participant tend to be closer to the subjective ratings than the equal-weight averages in Table 8.3. The exact weights you use for any given product should be determined by the business goals for the product. For example, if you’re testing a website for use by the general public, and the users have many other competitors’ websites to choose from, you might want to give more weight to self-reported measures because you probably care more about the users’ perception of the product than anything else.

However, if you’re dealing with an application where speed and accuracy are more important, such as a stock-trading application, you would probably want to give more weight to performance measures. You can use any weights that are appropriate for your situation, but remember to divide by the sum of those weights in calculating the weighted average.

These basic principles apply to transforming any set of metrics from a usability test. For example, consider the data in Table 8.5, which includes number of tasks completed successfully (out of 10), number of web page visits, an overall satisfaction rating, and an overall usefulness rating.

Calculating percentages from these scores is very similar to the previous example. The number of tasks completed is divided by 10, and the two subjective ratings are each divided by 6 (the maximum rating). The other metric, number of web page visits, is somewhat analogous to the time metric in the previous example. But in the case of web page visits, it is usually possible to calculate the minimum number of page visits that would be required to accomplish the tasks. In this example, it was 20. You can then transform the number of page visits by dividing 20 (the fewest possible) by the actual number of page visits. The closer the number of page visits is to 20, the closer the percentage will be to 100%. Table 8.5 shows original values, percentages, and then equal-weight averages. In this case, note that equal weighting (normal average) results in the same weight being given to performance data (task completion and page visits) and self-reported data (the two ratings).

To look at transforming another set of metrics, consider the data in Table 8.6. In this case, the number of errors is listed, which would include specific errors the users made, such as data-entry errors. Obviously, it is possible (and desirable) for a user to make no errors, so the minimum possible is 0. But there’s usually no predefined maximum number of errors that a user could make. In a case like this, the best way to transform the data is to divide the number of errors obtained by the maximum number of errors and then subtract from 1. In this example, the maximum is 5, the number of errors made by participant 4. This is how the error percentages in Table 8.6 were obtained. If any user had no errors (optimum), their percentage would be 100%. The percentage for the user(s) with the highest number of errors would be 0%. Note that in calculating any of these percentages, we always want higher percentages to be better—to reflect better usability. So, in the case of errors, it makes more sense to think of the resulting percentage as an “accuracy” measure.

When transforming any usability metric to a percentage, the general rule is to first determine the minimum and maximum values that the metric can possibly have. In many cases this is easy; they are predefined by the conditions of the usability test. Here are the various cases you might encounter.

8.1.3 Combining Metrics Based on Z Scores

Another technique for transforming scores on different scales so that they can be combined is using z scores. (See, for example, Martin & Bateson, 1993, p. 124.) These are based on the normal distribution and indicate how many units above or below the mean of the distribution any given value is. When you transform a set of scores to their corresponding z scores, the resulting distribution by definition has a mean of 0 and standard deviation of 1. This is the formula for transforming any raw score to its corresponding z score:

where x is the score to be transformed, μ is the mean of the distribution of those scores, and σ is the standard deviation of the distribution of those scores.

This transformation can also be done using the “=STANDARDIZE” function in Excel. Data in Table 8.2 could also be transformed using z scores, as shown in Table 8.7.

The bottom two rows of Table 8.7 show the mean and standard deviation for each set of z scores, which should always be 0 and 1, respectively. Note that in using z scores, we didn’t have to make any assumptions about the maximum or minimum values that any of the scores could have. In essence, we let each set of scores define its own distribution and rescale them so those distributions would each have a mean of 0 and standard deviation of 1. In this way, when they are averaged together, each of the z scores makes an equal contribution to the average z score. Note that when averaging the z scores together, each of the scales must be going the same direction—in other words, higher values should always be better. In the case of time data, the opposite is almost always true. Since z scores have a mean of 0, this is easy to correct simply by multiplying the z score by (–1) to reverse its scale.

If you compare the z-score averages in Table 8.7 to the percentage averages in Table 8.3, you will find that the ordering of the participants based on those averages is nearly the same: Both techniques yield the same top three participants (9, 5, and 3) and the same bottom three participants (4, 8, and 1).

One disadvantage of using z scores is that you can’t think of the overall average of the z scores as some type of overall usability score, as by definition that overall average will be 0. So when would you want to use z scores? They mainly are useful when you want to compare one set of data to another, such as data from iterative usability tests of different versions of a product, data from different groups of users in the same usability test, or data from different conditions or designs within the same usability test. You should also have a reasonable sample size (e.g., at least 10 participants per condition) to use the z-score method.

For example, consider the data shown in Figure 8.1 from Chadwick-Dias, McNulty, and Tullis (2003), which shows z scores of performance for two iterations of a prototype. This research studied the effects of age on performance in using a website. Study 1 was a baseline study. Based on their observations of the participants in Study 1, especially the problems encountered by the older participants, they made changes to the prototype and then conducted Study 2 with a new group of participants. The z scores were equal-weighted combinations of task time and task completion rate.

It’s important to understand that the z-score transformations were done using the full set of data from Study 1 and Study 2 combined. They were then plotted appropriately to indicate from which study each z score was derived. The key finding was that the performance z scores for Study 2 were significantly higher than the performance z scores for Study 1; the effect was the same regardless of age (as reflected by the fact that the two lines are parallel to each other). If the z-score transformations had been done separately for Study 1 and Study 2, the results would have been meaningless because the means for Study 1 and Study 2 would both have been forced to 0 by the transformations.

8.1.4 Using Single Usability Metric

Jeff Sauro and Erika Kindlund (2005) developed a quantitative model for combining usability metrics into a single usability score. Their focus is on task completion, task time, error counts per task, and post-task satisfaction rating (similar to ASQ described in Chapter 6). Note that all of their analyses are at the task level, whereas the previous sections have described analyses at the “usability test” level. At the task level, task completion is typically a binary variable for each participant: that person either completed the task successfully or did not. At the usability-test level, task completion, as shown in previous sections, indicates how many tasks each person completed, and it can be expressed as a percentage for each participant.

Sauro and Kindlund used techniques derived from Six Sigma methodology (e.g., Breyfogle, 1999) to standardize their four usability metrics (task completion, time, errors, and task rating) into a SUM. Conceptually, their techniques are not that different from the z score and percentage transformations described in the previous sections. In addition, they used Principal Components Analysis, a statistical technique that looks at correlations between variables, to determine if all four of their metrics were contributing significantly to the overall calculation of the single metric. They found that all four were significant and, in fact, that each contributed about equally. Consequently, they decided that each of the four metrics (once standardized) should contribute equally to the calculation of the SUM score.

An online tool for entering data from a usability test and calculating the SUM score is available from Jeff Sauro’s “Usability Scorecard” website at http://www.usabilityscorecard.com/. For each task and each participant in the usability test, you must enter the following:

After entering these data for all the tasks, the tool standardizes the scores and calculates the SUM score for each task. Standardized data shown for each task are illustrated in Table 8.8. Note that a SUM score is calculated for each task, which allows for overall comparisons of tasks. In these sample data, participants did best on the “Cancel reservation” task and worst on the “Check restaurant hours” task. An overall SUM score, 68% in this example, is also calculated, as is a 90% confidence interval (53 to 88%), which is the average of the confidence intervals of the SUM score for each task.

The online tool also provides the option to graph task data from a usability study, including the SUM scores. Figure 8.2 shows a sample graph from the tool.

8.3.1 Comparison to Goals

Perhaps the best way to assess the results of a usability test is to compare those results to goals that were established before the test. These goals may be set at the task level or an overall level. Goals can be set for any of the metrics we’ve discussed, including task completion, task time, errors, and self-reported measures. Here are some examples of task-specific goals:

Similarly, examples of overall goals could include the following:

Typically, usability goals address task completion, time, accuracy, and/or satisfaction. The key is that the goals must be measurable. You must be able to determine whether the data in a given situation supports the attainment of the goal. For example, consider the data in Table 8.9.

Table 8.9 shows data for eight tasks in a usability study of a website. For each task, a target number of page visits has been predetermined (ranging from 4 to 10). Figure 8.6 depicts the target and actual page views for each task graphically. This chart is useful because it allows you to visually compare the actual number of page visits for each task, and its associated confidence interval, to the target number of page views. In fact, all the tasks had significantly more page views than the targets. What’s perhaps not so obvious is how the various tasks performed relative to each other—in other words, which ones came out better and which ones worse. To make that kind of comparison easier, Figure 8.7 shows the ratio of the target to actual page views for each task. This can be thought of as a “page view efficiency” metric: the closer it is to 100%, the more efficient the participants were being. This makes it easy to spot tasks where the participants had trouble (e.g., Task 3) versus tasks where they did well (e.g., Task 7). This technique could be used to represent the percentage of participants who met any particular objective (e.g., time, errors, SUS rating) either at the task level or at the overall level.

8.3.2 Comparison to Expert Performance

An alternative to comparing the results of a usability test to predefined goals is to compare the results to the performance of experts. The best way to determine the expert performance level is to have one or more presumed experts actually perform the tasks and to measure the same things that you’re measuring in the usability test. Obviously your experts really need to be experts—people with subject-matter expertise, in-depth familiarity with the tasks, and in-depth familiarity with the product, application, or website being tested. And your data will be better if you can average the performance results from more than one expert. Comparing results of a usability test to results for experts allows you to compensate for the fact that certain tasks may be inherently more difficult or take longer, even for an expert. The goal, of course, is to see how close the performance of the participants in the test actually comes to the performance of the experts.

Although you could theoretically do a comparison to expert performance for any performance metric, it’s used most commonly for time data. With task success data, the usual assumption is that a true expert would be able to perform all the tasks successfully. Similarly, with error data, the assumption is that an expert would not make any errors. But even an expert would require some amount of time to perform the tasks. For example, consider the task time data shown in Table 8.10.

Graphing the ratio of expert to actual times, as shown in Figure 8.8, makes it easy to spot tasks where the test participants did well in comparisons to the experts (Tasks 3 and 9) and tasks where they did not do so well (Tasks 2 and 4).

1. An easy way to combine different usability metrics is to determine the percentage of users who achieve a combination of goals. This tells you the overall percentage of users who had a good experience with your product (based on the target goals). This method can be used with any set of metrics and is understood easily by management.

2. One way of combining different metrics into an overall “usability score” is to convert each of the metrics to a percentage and then average them together. This requires being able to specify, for each metric, an appropriate minimum and maximum value.

3. Another way to combine different metrics is to convert each metric to a z score and then average them together. Using z scores, each metric gets equal weight when they are combined. But the overall average of the z scores will always be 0. This metric is useful in comparing different subsets of the data to each other, such as data from different iterations, different groups, or different conditions.

4. The SUM technique is another method for combining different metrics, specifically task completion, task time, errors, and task-level satisfaction rating. The method requires entry of individual task and participant data for the four metrics. Calculations yield a SUM score, as a percentage, for each task and across all tasks, including confidence intervals.

5. Various types of graphs and charts can be useful for summarizing the results of a usability test in a “usability scorecard.” A combination line and column chart is useful for summarizing the results of two metrics for tasks in a test. Radar charts are useful for summarizing the results of three or more metrics overall. A comparison chart using Harvey Balls to represent different levels of the metrics can summarize effectively the results for three or more metrics at the task level.

6. Perhaps the best way to determine the success of a usability test is to compare the results to a set of predefined usability goals. Typically these goals address task completion, time, accuracy, and satisfaction. The percentage of users whose data met the stated goals can be a very effective summary.

7. A reasonable alternative to comparing to predefined goals, especially for time data, is to compare actual performance data to data for experts. The closer the actual performance is to expert performance, the better.