10.1.1 Methods

In 2010, we launched a survey aimed at measuring user satisfaction with the discoverability, ease of use, and relevance of a feature of our software we’ll refer to here as the L&T feature. We used an 11-point scale and asked users’ satisfaction with the feature, along with their likelihood to recommend the product. The recommend question is the question that is the defining feature of the Net Promoter model. To calculate the NPS, we:

This calculation gave us a NPS. Knowing that we had 40% more customers promoting than detracting our product does mean something. But it also begged the question: is 40% a good score?

Industry benchmarks do exist for NPS. For example, the consumer software industry (Sauro, 2011) has an average NPS of 21%—meaning a 20% is about average for products such as Quicken, QuickBooks, Excel, Photoshop, and iTunes. Common practice at Autodesk is to place less stock in benchmarks but rather focus carefully on the aspects of the user experience that drive up the promoters while reducing the detractors.

To isolate the “drivers” of a good user experience, we also included rating questions in our survey that asked about the overall product quality, product value, and product ease of use. We asked these questions on the same 11-point scale used for the recommendation question. We then calculated mean satisfaction scores for each experience variable. Satisfaction is plotted along the x axis shown in Figure 10.1.

Next we ran a multiple regression analysis with Net Promoter as the dependent variable and the attributes as independent variables. This analysis showed us which experience attributes were significant contributors to users’ likelihood to recommend the product. Because it uses the beta coefficient, the analysis takes into account the correlation between each variable. Those correlations are plotted against the y axis in Figure 10.1. The y axis represents the standardized beta coefficient. We call the y axis “Importance” because correlation to the question “would you recommend this product?” is what tells us how important each experience variable is to our users. Plotting satisfaction against importance gives us insight into which experience attributes (interface, quality, or price) are most important to our users.

10.1.2 Results

According to Reichheld (2003), no one is going to recommend a product without really liking it. When we recommend something, especially in a professional setting, we put our reputations on the line. Recommending a product is admitting we are more than satisfied with the product. It signifies we are willing to do a little marketing and promotion on behalf of this product.

This altruistic, highly credible, and free promotion from enthusiastic customers is what makes the recommend question meaningful to measure. Promoters are going to actively encourage others to purchase our product and, according to Reichheld’s research, are more likely to repurchase.

We wanted to determine how a customer’s likelihood to recommend a given product was driven by specific features and by the overall ease of use of that product. A new feature (we’ll call L&T) was included in the product we were studying. When we plotted users’ satisfaction with the L&T feature against their willingness to recommend the product containing the L&T feature, we found that the L&T feature was lower on the y axis relative to other aspects of the interface (as shown in Figure 10.1). Using the L&T feature (L&T Ease of Use) and locating it (L&T Discoverability) scored lower in satisfaction than Product Quality, Product Value, and Product Ease of Use, but they also scored lower in Importance. Users place less importance on this new feature relative to quality, value, and ease of use. Data show that users’ satisfaction with the L&T feature is not as strongly correlated as quality and ease of use to their likelihood to recommend the product and is therefore not as important to driving growth of product sales.

The labels on the quadrants in Figure 10.1 tell us exactly which aspects of the user experience to improve next. Features that plot in the upper left quadrant, labeled FIX, are the highest priority because they have the highest importance and lowest satisfaction.

Data in Figure 10.1 indicate that if we were to redesign the L&T feature, we should invest in L&T Relevance as it plotted higher on the Importance axis than L&T Discoverability and Ease of Use. Discoverability and ease of use of the L&T feature are in the HOLD quadrant, indicating that these should be prioritized last.

10.1.3 Prioritizing Investments in Interface Design

So how much does the user interface of a software product contribute to users’ willingness to recommend the product? We had been told by our peers in the business intelligence department that the strongest predictors to a user’s willingness to recommend a product are:

We ran a multiple-regression on our survey data set (Figure 10.2) and found that variables for the software user experience contribute 36% to the likelihood to recommend (n = 2170). Product Value accounted for 13% and Support accounted for another 9%. To verify the contribution of software user experience to willingness to recommend, we ran another multiple regression on data from a second, similar survey (n = 1061) and found the contribution of user experience variables to be 40%.

We then ran a third survey 1 year later. Regression formulas from the first survey and the third survey are shown, where LTR represents Likelihood to Recommend. In Year 1 we calculated the improvement targets shown in Figure 10.3 (left). We set a target of 5% increase in users’ likelihood to recommend our product and we knew how to achieve that increase from the regression formula: assuming that the other contributing factors remain constant, if we could increase the satisfaction scores for the overall product ease of use, for the usability of Feature 1 and for the usability of Feature 2, then we would see an increase in users’ Likelihood to Recommend of 5%.

In Year 2, we reran the analysis. We found that the actual increase in Likelihood to Recommend was 3%. This 3% increase was driven by a 3% increase in ease of use, a 1% increase in Feature 1’s usability, and a 0% increase in Feature 2’s usability, as summarized by Figure 10.3 (right). Regression formulas for the product studied are shown here:

10.1.4 Discussion

Thus, we found that running the multivariate analysis showed that the user experience contributed 36% to increasing product recommendations. At Year 2, we hadn’t met our target of increasing Likelihood to Recommend our product by 5%, but by investing in ease of use and in a few key features we were able to improve the Likelihood to Recommend by 3%. The Net Promoter model had provided us with a way to define and prioritize investment in user experience design and had given us a way to track the return of that investment year after year.

We wanted to test the Net Promoter model further. Could the model be used as a predictor of sales growth, as it was originally intended (Reichheld, 2003)? We know the average sales price of our products. We know, from the multivariate analysis, that interface design contributes 36% to motivating users to recommend our product. If we knew how many promoters refer the product actively, we could estimate the revenue gains associated with improved user experience of our software.

What we did next is determine if there is a link between “promoters” and an increase in customer referrals. In our survey, we asked if the respondent—all were existing customers—had referred the product to a friend in the last year (Owen & Brooks, 2008). From these data we derived the proportion of customers obtained through referrals and who likely refer others. This allowed us to approximate the number of referrals necessary to acquire one new customer (see Figure 10.4). Data used to derive this number are proprietary. For the purpose of this chapter, we use the number eight: we need eight referrals to acquire one new customer. In the NPS model, it is promoters who refer a product actively. But we didn’t want to assume that every respondent who answered 9 or 10 to the likelihood to recommend question, that is, every promoter, had referred our product actively. The actual percentage of promoters who referred our product actively within the last year was 63%. From this, we derived that the total number of promoters needed to acquire one new customer was 13.

10.1.5 Conclusion

By calculating the number of promoters required to acquire a new customer, we were able to connect the proverbial dots in the software business: a good user experience design drives our users to recommend our products and product recommendations increase customer acquisition, which increases revenue growth. Through multivariate analysis, we have shown that experience design contributes 36% to motivating users to recommend our product. Since we know the average sales price of our product, we were able to estimate the revenue gains associated with improving the user experience of our software. We quantified the value of a good user experience. By tying user experience to customer acquisition, we are able to prioritize design investment in ease of use and in research to improve the user experience of our products.

In summary, this case study shows:

At Autodesk, we’ve found that calculating a net promoter score isn’t as useful as graphing and using key driver charts. Key driver charts target the aspects of the user experience that are needed most urgently in design improvements. By calculating drivers from year to year, we see how our investments in key areas pay out by increasing our users’ likelihood to recommend our products. We watch a features move from the FIX quadrant safely into the LEVERAGE quadrant. Inspiring more customers to promote our product through designing excellent user experiences is what motivates us. It’s not about a score or solely about acquiring new customers, it’s about designing software experiences that are so good, our users will promote them actively.

10.2.1 Methodology

10.2.2 Discussion

In this study, we examined whether people can perform fingerprint self-captures successfully using the proposed real-time feedback system. We found that the participants were very effective and efficient in performing the self-capture tasks with great satisfaction.

The participant completion rate was high (92.5% in Task 1, then improved to 100%) with fewer than 11 errors on average. When examining only positioning errors (errors related to pressure or angles were excluded), the average errors dropped (mean = 5.94). Fingerprint image quality was comparable, if not better, to images taken in the attended situation. In a study with an attended setup, Stanton et al. (2012) reported that the acceptance rates of slaps ranged from 55 to 63%, based on the US-VISIT acceptance criteria. In this self-capture study, acceptance rates ranged from 67.5 to 68.4% for slaps, higher than those in Stanton et al. (2012).

Using the on-screen instructions, participants were able to position their hands accordingly, make adjustments when needed, and capture fingerprint images in approximately 1½ minutes. Ratings on the post-task questionnaire indicate that the participants felt comfortable and confident and interacted without much difficulty with the self-capture process. It was clear to the participants when the capture process began and ended. In debriefing, participants indicated that the self-capture process was easy, straightforward, and quick. They praised the experience of “do-it-yourself” as it gave them a sense of being in control and being trusted; as one participant put it: “The self capture process was very neat. It is easy enough that anybody can do it. It is elementary, easy to use, even children can do it.”

Our second research question was to investigate whether an overlay facilitates the self-capture fingerprint process. The overlay condition did not show consistent advantages or disadvantages of performance, that is, time, errors, and image quality, over the nonoverlay condition. However, more participants (57.5%) perceived that having the overlay helped them with the positioning of their hands and providing visual feedback. One reason for the discrepancy between performance and preference is the experimental configuration. The setup required participants to place their hands on the scanner (often looking down) and look up at the LCD monitor for the fingerprint image and feedback for corrections if needed. The overlay guide was superimposed onto the screen, which added another level of hand–eye coordination. Participants realized corrections were needed according to the visual feedback of their fingerprint image in relation to the overlay on the screen, but had to move their hands to make the actual corrections on the scanner. Participants were observed being more careful in placing their hands on the scanner in the overlay condition as if they wanted to make sure their hands were aligned properly with the overlay, whereas participants positioned their hands more freely in the nonoverlay condition. Sixteen participants indicated that the overlay guide would be very helpful if it were placed on the scanner in order to align their hand on the overlay as they placed their hand on the scanner instead of looking up and down trying to make a perfect alignment.

We observed that participants learned to use the system very quickly. Within-subject comparisons were performed to examine the ease of learning of the system. When comparing the performance between Task 1 and Task 2, it was found that Task 2 performed significantly better on RS (attempt time and errors), RT (errors), task completion time, and total errors. Learning was even evident within a task.

10.2.3 Conclusion

The real-time feedback fingerprint system is a highly usable system and shows evidence of great potential for fingerprint self-captures. By following the on-screen, real-time instructions, participants quickly learned and felt comfortable and confident in capturing their own fingerprints without any assistance. The next step is to determine if users will benefit more in a language-free environment in which all instructions are presented in graphical format (symbols or icons, without any textual elements). With the findings from this study, we are planning future research to answer this question. Although the overlay guide did not show consistent advantages or disadvantages with respect to performance, it was perceived as helpful with hand positioning and provided visual feedback of where users’ hands were in relation to the scanner area. Use of the overlay guide is recommended for use in the fingerprint self-capture process; however, we would recommend that it be placed directly on the scanner.³

10.3.1 Test Iterations

Using Axure-generated HTML prototypes, we performed a total of six rounds of usability testing on the toolbar approach during the design phase. We iterated on the design between rounds of testing, adding functionality as more requirements were incorporated. If a particular area of the product performed below our expectations, we often retested it in subsequent rounds. Because our focus was improving the design, if data integrity conflicted with the right thing to do for the design, we naturally always chose the design. Since we work in an Agile environment, we had to keep the usability testing cycles quite short, usually less than 2 weeks per round, often aiming for 1 week. Generally, we tried to keep one or two iterations ahead of the actual coding work that was going on.

For the purpose of this case study, we focused on four rounds of testing because they repeated the same workflow task with four different designs. We will refer to these as Rounds 1–4.

Employing a resource we often use to expand the coverage of our team, Round 1 of testing was performed in person by a graduate student from the School of Information at the University of Texas at Austin under the mentorship of Associate Professor Randolph Bias. We conducted Rounds 2, 3, and 4 of testing remotely via a conference call and WebEx. We shared our desktop via WebEx and gave the participant control of the mouse and keyboard.

There were a total of 25 participants across the four rounds of testing. We had 3 participants in Round 1, 9 participants in Round 2, 4 participants in Round 3, and 9 participants in Round 4. Participant groups varied for the rounds of testing, partially due to budget constraints, but also due to the fact that users of the Web Experience Management product can vary significantly. Users can range from being long-time, full-time users of the system to brand new, occasional users of the system. Rounds 2 and 3 involved current users of the system, as well as users of competitors’ systems recruited by a market research firm. Round 1 involved representative users from the University of Texas, and Round 4 involved current customers exclusively.

10.3.2 Data Collection

Even though all the usability tests were “formative” in nature, we collected usability metrics for each of the rounds of testing similar to the methodology reported by Bergstrom, Olmsted-Hawala, Chen, and Murphy (2011). From our perspective, usability metrics are just another way of communicating what happened during a usability test. Of course, we also collect qualitative data and that data still represent the bulk of our formative usability test results and recommendations. However, we have found metrics, being numbers, are concise and easy for management and developers to digest. Also, we find them quick and easy to collect and analyze.

We have standardized a set of metrics at OpenText and these were reported and tracked across the rounds of tests to communicate improvements in the design to product owners. Following the ISO definition of usability of “effectiveness,” “efficiency,” and “satisfaction,” we collected task completion rate, time on task, the Single Ease Question (SEQ; Sauro & Dumas, 2009; Sauro & Lewis, 2012) after every task and the System Usability Scale at the end of the test. The SEQ is a single question asking participants to rate the difficulty of the task on a seven-point scale, where 1 is “very difficult” and 7 is “very easy.”

We collected the data using Excel spreadsheets based on a template we use for every usability test that includes formulas for calculating our metrics. As a result, we can report the quantitative results almost immediately after finishing a study. Since we had product release goals, the team was very interested in hearing quickly if we were getting closer to our goals.

The only downside with our spreadsheet methodology is that one of our experimenters had a difficult time with the spreadsheet’s keyboard shortcuts in the beginning of these studies. As a result, we lost many time-on-task measurements and do not have enough data to present those findings here. However, we often see interesting differences in time on task, even when testing mockups.

10.3.3 Workflow

For the purposes of this case study, we focused on one aspect of the project: workflow design and results. The workflow task in all four rounds of usability testing was essentially the same: accept a workflow task assigned to the participant, approve the page being edited, add a note to the workflow, and finish the workflow task assigned. Task instructions given to the participant were:

You’ve received an automatic workflow item! When you made changes to the homepage, a workflow automatically triggered and sent a workflow to you to approve the page. Please complete the workflow and add the note “fixed typo, changed image” so that your editor knows what you’ve changed.

Workflow was a difficult task for us to get right from a design perspective. In the new slim design, we had very little room to work with to communicate complex task requirements, and we had to work within the existing system limitations. There wasn’t time or resources to allow for a complete rewrite of the code. Figures 10.11–10.15 show screenshots of the design and how it changed over the four testing iterations. We started out with a very modular approach, requiring the user to find each piece of the functionality on the toolbar and ended with a more “wizard”-like approach where the user is guided through the process. In each version of the design, the toolbar is shown at the bottom of the screen in gray. The “task panel” (or later “Task Inbox”) is the blue panel just above the toolbar on the left-center portion of the screen. Early versions of the design (1 and 2) included some task functions (“Accept Task”) within the task panel, while later versions (3 and 4) moved those actions into a second “Task Editor” window (the panel on the right side of Design 3, and Screen 2 for Design 4). The designs, along with a brief description, appear in Figures 10.11–10.15.

10.3.4 Results

We were able to demonstrate an improvement in the design of workflow, as indicated by the SEQ and task completion rate (see Figures 10.16 and 10.17).

The SEQ yielded our most interesting results. The mean SEQ score for the workflow task increased in each round of testing, indicating that participants found the task easier with each design iteration. In the initial round of testing, the mean SEQ for workflow was a 3.0. In Round 2, the mean SEQ increased to 3.9, and in Round 3, to 4.5. By Round 4 of testing, the mean SEQ had increased to 5.9.

The task completion rate for workflow also increased from Round 1 to Round 4, but not in the same way as the SEQ scores. The task completion rate went from 33% in Round 1 to 44% in Round 2, but decreased to 25% in Round 3. Finally, the task completion rate was highest in Round 4, at 67%. The drop in the task completion rate in Round 3 was interesting; although fewer participants completed the task successfully, the SEQ score was higher than Rounds 1 and 2. Participants rated the design in Round 3 easier to use than Round 1 or 2, even though it was actually more difficult for them to use.

Because we were conducting formative testing, we were not overly concerned with statistically significant differences between rounds of testing. However, we still calculated 95% confidence intervals as a way of assessing the variability in our data. Even with the small and unequal sample sizes between rounds, we were able to resolve some differences in SEQ scores. The confidence intervals for SEQ (shown as error bars in Figure 10.16) suggest that participants thought the workflow task in Round 4 was substantially easier than in Round 1 or 2. In contrast, the confidence intervals for task completion rate (error bars in Figure 10.17) were much larger and suggest that the differences we saw were small.

Because the new toolbar design was such a large departure from previous designs, we also looked for any differences in data between current users of the system and users of competitive systems during Rounds 2 and 3. We did not see any differences between the different user groups.

10.3.5 Conclusions

Like Bergstrom and colleagues (2011), we found quantitative metrics to be useful in formative testing with rapid iteration cycles. For us, that included rounds of testing conducted both by us and by partners at the University of Texas at Austin. By using task-level measures such as task completion rate and SEQ, we could retest certain aspects of the design, such as workflow, across multiple design iterations independently from the rest of the product. That allowed us to track the progress we made in our design, while also allowing us to add or remove tasks to test other parts of the product.

Because our test changed from round to round, we found that using the SEQ was very useful for us. The SEQ provided us with a metric we could use at the task level to get participants’ subjective impressions of ease of use. We found that the SEQ could be sensitive enough to resolve differences between different designs, even with only three or four participants per round of testing.

10.4.1 Example 1: Deciding on Actions after Usability Testing

One of our earliest examples of triangulation started with a usability test. The prospectus homepage consisted of a long list of subjects (Figure 10.19).

Most people who consider university study start by looking for the subject they are interested in. When we asked participants in a usability test to look for the subject they wanted, we observed that some of them struggled:

We could have done more testing with more participants to measure exactly how much of a problem this was, but instead we decided to use web analytics to investigate the actual behavior of site visitors.

We can also distinguish between visits by logged-in visitors (students and staff) and by other visitors. We distinguish between a single visit—the path someone takes through our site in one continuous experience—and the experience of a visitor—the aggregation of multiple visits from a computer where someone has given us permission to use cookies.

It can be tricky to distinguish different types of visit and visitor, so we find that it’s best to try to focus on the big overall picture and not stress too much about finer details.

For example, we discovered that 37% of visits that involved Information Technology also involved Computing, but that only 27% of visits that involved Computing also involved Information Technology. In addition, we found that Computing was receiving 33% more visitors than Information Technology. This confirmed what we’d seen in usability testing: our participants were more likely to click on Computing (above the fold) than on Information Technology (below the fold).

We looked at the content of these two subjects and discovered that prospective students should really think about both of them before choosing either. From this type of analysis, across the entire list of subjects, we recommended a new design with a much shorter list of subject areas based on actual user behavior, and the clusters of subjects they tended to view together (see Figure 10.21).

The previous organization of subject areas reflected the internal structure of the university at that time; for example, the Mathematics and Computing faculty taught Computing, but the Technology faculty taught Information Technology. The revised organization aligns with visitor expectations and needs and has performed well (with a few tweaks) ever since.

10.4.2 Example 2: Site-Tracking Data

The usability test described in Section 10.4.1 was a major initiative that required a lot of data to persuade many different stakeholders—the type of thing you only want to do occasionally.

This second example is more typical of our everyday work. Some stakeholders came to Viki Stirling, who looks after analytics and optimization, with a problem: they weren’t getting the expected level of conversion from part of their website.

Viki took the site-tracking data and fed the appropriate tracking data into NodeXL, a visualization tool.

Looking at the flows by visits, the problem jumped out at her immediately: A lot of visits arrive at a particular page, but few continue after that (highlighted in red in Figure 10.22). The big arrows from node to node should continue, getting only slightly smaller at each step. At the problematic page, we suddenly see that the larger arrow flows backward up the chain, with only a small arrow moving on to the next step.

When she investigated the problematic page, it was obvious how to revise it. But Viki was suspicious: although this task isn’t common, it’s important for the relatively few visitors who attempt it. She investigated further, looking at the flows by visitor (see Figure 10.23). This revealed that the previous step in the process was also causing problems: visitors are moving backward and forward from that step, clearly trying to make progress but failing. Once again, a look at the relevant web page quickly revealed the necessary changes.

From the UX point of view, we might immediately ask: why didn’t the stakeholders do usability testing, which would probably have revealed these problems ahead of time? The answer is that, of course, the Open University does lots of usability testing, but they face a challenge familiar to any organization with a huge and complex website, which is one of prioritization. In this example, the problematic task is rather unusual and relevant only to a small number of users at a very specific point in their progression from enquirer to student.

10.4.3 Example 3: Triangulation for Iteration of Personas

The two previous examples demonstrate the use of measurement techniques for specific changes. Our third example illustrates the use of metrics for one of the UX tools we use all the time: personas.

We first started using personas after Caroline Jarrett learned about them from Whitney Quesenbery at the Society for Technical Communication Conference in 2002. They were based on our experience of usability test participants over a few years—by that point we had been usability testing since 1998—and Sarah Allen validated them against various internal data sources at the time. With Whitney’s help, we’ve been using, updating, and revalidating the personas ever since. Pruitt and Adlin (2006) include a short overview of our experience with personas.

For example, the Open University introduced Foundation Degrees, shorter degree programs focused on training for particular jobs that are somewhat similar to the U.S. “Associates Degree.” To help with our design activities around Foundation Degrees, we added in a persona, “Winston,” who was interested in the Foundation Degree in Materials Fabrication and Engineering. But we discovered that we weren’t meeting Winstons in usability tests. Viki Stirling had the idea of doing some visit tracking to see whether the routes through the site that we envisaged for the personas were actually sufficiently based in data. She discovered that most of them were, but Winston really wasn’t justified, the numbers just weren’t there. Winston became Win, interested in the Foundation Degree in Early Years (see Figure 10.24).

Lindsay’s reasons for studying are slightly different to Win’s, and she’s focusing slightly more on costs and fees—but overall, she’s close enough that we can be confident that a design intended for persona Win will also work for real aspiring students like Lindsay.

10.4.4 Summary

Most user experience techniques are valuable on their own, and we’re happy to use them individually, as illustrated by our everyday example, number 2 above.

We find that the real value comes from comparing what we learn from larger scale quantitative techniques with what we learn from small-scale, qualitative techniques—and continuing to do that over many years.

10.5.1 Background

As user experience research achieves greater prominence in business organizations, we are often asked to help gauge the emotional experience of a product, as well as its usability. Usability professionals have a variety of tools and techniques available to understanding human behavior. However, the tools used commonly to measure the emotion of participants while attempting to complete tasks rely on either an observer’s interpretation of how the participant is feeling or the participants’ description of their reactions (e.g., through think-aloud, protocolor post-task ratings). These interpretations are subject to phenomena such as the observer effect, the participants’ inclination to please, and the time passed since they had the reaction. Other tools, such as the Microsoft Product Reaction Cards, show the direction of a participant’s response (positive or negative) to a product, but not the magnitude of that emotion (Benedek & Miner, 2002).

Adding biometric measures to user research provides a way to measure users’ arousal as they use a product. Arousal describes the overall activation (emotional stimulation and cognitive load) experienced by a user, as measured by biometric measures such as electrodermal activity (EDA). These measures capture physiological changes that co-occur with emotional states (Picard, 2010). Because biometric measures are collected in real time during a user’s interaction with a product, they provide a direct measurement of arousal that is not affected by observer or participant interpretation.

This case study describes initial research to gauge the effectiveness of a new technique that assigns meaning to biometric measures. We hypothesized that by combining biometric measures with feedback from the Microsoft Product Reaction Cards, we could gain a detailed description of a user’s arousal, the interaction that activated a change in arousal, and assignment of emotion (positive or negative) to that interaction. For example, if we saw a user’s arousal level increase sharply while attempting a search task, the Microsoft Product Reaction Cards selected would indicate whether the arousal increased due to frustration or pleasure. This combination would let practitioners quickly identify areas of a product that participants found more or less engaging or frustrating, even if the participants do not articulate their reaction.

Our Bentley DUC team partnered with Pearson Education, a textbook publisher that had recently moved into the digital textbook space on the iPad. We focused this new technique on a usability study that would highlight any differences in arousal between digital and paper textbooks.

10.5.2 Methods

10.5.3 Biometric Findings

10.5.4 Qualitative Findings

Once we observed that average peaks per minute were trending higher for the paper textbook than the digital textbook, we compiled the qualitative feedback that we had collected from the Microsoft Reaction Cards, as well as the poststudy questions. While participants described the digital version as “organized,” “easy to use,” and “efficient,” participants described the paper textbook as “slow,” “time-consuming” and “old.” Figures 10.28 and 10.29 show word clouds for the paper text book and digital textbook, respectively. The larger the font, the most frequently the card was selected. The shade of the text does not have any meaning.

Affectiva Q Sensor data showed us that participants experienced higher arousal while using the printed textbook, but combining those results with qualitative data from the Microsoft Product Reaction Cards, as well as moderated discussion, revealed that that the higher levels were due to negative emotions from a difficulty in performing search and comprehension tasks.

At the end of the session, the moderators asked participants to choose between digital and printed versions of the textbook; surprisingly, participants were split with five preferring the digital and five preferring the printed textbook. Although this was not the focus of our research, this split suggests that the decision between paper and digital textbooks relies on more than the difference in emotional experiences.

10.5.5 Conclusions and Practitioner Take-Aways

This study successfully tested the feasibility of integrating biometric measures with qualitative user feedback. Results showed that the benefits of this technique over standard usability testing include:

Specifically, additional information gained from the Q Sensor for this study redirected and clarified the impressions that we had based on observation and the participants’ thinking aloud. Based on those measures, we expected that the digital textbook would have been more stimulating based on how the participants described and interacted with it. However, Q Sensor data revealed that participants had a higher arousal level while using the paper textbook. Other qualitative data indicated a negative direction for those emotions, as participants struggled with their tasks. This negative emotion was stronger than the pleasurable emotions felt while using the digital textbook.

This approach would be ideal for projects whose goal is to understand participant emotional responses and the severity of those reactions throughout their interaction with a product. By associating metrics across data sets, researchers can pinpoint a participant’s exact emotional reaction, and what was causing that reaction, at any point during their session. This unique combination of metrics provides a new window into a participant’s emotional reactions above and beyond what is articulated during a standard think-aloud usability study.

However, these techniques require additional time to set up the study appropriately and to analyze the results. For example, researchers will want to plan on using time markers with the Q Sensor. We learned during our data analysis that we could have saved significant efforts postanalysis by adding more markers to Q Sensor data during the sessions. For this study, the team spent approximately 2 work weeks scrubbing, combining, and analyzing the data. However, a more recent project that’s used these same techniques only took us 3 work days as we used more Q Sensor markers during the sessions. This method probably won’t make sense for a basic formative usability study, but we believe would offer benefits for projects with a larger scope.

We are continuing to refine and build out these techniques through additional projects and are applying them to new domains.