• Strongly positive comments (e.g., “This is terrific!”)

• Other positive comments (e.g., “That was pretty good.^”)

• Strongly negative comments (e.g., “This website is terrible!”)

• Other negative comments (e.g., “I don’t much like the way that worked.^”)

• Suggestions for improvement (e.g., “It would have been better if…^”)

• Questions (e.g., “How does this work?^”)

• Variation from expectation (e.g., “This isn’t what I was expecting to get.”)

• Stated confusion or lack of understanding (e.g., “This page doesn’t make any sense.”)

• Stated frustration (e.g., “At this point I’d just leave the website!”)

7.2.1 How Eye Tracking Works

Although a few different technologies are used, many eye-tracking systems, such as the one shown in Figure 7.2, use some combination of an infrared video camera and infrared light sources to track where the participant is looking. The infrared light sources create reflections on the surface of the participant’s eye (called the corneal reflection), and the system compares the location of that reflection to the location of the participant’s pupil. The location of the corneal reflection relative to the pupil changes as the participant moves his eyes.

The first activity in any eye-tracking study is to calibrate the system by asking the participant to look at a series of known points; then the system can subsequently interpolate where he is looking based on the location of the corneal reflection (see Figure 7.3). Typically the researcher can check the quality of the calibration, usually expressed as degrees that deviate from the X and Y visual planes. Deviations less than one degree are generally considered to be acceptable, and less than one-half of a degree is very good. It is critical that the calibration is satisfactory; otherwise all the eye movement data should not be recorded or analyzed. Without a good calibration there will be a disconnect between what the participant is actually looking at and what you assume he is looking at. Following calibration, the moderator makes sure the eye movement data are being recorded. The biggest issue tends to be participants who move around in their seat. Occasionally the moderator is required to ask the participant to move back/forward, left/right, or raise/lower their seat to recapture the participant’s eyes.

Information provided by an eye-tracking system can be remarkably useful in a usability test. Simply enabling observers to see where the participant is looking in real time is extremely valuable. Even if you do no further analyses of eye-tracking data, just this real-time display provides insight that would not be possible otherwise. For example, assume a participant is performing a task on a website and there’s a link on the homepage that would take him directly to the page required to complete the task. The participant keeps exploring the website, going down dead ends, returning to the homepage, but never reaching the required page. In a situation like this, you would like to know whether the participant ever saw the appropriate link on the homepage or whether he saw the link but dismissed it as not what he wanted (e.g., because of its wording). Although you could subsequently ask participants that question, their memory may not be completely accurate. With an eye-tracking system you can tell whether the participant at least fixated on the link long enough to read it.

7.2.2 Visualizing Eye-Tracking Data

There are many ways to visualize eye-tracking data. These visualizations tell the story about where people were looking and when. They might be the only thing that your stakeholders really care about. All eye-tracking visualizations are either at an individual level, showing eye movements for one participant, or at an aggregate level, showing eye movements for more than one participant.

Figure 7.6 shows the series or sequence of fixations that an individual participant made on the Amazon Video website, also known as a scan path. This is perhaps the most common way to visually represent the eye movements for a single participant. A fixation is defined by a pause in the eye’s movement within a well-defined area. Normally these pauses last at least 100 msec (1/10th of a second) or longer. Fixations are usually numbered to indicate their sequence. The size of each circle is proportional to the length or duration of the fixation. The saccades, or movements between fixations, are shown by the lines. In Figure 7.6 it is easy to note that the participant was focused primarily on the faces, as well as the first “learn more” box (on the far left). Scan paths are an excellent way to show how a participant looked at the page and what elements they saw in what order.

By far the most common way to visually represent eye movement for multiple participants is through a heat map (see Figure 7.7). In this visualization, the brightest areas (red) represent a greater density of fixations. It is an excellent way to get a sense of what areas of the page attract more (and less) visual attention. It is important to keep in mind that the analysis software allows the researcher to define the scale of what is considered “red” versus “orange,” etc. So, beware that the researcher can easily exaggerate the heat maps to show more or less color. We recommend using the default settings on most software; however, it is important to experiment with using different scales. The opposite visualization is called a focus map, which makes transparent those areas that received more visual attention and darkens those areas that received little or no visual attention. In some sense, a focus map is more intuitive, but a little less common since it is hard to see those areas that are ignored by users.

7.2.3 Areas of Interest

The most common way to analyze eye-tracking data is by measuring visual attention on specific elements or regions. Most researchers are not just interested in how visual attention is distributed across a web page or scene, but whether participants noticed certain things and how much time was spent looking at them. This is particularly the case in marketing, whereby the success of an ad campaign is tied directly to getting customers to notice something. Also, it’s a concern when there are certain elements critical to task success or having a positive experience. When users don’t see them, you can be sure that is a problem.

Figure 7.8 is an example of how to define specific regions on the page. These regions are typically referred to as “look zones” or “areas of interest” (AOIs). AOIs are essentially those things that you want to measure, as defined by a set of x,y coordinates. When analyzing time spent looking at different regions, keep the following in mind:

Another useful way to analyze eye movement data by AOIs is through a binning chart (see Figure 7.9). A binning chart shows the percentage of time spent looking at each AOI by some time interval. Keep in mind that the percentages might not add up to 100% unless all the available space is represented within an AOI. Figure 7.9 shows that AOI 1 (green) received more visual attention in the first few seconds relative to the last few seconds. Conversely, AOI 2 (gray) received more visual attention in the last few seconds compared to the first few seconds. This is a useful way to see the relative prominence of each AOI, not just expressed as a total amount of time. Figure 7.10 is a gridded AOI that shows the amount of visual attention given to equal-sized cells. This is a helpful visualization to see the visual attention across a page, particularly when the elements are not consistent across all pages. For example, the researcher may choose to aggregate data from more than one web page into a single gridded AOI to see generally where users are looking.

7.2.4 Common Eye-Tracking Metrics

There are many metrics associated with eye-tracking data. The following are some of the most common eye-tracking metrics used by UX researchers. It’s important that all of these metrics are associated with specific AOIs. Figure 7.11 is an example of the type of metrics derived from a single AOI.

7.2.5 Eye-Tracking Analysis Tips

Over the years we have learned a few things about how to analyze eye-tracking data. Above all else, we strongly recommend you plan your study carefully, as well as taking time to explore the data. It’s very easy to draw the wrong conclusion based on a few heat maps. Here are a few other important tips to keep in mind as you dive into the data.

7.2.6 Pupillary Response

Closely related to the use of eye tracking in usability studies is the use of information about the response of the pupil. Most eye-tracking systems must detect the location of the participant’s pupil and calculate its diameter to determine where he or she is looking. Consequently, information about pupil diameter is included in most eye-tracking systems. The study of pupillary response, or the contractions and dilations of the pupil, is called pupillometry. Most people know that the pupil contracts and dilates in response to the level of ambient light, but many people don’t know that it also responds to cognitive processing, arousal, and increased interest. Typically the greater the level of arousal or interest, the larger the pupil size.

Because pupil dilation is correlated with so many different mental and emotional states, it’s difficult to say whether pupillary changes indicate successes or failures in everyday usability testing. However, measuring pupil diameter may be useful in certain situations where the focus is on the amount of mental concentration or emotional arousal. For example, if you are interested mainly in eliciting an emotional response to a new graphic on a website, then measuring changes in pupil diameter (from baseline) may be very useful. To do this, simply measure the percentage deviation away from a baseline for each participant and then average those deviations across the participants. Alternatively, you can measure the percentage of participants who experienced dilated pupils (of a certain amount) while attending to a particular graphic or performing a specific function.

7.3.1 Affectiva and the Q-Sensor

Based on an interview with Daniel Bender, product manager, Affectiva (www.affectiva.com).

The Affective Computing Research group at MIT’s Media Lab was founded in 1998 by Professor Rosalind Picard Sc.D. in an effort to develop technologies that advance understanding of emotions. The aim of the research group is to restore a proper balance between emotion and cognition in the design of technologies for addressing human needs (http://affect.media.mit.edu/). Picard and coinvestigator Rana el Kaliouby, Ph.D., cofounded Affectiva in April 2009 to commercialize technologies developed at the MIT research group. The first product to come from Affectiva is called the Q Sensor (see Figure 7.12).

The Q Sensor is a device worn on the wrist that measures the electrical conductance of the skin known as electrodermal activity (EDA). EDA increases when you sweat—small increases in moisture are associated with increased sympathetic nervous system activity indicating emotional activation or arousal. Three types of activation can lead to increases in arousal: increases in cognitive load, affective state, and/or physical activity. Emotional states associated with EDA increases include fear, anger, and joy. Arousal increases are also associated with cognitive demands and may be seen when you are engaged in problem-solving activity. Our state of arousal—and hence the conductivity of our skin—is lower when we are in a relaxed state or bored.

Researchers in a number of fields are using the Q Sensor to measure sympathetic nervous system activity objectively. One of the initial use cases for the Q Sensor has been in understanding the emotional state of students on the autism spectrum. Individuals with autism spectrum disorders often present neutral facial expressions, despite feeling threatened, confused, or otherwise emotionally distressed. Researchers working with autistic students are reviewing EDA data captured with the Q Sensor to better understand the triggers for emotional outbursts. Eventually, the technology will make its way into the classroom where it will serve teachers by providing early warning signals that students are becoming stressed without outward displays of duress. This will enable teachers to respond to their students in a timely and appropriate way.

In the area of user experience research, the Q Sensor can be used to help pinpoint moments of excitement, frustration, or increased cognitive load experienced by the participant. The UX researcher establishes a baseline for each participant. Experiences are then compared to their baseline, with particular attention given to the peaks, or places where there was a peak level in arousal.

While it is helpful knowing what may have triggered an increased level of arousal, it does not tell the researcher whether the experience was positive or negative. This is known as valence. Picard recognized the need to measure valence objectively as she brought Affectiva cofounder el Kaliouby to MIT in January 2007. El Kaliouby’s research had been focused on measuring facial expressions using computer-vision and machine-learning techniques. This technology matured and was incorporated into Affectiva’s second product, the Affdex facial expression recognition system. Affdex is a passive web-based platform that can take streaming video as an input and predict the presence of facial expressions in close to real time. Affdex is being used to measure emotional response to media in online panels and in usability labs. Affdex facial-expression recognition provides an indication of the type of experience associated with the state of arousal.

Facial expressions are captured through a standard web camera on the participant’s computer and time synchronized with data from the Q Sensor. This provides a rich data set, as peaks in arousal can be associated with a positive or negative valence. With Affdex, Affectiva is building the largest database of spontaneously generated facial expressions in the world. This will allow Affectiva to develop more advanced classifiers of different emotions, which will be used to predict increases in sales or brand loyalty. This powerful technology will arm the UX researcher with an additional set of tools to better understand emotional engagement across a wide variety of experiences. Case study 10.5 highlights use of the Q Sensor in the context of using an onscreen and tablet-based textbook.

7.3.2 Blue Bubble Lab and Emovision

Based on an interview with Ben van Dongen, CEO and founder, BlueBubbleLab (www.bluebubblelab.com)

Blue Bubble Lab is a media and technology company based in Palo Alto and Amsterdam that focuses on bringing more relevant messages to consumers based on their emotions and behavior. ThirdSight (www.thirdsight.com), a subsidiary of Blue Bubble Lab, has developed a suite of technology products that bring together computer vision, facial expression analysis, and eye tracking. One product, Emovision, is an application that allows the researcher to understand the participants’ emotional state while pinpointing what they are looking at. It is a powerful combination of technologies because the researcher can now draw a direct connection between visual stimuli and an emotional state at any moment in time. This will be invaluable in testing how different visual stimuli produce a range of emotional responses.

Emovision determines the emotional state based on the participants’ facial expressions. In the 1970s, Paul Ekman and Wallace Friesen (1975) developed taxonomy for characterizing every conceivable facial expression. They called it the Facial Action Coding System, which included 46 specific actions involving the facial muscles. From his research, Ekman identified six basic emotions: happiness, surprise, sadness, afraid, disgust, and anger. Each of these emotions exhibits a distinct set of facial expressions that can be reliably identified automatically through computer vision algorithms. Emovision uses a webcam to identify the facial expressions at any moment in time and then classifies it into one of seven unique emotions: neutral, happy, surprise, sad, scared, disgusted, and puzzled. At the same time, the webcam is used to detect eye movements.

Figure 7.14 shows how the Emovision application works. On the left side of Figure 7.14 the participant’s facial expressions are analyzed. Distinct facial muscles are identified and, depending on their shape and movement, an expression is identified. The right side of the application (Figure 7.14) shows the stimulus that is being viewed and the eye movements. In this case the participant is watching a TV commercial, and fixating (as represented by the red dot) in between the two women. The bottom of the screen shows the emotion (in this case it is happy) and assigns a percentage. The line graph depicts the change in emotion over time. When analyzing these data, the researcher can look at any moment in time and identify the associated emotion(s). Also, the researcher can view the overall mood of the experience by seeing the frequency distribution of all the emotions across the experiment. This might be valuable data when comparing different products.

One of the more fascinating applications of this technology is being able to target messages to consumers based on their mood. Figure 7.15 is an example of how this technology can be used to capture facial expressions in the real world, determine the overall mood (positive or negative), as well as demographics such as age and gender, and then deliver a targeted message on a digital billboard or other platform.

7.3.3 Seren and Emotiv

Based on an interview with Sven Krause, key account director, Seren (www.seren.com/)

Seren is a customer experience consultancy based in London. Sven Krause developed a way of measuring a user’s emotional engagement and behavior by combining electroencephalography and eye-tracking data. Seren is applying this technology to a wide variety of contexts, including branding, gaming, service, and website design. Researchers at Seren feel this new technology allows them to gain a more complete picture of the user experience as it measures participants’ unconscious responses to a stimulus.

Seren uses an EEG device developed by Emotiv (www.emotiv.com). EEG measures brain waves, specifically the amount of electrical activity on different parts of the participant’s scalp. Electrical activity is associated with cognitive and emotional states. There is a certain pattern of electrical activity when the participant is in a more excited state relative to a calm state. Also, specific patterns of electrical activity have been associated with other emotional states, such as frustration, boredom, and engagement. EEG technology has been used for many years, for example, helping diagnose patients with epilepsy, sleep disorders, strokes, and other neurological conditions. Only recently has it been applied within the field of marketing and customer experience.

Seren has worked with SMI (www.smivision.com) to integrate the Emotiv headset with their SMI eye tracker. This allows Seren’s researchers to determine what participants are looking at and what triggers their emotional and cognitive state. The integration of both EEG and eye-tracking data is critical, as all data will have a consistent time stamp, allowing the researcher to explore both eye movement and EEG data for a specific event.

Setting up and using their system is fairly straightforward. Participants wear the EEG device on their head, with a series of small conductive pads that contact the scalp and forehead. The EEG device is connected wirelessly to the eye tracker. Baseline measures are taken for a few minutes to allow the participant to get comfortable with the setting. After the researcher feels she has achieved an acceptable baseline, the study begins. Figure 7.16 shows a typical setup. The researcher is monitoring both eye movements and EEG feedback in real time (as shown in Figure 7.17).

Electroencephalography data are extremely useful in monitoring the emotional engagement of a participant throughout a session. Results can be used to prompt additional questions or to create “emotional heatmaps” that identify areas that led to a change of the emotional state.

7.4.1 Heart Rate Variance

One of the most common ways to measure stress is by heart rate, specifically heart rate variability (HRV). HRV measures the time intervals between heart beats. Somewhat counterintuitive, having a certain level of variability in heart rate is healthier than not having any variability at all. Measuring HRV has become much easier in the last few years, thanks primarily to the obsession with fitness and health and, of course, mobile technology. Many runners and other athletes are interested in measuring their heart rates when running. These athletes are likely to be wearing a device on their chest that measures their heart rate and pulse directly. This information can be sent directly to any device. People who aim to reduce stress in their life now can use a handful of smartphone apps to help them measure and monitor their stress levels. One popular app, called Azumio (www.azumio.com), allows users to measure their stress levels using their own smartphone. The user gently places their finger over the camera, and the software is able to detect the heat rate and calculate HRV (see Figure 7.18). HRV is calculated after about 2 minutes, and a stress score is calculated.

These new apps might be useful for UX research, particularly when evaluating more emotionally based products such as dealing with a person’s health or finances. It would be very easy to measure HRV before and after using different designs. It is quite possible that one design resulted in a greater overall HRV rate across participants than other designs. We certainly don’t recommend this as a sole measure of their experience, but it might offer an additional set of data points and potentially insight into causes of stress in their experience.

7.4.2 Heart Rate Variance and Skin Conductance Research

Several studies have sought to determine whether skin conductivity and heart rate could be used as indicators of stress or other adverse reactions in a usability setting. For example, Ward and Marsden (2003) used skin conductance and heart rate to measure user reactions to two versions of a website: a well-designed version and a poorly designed version. The poorly designed version included extensive use of drop-down lists on the homepage to “hide” much of the functionality, provided impoverished navigational cues, used gratuitous animation, and had occasional pop-up windows containing ads. Heart rate and skin conductance data were plotted as changes from the participant’s baseline data established during the first minute of the session.

Both measures showed a decrease in heart rate and skin conductance for the well-designed website. For the poorly designed site, skin conductance data showed an increase over the first 5 minutes of the session, followed by a return to baseline over the final 5 minutes. Heart rate data for the poorly designed version showed some variability, but the overall trend was to stay at the same level as the baseline, unlike the well-designed version, which showed a decrease relative to baseline. Both measures appear to reflect greater stress in interacting with the poorly designed site.

Trimmel, Meixner-Pendleton, and Haring (2003) measured skin conductance and heart rate to assess the level of stress induced by the response times for web pages to load. They artificially manipulated page load times to be 2, 10, or 22 seconds. They found significant increases in heart rate as response time (page load time) increased, as shown in Figure 7.19. A similar pattern was found for skin conductance. This is evidence of physiological stress associated with longer response times.

7.4.3 Other Measures

A few creative researchers have come up with some other techniques that might be appropriate for assessing the user’s level of frustration or engagement while interacting with a computer. Most notably, Rosalind Picard and her team in the Affective Computing Research Group at the MIT Media Lab have investigated a variety of techniques for assessing the user’s emotional state during human–computer interaction. Two of these techniques that might have application to usability testing are the PressureMouse and the Posture Analysis Seat.

The PressureMouse (Reynolds, 2005), shown in Figure 7.20, is a computer mouse with six pressure sensors that detect how tightly the user is gripping the mouse. Researchers had users of the PressureMouse fill out a five-page web-based survey (Dennerlein et al., 2003). After submitting one of the pages, participants were given an error message indicating that something was wrong with their entries on that page. After acknowledging the error message, participants were then taken back to that page, but all the data they had entered had been deleted and they had to reenter it. As illustrated in Figure 7.21, participants who had been categorized as members of a “high-response” group (based on their negative ratings in a usability questionnaire about the online survey) gripped the mouse significantly tighter for the 15 seconds after their loss of data than they did for the 15 seconds before.

The Posture Analysis Seat measures the pressure that the user is exerting on the seat and back of the chair. Kapoor, Mota, and Picard (2001) found that they could reliably detect changes in posture on the part of the participant, such as sitting upright, leaning forward, slumping backward, or leaning sideways. These may be used to infer different levels of engagement or interest on the part of the participant. Of course, anyone who has taught can easily see a student’s engagement based on how much they slouch in their seat!

These new technologies have yet to be used in everyday usability testing, but they look promising. As these or other technologies for measuring engagement or frustration become both affordable and unobtrusive, they can be used in many situations in which they could provide valuable metrics, such as designing products for children who have limited attention spans, evaluating users’ patience for download times or error messages, or measuring teenagers’ level of engagement with new social networking applications.

1. A structured approach to collecting unprompted verbal expressions during a usability test can be very helpful by tabulating the number of positive and negative comments made by participants during each of the tasks.

2. Eye tracking can be a significant benefit in many kinds of usability tests. The technology continues to improve, becoming more accurate, easier to use, and less intrusive. The value is being able to compare the effectiveness of different designs, as well as calculate metrics based on areas of interest. Key metrics include dwell time, time to first fixation, and hit ratio. There are many ways to visualize results from eye tracking, such as heat maps and gridded AOIs.

3. There are three ways to measure emotions: skin conductance, facial expressions, and EEG. Skin conductance measures level of arousal, facial expressions are classified and associated with six basic emotions, and EEG measures brain wave activity with unique signatures tied to specific emotional responses. There are new technologies based on each approach that even integrate eye movement data into their applications. These are powerful new tools used to gain insight into the emotional response of the user.

4. Stress is an important component of the user experience. It is measured most often as heart rate variance. New apps allow the researcher to calculate HRV very easily. However, there are many factors that impact stress beyond the user experience.

5. Other techniques for capturing information about the participant’s behavior, such as a mouse that registers how tightly it is being gripped, are on the horizon and may become useful additions to the battery of tools available for use in usability testing.