Chapter 6
You can’t manage what you don’t measure
Signs that indicate research is being used well and how you can systematically track success (or failure)
This chapter identifies nine signs that research is making an impact on stakeholders, teams, and organizations and describes ways to determine whether it is being used well. It also suggests techniques to track and monitor the impact of research.
Livia Labate
President of the company explaining what an ethnograhic study is during All Hands Meeting. My work here is done.
Introduction
Think about the reasons that people and organizations decide to conduct UX research. Why are they doing this? Why all the effort? The number one reason they do it is because they want to learn about what their customers want and to make necessary changes to achieve that. When stakeholders act on research, you can clearly point to the positive effect that UX research is having on the organization, its products, and its customers. All you need to do is pay attention and be aware.
You can tell whether or not stakeholders act upon UX research results if you notice, track, and monitor their actions. Pay attention to what your stakeholders are saying about and doing with research results. By monitoring factors such as consumption of research, allocation of more funding, signs of stakeholder trust in UX research, changes the organization is making, and recognition researchers (or you) are getting, you can better understand what is working well and what needs to be improved with your practice. By tracking the status and the decisions made by stakeholders based on research, you can inform yourself about areas that require your attention.
This chapter identifies nine signs of stakeholder buy-in for research and offers practical techniques for monitoring agreement.
Signs that research is being used well
One of the hardest things to measure is how much your stakeholders buy in to UX research. There is no clear, quantifiable answer to this question. In one of my recent conference appearances, I was asked this question about stakeholder buy-in: “What does winning look like when it goes wonderfully right? Horribly wrong?” I thought about it a little bit and came up with nine signs that your stakeholders have bought into UX research. Here is what winning looks like when it goes wonderfully right:
2. Budget is allocated for more research.
3. Findings are long and lasting.
5. Skeptical stakeholders become believers.
If you see evidence of at least two or three of these signs, you can say that your stakeholders have bought into UX research. On the other hand, if you identify one or none of those signs, things are not going so well. Let’s discuss the signs.
Research is consumed
A great indication of research uptake is that research recommendations – especially the most important ones – are followed. The most basic example of research consumption is when a problem that was uncovered during a research activity was solved or fixed as a result of the study. If stakeholders choose to fix only the problems that are easy to solve, that’s one thing. A better indication of consumption is when they deal with fixing big problems identified during research. When significant changes are made to product design following research, that’s a great sign for research consumption.
Watch my interview with Giles Colborne, author of Simple and Usable, Managing Director of cxpartners, and former president of the UK UPA (Usability Professionals’ Association). Giles argues that you are successful if you shift people’s point of view of customers and how they use products. Use QR code 126 to access the video, a quick summary of the interview, and Giles’s biography.
Another sign – less meaningful, yet favorable – of research consumption is when the company’s representatives are using the company’s investment in user research as a part of the sales pitch. When heat maps from your recent eye-tracking study are shown to potential customers, it’s a sign that the company wants to show that it is serious about design. It does not necessarily mean that this is the case, but it is a positive indication that stakeholders perceive research as a unique selling proposition. If people are using research to tell customers how good a job the company is doing when in reality the research recommendations aren’t followed, that stinks. In such cases, UX is seen as a way to dupe customers. If this happens a lot to you, I suggest that you refer to Chapter 1’s discussion about difficult people, teams, and organizations.
This story from Italy starts by describing a standard usability study and has a surprising ending about how UX research can become a unique selling proposition.
Better Buying Experience for Blackberry Phones
Michele Visciola, General Manager, Experientia, Italy
When Research in Motion (RIM) asked Experientia to improve the out-of-the-box experience for Blackberry phones, we knew it was vital that our testing processes were as realistic as possible. We needed to explore the entire experience of purchasing a smartphone in a retail environment; we decided that the best way to do so was to simulate a full purchasing process, then put participants in our usability lab to see what really happened after the purchase.
“Out of the box” involved two distinct phases: (1) onsite observations carried out in mobile phone shops and showrooms, and (2) usability testing in the Experientia lab.
The findings of the research gave RIM information on participants’ emotional reactions and satisfaction with the purchase and unboxing experience. As six models of phones were benchmarked, RIM could clearly see how the Blackberry products compared with competitors.
We presented recommendations for both project phases, including suggestions on point of sale information, packaging, improving the purchase experience, and making the configuration and interface of the phone more user-friendly and navigable. RIM thought our findings were so important that it turned them into a white paper, which it distributed to its sales points worldwide.
Many research reports have been admired for their correctness but then ultimately ignored when making product decisions. A white paper might not always be a good sign. It’s up to you and your stakeholders to ensure that such a report is backed up by action.
Budget is allocated for more research
Success means that stakeholders or clients who have experienced UX research and enjoyed its benefits allocate more funding. It is a great sign when it comes from people, teams, and organizations you have already worked with before – and even more so when it is coming from people new to research. The latter means that rumors about UX research are spreading, which is definitely a good sign.
Watch my interview with Kim Goodwin, author of Designing for the Digital Age and independent consultant. Kim says that when stakeholders budget for more research, that’s the best sign for you that they have bought into it. Use QR code 124 to access the video, a quick summary of the interview, and Kim’s biography.
Findings are long and lasting
The nature of generative research is that study results have long shelf life. These are usually the studies that have goals such as identifying user needs or uncovering who the users are. Chances are good that findings from these studies will remain valid and true for long periods of time. When I reflect on generative studies that I have conducted in the past, I see that they have had shelf lives of a few years. The reason, I believe, is that findings that emerge from that type of research relate more to human behavior and human nature, which tend not to change too often. An example would be the study mentioned in Chapter 5 in the discussion about running a research expo. That study had a goal of identifying the users of a product the company wanted to develop and their needs. Findings from this study lasted and were used for more than two years after it was completed because the basic needs of users in that realm did not change and will probably not change dramatically in the future.
Methodology is not always the factor that determines the length of shelf life of a research project. Sometimes it is the research goal that sets it. For instance, different stakeholders might use results of a study that compares the efficiency of two competing designs or products for long periods of time.
When study results are used and consumed by stakeholders for a long time after it has been completed, that is a good indicator for success. When this use does not happen, it might mean that something went wrong with planning, collaboration, and communication. It might also mean that stakeholders internalized study findings so well that they now think it is something they have always known.
Trust is established
One of the most frustrating things that can happen to a UX researcher is realizing that stakeholders do not trust his or her work. It is probably very rare that stakeholders will say they don’t trust your work to your face. On the other hand, one of the most encouraging things a UX researcher can experience is when trust is established with stakeholders. In companies where people frequently move on to other positions, you are faced with having to establish trust all over again with new stakeholders. There are two clear signs for established trust. It is important for me to emphasize that the absence of these signs does not mean your stakeholders do not trust you. It might, but there is no single sign for it. The two signs are:
Researchers are invited to important discussions. When people want to hear what you have to say, it is usually a good sign that trust is being established. When you get a seat at the table where important decisions are being made, that’s a great sign. I realize that UX researchers are rarely invited to the table. But – and this is a big “but” – more and more companies now realize that they need to make the table bigger so that it includes UX people.
Teams want to work with researchers. When UX research is having an impact, people talk with each other about it. It is not uncommon to see “new business” arriving at the doorstep of UX researchers who make an impact. It could start with an email, an IM conversation, or a hallway chitchat. Or it could show up as a formal request for additional UX research staff members or an allocation that is temporary or permanent. When more teams want to work with UX researchers, that’s usually a good sign of buy-in for research. Whether to positively respond to these requests is another issue. If UX management topics interest you, I recommend Arnie Lund’s User Experience Management: Essential Skills for Leading UX Teams (2011).
Skeptical stakeholders become believers
One of the best moments as a UX or usability practitioner is when a stakeholder gets it. There’s that spark in their eyes that tells you they believe. It doesn’t happen often, but when it does – especially if those stakeholders are senior decision makers or executives – that spark in their eyes can move mountains for you and for the UX practice in the organization you work for.
The following is a story about such a moment.
What Happened When I Invited Executives to Use the Product?
I worked for a mature startup at the beginning of my career. I can divide my time there into two big parts; before and after a certain usability study. Prior to that study, I was mainly struggling with my stakeholders, especially with executives and the two founders of the startup. They were very opinionated about design issues, big and small. They preferred to rely on their own intuition rather than on research findings and recommendations. The primary product that the company developed has had many design flaws that were consistently overlooked by management. It was very clear to me that they needed to experience what users were going through and I had found the way to do just that.
In the spirit of “show, don’t tell,” I decided to stop talking about UX research and advocating for our users. Instead, I designed a usability study around the primary tasks that our users need to complete with the product. The twist was that I did not recruit users to participate in this study. Instead, I invited members of the company’s executive team to act as users and participate. During the study, executives were not able to use our product and were not successful at completing basic tasks. I did not have a goal to learn anything about the design of the product. I learned everything I needed to learn about the design from past research with product users. My goal was to prove a point. This exercise resulted in several quick decisions. First, the product roadmap was changed and an overall redesign plan was put in place. That result definitely matched my expectations. The second decision surprised me. The executive management team asked me to take the position of product manager for the redesign while conducting any UX research I deemed appropriate. I hesitated for a moment, and then decided to go for it.
Business is changed
The following signs might tell you that the business is changing due to UX research:
Business decisions are based on research findings. Strategic partners are sought, acquisitions are made, mergers happen, and new target audiences are defined. When these things happen after someone made a decision based on user research, that’s a huge indicator of success.
Product roadmaps are reshaped. When product decisions – such as developing new, significant features or investing in more research or stopping a launch of a faulty product – are made based on UX research, you can say that research has made its mark.
New products are born based on research recommendations. Sometimes research points out new opportunities for developing products. One study will not cause people to decide to develop a new product, but a research program combined with market research and collaboration with other people and departments in the organization can make this happen.
Watch my interview with Aza Raskin, cofounder of Massive Health, who was until recently Creative Lead for Firefox. Previously, he was a founding member of Mozilla Labs. Aza says that when decisions are being made, research is being listened to. That’s the clearest sign. Use QR code 113 to access the video, a quick summary of the interview, and Aza’s biography.
Staffing is changed
Two positive signs of UX research effects on stakeholders relate to staffing:
Engineers are allocated to act upon research recommendations. When managers decide to dedicate teams’ or individuals’ time to act upon research, that’s a great success. People can be allocated to fix bugs, solve design issues, implement improved designs, develop new features, and even work on new products – all stemming from research. I refer specifically to engineers because they are the ones who can actually make changes to products. When more designers, product managers, and researchers are allocated to solve problems that emerge from UX research, that is really, noncynically awesome, but engineers are the frontline troops that make things happen.
Job descriptions emphasize UX skills. I’m a great believer in specialization. I think people should know a lot about one thing and a little about many things. But when I see the following requirement appearing on the top of a job description for a product manager in my team, it warms my heart: “Fight for the user. As a product manager, you drive the team to achieve excellence in usability, look and feel, and fit and finish.” This means that your team’s leadership appreciates what UX has to offer and is actively seeking to hire people with a mind open to UX in general, and, more specifically, to research.
Repeated requests are made for UX research training
An important sign is when stakeholders make repeated requests to go through some kind of training that would help them become better at quick research. My experience with these training requests shows that stakeholders usually want to learn the following:
Researchers are recognized
A good word or an award given to a UX researcher individually or as a part of a team is a sign of a mature organization that understands the importance and contribution of research to its operations and product development. Know these “hey-we-launched-X-thank-you-everyone” emails that name all the people who were involved in a certain effort? When UX researchers are recognized in such emails, there’s a good reason to celebrate. It means people see you, and UX research, as an integral part of the team.
Now that you know some signs that your research is being used well, how do you improve the chances of your research being perceived as having an impact? When you plan research activities based on listening to your stakeholders (see Chapters 2 and 3), when you collaborate with them throughout the process (see Chapter 4), and when you communicate results well (Chapter 5), you are on the right track.
Next, I’ll introduce some techniques that will help you track the impact you are making with research.
How to track the impact of research
Have a buy-in tracking strategy
Don’t just sit back, do your job, and assume that people will get engaged because you are bringing such great value with your research. Yes, more and more stakeholders worldwide get it and understand how important it is to conduct UX research and act upon its results. But many don’t. And chances are good that you are or will be working with stakeholders who don’t get it. You should make it a habit to think, plan, execute, measure, and track your stakeholder buy-in level for research. Whether it is taking 25 percent or 50 percent of your time or if your manager does (or should do) it, you need to put time aside for making your research count. Conducting extraordinary, high-quality research is very important, but if it doesn’t count – if people don’t act upon it – it isn’t worth much.
This section introduces some techniques for tracking the effectiveness of UX research in your organization or with your clients. I don’t prescribe all of these techniques for all practitioners in every situation. Feel free to mix and match, develop your own techniques, and use any or all of these. The most important thing to do with these techniques is share their outcomes with your stakeholders. I’m not suggesting that you track your stakeholder buy-in so you can make a case when people ask you how effective your research is. Instead, I strongly encourage you to use these issues with your team or clients. For example, don’t wait for people to ask you what the effect of the study was on the design. Instead, show before and after screenshots to your stakeholders. When you do that on a regular basis, no one will doubt the value of UX research.
Before and after screenshots
Before and after images have a tremendous effect. Ask any advertising manager who has used before and after ads. Ask anyone who’s lost a lot of weight. These images are also extremely valuable as a tracking tool. If you haven’t tracked UX research impact with before and after screenshots, you’ll be amazed at their value. There are two things to be aware of when you use this technique. The first is not to use an “after” image that is too far from the “before” image. The next three examples demonstrate what I mean.
Usabilitynet.org is a website that was launched in 2003 to organize information and resources for usability practitioners. Figure 6.1 is a screenshot of the site’s home page. This is the “before” screenshot. Figure 6.2 is a sample of a redesigned version of the home page, which is based on research findings. The number-one thing to be aware of when using before and after screenshots is that it is not always clear how the changes in the redesign are mapped to research findings. It is up to you to make sure that it is very clear. You can associate the changes with the research by adding callouts or by listing the research findings that affected each design change.
Figure 6.1 Usabilitynet.org: a “before” screenshot.
Figure 6.2 Usabilitynet.org: an “after” screenshot.
The Usability Body of Knowledge (BoK; Usabilitybok.org) is a project led by UPA (Usability Professionals’ Association) to serve as a living reference that represents the collective knowledge of the usability profession (Usabilitybok.org 2011). The project was launched in 2005. The BoK can be considered a redesign or a better version of the Usabilitynet.org website. For the sake of the example, let’s assume this is the case. When you look at the site’s home page (see figure 6.3), you can pretty quickly see that it is very different from Usabilitynet.org, which brings me to the second thing you need to be aware of when you use before and after screenshots. When the delta between the before and after designs is too great, it is hard to understand that this is actually an “after” image. It would probably seem to stakeholders like a completely new design, which in many cases it actually is. Again, there will be a challenge with mapping research findings to the actual design. A better way would be to use only the “after” screenshot and use callouts to track what came from research.
Figure 6.3 Usabilitybok.org: a possible “after” screenshot.
Screenshots with callouts
Probably one of the most effective visual manners for tracking stakeholder buy-in for research is a screenshot with callouts. If your product does not have a screen, you can also use this technique if you take a picture of it (see Figure 6.4). The screenshot can be taken from the version that was evaluated during the study or after study results are implemented. Callouts might point out findings and recommendations that were fixed or not fixed. Figure 6.5 shows an example of both.
Figure 6.4 Using callouts with a picture of a physical product.
Figure 6.5 Using callouts with a digital product.
Screenshots with callouts are a great tool for tracking research buy-in. A quick glance allows you and your stakeholders to understand what was done, what wasn’t, and what should be done next. You can print all these screenshots in color and hang them on the walls around you and your team. This way, you can put a big green checkmark on each callout to indicate that it has been taken care of. That’s a great visual way to follow up on research results.
Screenshots with callouts are a great tool for tracking not only what happened in a study, but also why it happened, as described in the following story. Understanding why something happened usually helps stakeholders make the correct design changes and prevents them from making design mistakes in the future.
Why?
Agnieszka (Aga) Bojko, Associate director, User Centric, Inc., United States
I often see usability reports in which findings are listed only in terms of what happened during test sessions: “Five out of twelve participants did this … Two could not do that … Most said this … Some mentioned that ….” It makes me wonder how much the stakeholders actually get out of reports like that.
I recently talked to an executive at a hardware ecommerce website. “The reason I dropped $50 K on this study was not just to learn about what people do, but more importantly, why,” he said with disappointment as he handed me the report he received from the firm that conducted the study.
One of the main findings stated, “Most participants didn’t realize that they were supposed to create their password on the order confirmation page.” That was it. Several questions immediately came to mind. Was it because of the location of the password creation prompt on the page? Was it because of its location in the process? Was it because of the way it was presented? Or was it due to unclear instructions? However, the document didn’t even entertain any of these possibilities.
The problem of “why”-less reports is even further exacerbated by the fact that stakeholders sometimes explicitly ask for the what with little care for the why. Let’s take eye-tracking studies as an example. If I had a dime for every time I heard, “We just want to know where people are looking” as a study objective, I could buy myself another eye tracker. However, many researchers do what they are told – they determine where participants’ eyes are fixated and present the results as numbers in tables, graphs, or heat maps. Only after the findings reach the stakeholders do they realize that something is missing.
What does it matter that most participants looked for the Submit button on the right side of the screen when in fact it was located on the left? What does that mean? Was it because participants expected to see it on the right? Because there was something else on the right that looked like a Submit button? Or perhaps because they were distracted by something that was unrelated but was also located on the right?
Without articulating potential sources of the usability problems found, usability reports provide very little value, for two reasons. First, stakeholders will not know which design mistakes they should avoid in the future. Second, if the recommendations provided in the report are not feasible, without understanding what in the design caused the problem stakeholders won’t be able to come up with alternative solutions. Therefore, if we want to provide value, it is our job as UX researchers to always present the why with the what.
Spreadsheets
Probably the most popular technique for tracking engagement of stakeholders with research is with a straightforward spreadsheet that lists the issues that were uncovered in research while indicating what is going to be done about them, by whom, and when. This spreadsheet usually serves as a team collaboration tool for tracking the status of research results. This spreadsheet is also pretty useful when people ask to see a list of the most updated research findings and their status. Keep in mind that this spreadsheet is a live, dynamic document that is constantly updated. The trick is to get the information. It’s a very easy task if you are well immersed within your team, more difficult if you are not.
Table 6.1 is an example of what such a spreadsheet might look like.
Table 6.1. Sample research follow-up spreadsheet
You can be creative with using background colors to indicate the status and severity of listed issues, but don’t overdo it. The spreadsheet needs to be straightforward and crystal clear. You can of course add more columns and details such as ease of implementation, whether the team agrees to the solution, and other details. The minimum is probably four columns: solution, target fix date, owner, and status. I have some stakeholders who are not interested in learning what the findings were. They trust me to reach the right solutions and don’t bother reading reports. They only care about what they should do to make things better. Needless to say, I highly appreciate this approach. For this state of mind, a spreadsheet is probably the most effective report and tracking tool. One other suggestion is to not develop this spreadsheet as a separate file. Instead, include it as a table at the end of the research report (if you prepared one). This way, it is easy for everyone involved to find and access it.
Quotes and videos
Stakeholder quotes and videos are a powerful tool for tracking stakeholder buy-in for research. Here are some ideas for collecting stakeholder quotes and videos:
Company-wide or division-wide (if you work in a large organization) meetings at which chief officers talk about UX and UX research.
Key stakeholder reactions, especially first reactions, to research results. Usually the best place to get these is during your presentations.
“Invited” responses to research results. Shortly after you share research results, directly approach key stakeholders face to face, by phone, or in email and ask them how useful they think the results are.
Product marketing materials. These often tend to highlight UX research as a unique selling point.
Stakeholder speeches in professional conferences and at big client events. If you are doing a great job of engaging your stakeholders, they will be proud of UX research and will present highlights of your work externally.
Email updates and messages sent to teams or to the entire company by key stakeholders and executives.
When the quotes include positive mentions of UX research, that is obviously a very good sign that your stakeholders are engaged with research. The fact of the matter is that even if they say the UX of the company’s products is not very well done, it is positive, in a way. It presents opportunities for pitching and conducting effective research with the goal of changing the current situation. It’s what you make of it that matters.
Watch my interview with Johanna Kollmann, Senior UX Consultant at EMC Consulting. Johanna says you know that stakeholders have bought into research when they tell other people stories that came from research. Use QR code 131 to access the video, a quick summary of the interview, and Johanna’s biography.
The primary thing you can do with all of these videos and quotes is collect them in one place. This collection will help you get a better picture of the level of buy-in your stakeholders have with UX research. Nothing about it is quantitative. I can’t really say that if the CEO says UX research is great three times per quarter rather than five, it means you are not doing a good job. It is a qualitative measure that must be combined with many other buy-in signals.
Research analytics
If you use a website, internal or external, that allows stakeholders to view research results or artifacts, I highly recommend that you use a web analytics service to monitor its traffic. Of course, as with every analytics service, you will uncover only what is happening in terms of traffic, not why. To know why traffic is changing for better or worse, you’ll need to add more qualitative approaches to monitoring engagement, such as the ones described so far.
The analytics service will not only tell you the traffic volume but will also help you understand where it is coming from. If your research results audience is international, this information can tell you whether the right people or offices of your company are interested in research. In addition, the time that people spend on your results site pages might be useful in understanding what is going on. Another important metric that an analytics service can provide is the bounce rate, or the percentage of site visitors who bounce away to a different website rather than continue exploring the site they are currently visiting. If you have an 85 percent bounce rate from your research report home page, it means that visitors were interested in the results, and hence accessed the site, but were discouraged from exploring them after they saw the content of the home page. There’s definitely more for you to explore about what you can do better if you identify such behavior.
References
Lund, A. 2011. User Experience Management: Essential Skills for Leading UX Teams Morgan Kaufmann, Waltham, MA.
Usabilitybok.org, 2011. Usability body of knowledge. <http://www.usabilitybok.org> (accessed 06.22.11).
Takeaways
In this chapter, I discussed ways to measure and track stakeholder buy-in for user experience research:
1. If you conduct UX research to drive change, you must develop ways to determine whether change is happening.
2. Pay attention to how your stakeholders consume research results.
3. A very clear sign of success is the allocation of funds to and sponsorship of UX research.
4. If stakeholders care about your opinions, it’s a good sign of trust.
5. When people are assigned to make changes that stem from research, that’s a great positive signal.
6. Use before and after screenshots of product designs to track research effectiveness.
7. Use a simple spreadsheet to track the implementation of research results.
8. Collect responses to research, both qualitative (stakeholder quotes and videos) and quantitative (web analytics for research reports).