The meetings you need to have

There are several meetings you should have on a regular basis. These are meetings that will help you better understand the team you belong to and help you and your stakeholders create relevant research plans together. They also serve another important purpose: they help other team members get to know you. It’s so much easier to collaborate with a familiar face.

A simple planning artifact

Assuming that people actually want research to happen, and that you know how to make the research fit properly into product development schedules, I recommend that once a quarter you devote about a week for planning studies for the next quarter. Meet people, ask questions, listen, think, and then create a draft research plan for the upcoming quarter. Share this plan with key stakeholders, collect their feedback, and then publish the plan. This plan will change during the quarter. Don’t make a fuss about it because every plan eventually changes. The key is having a plan to change from. To communicate the quarterly plan throughout the process described here, use the simple artifact demonstrated in Figure 4.2.

Figure 4.2 Quarterly UX research planning artifact.

Notice the following aspects of this planning artifact:

I use this artifact when I communicate with stakeholders about quarterly plans. Sometimes I print it out and hand it to them, especially when it changes.

Agreement and buy-in per study

All these planning meetings and artifacts should not prevent you from making sure that you still get ad hoc agreement and buy-in per study. The fact that you planned an eye-tracking study with the director of product management doesn’t mean that product managers will agree to change engineering priorities or fix certain designs following your study. You need to go through the process described in Chapter 2 (interview your stakeholders prior to planning a study to better understand the product at stake, its users, the study goals, the expected schedule, and expected actions following the study results). You already have a master plan – that’s your quarterly research plan – and before each study you just make sure everybody is on board with it. You cannot expect every stakeholder to be well versed on the quarterly research plan. They have other burning priorities. It’s your job to provide appropriate reminders.

After planning is done – and sometimes even shortly prior to that – you need to start recruiting participants for the study. Recruiting is a tough task, but there are people in your company who can be extremely helpful with participant recruiting.

Who can help and when

Recruiting the right participants for a user experience study is a challenging task. The task becomes even more challenging when there is a need to recruit users from special groups rather than the general population. For example, it is easier to recruit people who use email than people who use accounting software or an EKG machine. Even with special populations, there are various levels of recruiting challenges. The recruiting task is usually harder when the specialized user has an account manager assigned by your company. For example, it’s easier to recruit users of project management software (such as MS Project) compared to recruiting users of a customer relationship management (CRM) product (such as those of Salesforce.com).

My point is that the more challenging the recruiting, the more you will need other people’s help. I do not intend for this book to specify the ins and outs of recruiting. Read Nielsen’s comprehensive report about recruiting study participants (2003).

When it comes to recruiting specialized users who have an account manager from your company, people in three primary functions can help you: account managers (salespeople), technical support or services people, and product managers. All of them meet and talk with the users you are looking for on a regular basis.

Asking for account managers’ help is something you should do with extra care because some of them might be sensitive to a situation where someone else approaches “their” clients with different requests. Appendix A (use QR code 142 to access it) includes two practical tools for connecting and communicating with people who can help you recruit study participants.

Why bother? I can do without them!

When the company’s clients have account managers (or salespeople) assigned, you might get the feeling that salespeople are protective of the entire communication with customers. What seems protective to you is not at all protective. When Sales insist you need to go through them to contact a customer, there is a very good reason for that. Customers can be difficult to handle, especially those who pay your company a lot of money. These relationships might get extremely delicate; sometimes one word said in a tone that is perceived in the wrong way might jeopardize million-dollar deals and long-term relationships. Salespeople are aware of this. You might not be. When they want to control the communication with customers, they want to make sure that customers understand who is asking what, for what reason, and how this might affect the relationship. Let salespeople do what they’re best at. So when they tell you not to talk to customers, or that they will recruit for you, or that you only need to tell them what you want – you need to take a leap of faith and trust them. It might not work smoothly at first, but it will in time. When you and salespeople learn to know and trust one another, things will get a lot better, and recruiting study participants might not be such a tough task.

How to ask for help

When you meet someone who can help you recruit study participants, it’s always best to meet him or her in person, if possible. Having that face-to-face aspect always improves communication. When you meet in person, you can tell very quickly whether the person you talk with is going to assist you. It’s very similar to the difference between conducting research face to face compared to running remote studies.

I always start my recruiting discussions with sales managers by asking them if they are interested in having me share recent research findings with their teams. I offer a couple of recent studies that I assume will be helpful for salespeople in their client meetings – for example, if I found that the product is working extremely well for our users in a certain aspect or if I gathered data that compares our product to the competition. Generally speaking, studies that evaluate designs are usually not interesting for salespeople. Because I realize salespeople should spend their time selling, I offer to share study results in the already scheduled team meetings. I also commit to presenting for no more than ten minutes plus an additional five minutes for Q&A.

Only when I learn how I can help salespeople by giving them ammunition for their sales pitch do I turn to asking how they can assist my efforts to improve the usability and usefulness of our products. I ask for recruiting support while specifying the audiences I am interested in. Salespeople have tons of experience with clients, so I ask them for their opinion about the audience I’m looking for in the study at hand. I always hear interesting views.

What I have found to work extremely well with sales teams is the following: meet with the sales manager and discuss his or her needs as well as yours. Then, ask him or her to walk with you to where the sales team is located in the office. Some sales managers will be perfectly okay with that. Walk around with the manager, introduce yourself to salespeople, and quickly explain what you are looking for, and they will immediately respond with potential participants. That’s good enough for you at this point. You’ll follow up with them later on for the details.

Salespeople are very busy people

Salespeople should sell stuff and make customers happy. They are the company’s front line of success. Their success creates the company’s success. Think about it: salespeople have pretty much the same goal as you do. You also want to make people happy with useful, usable, and beautiful products. Time not spent selling the company’s products is perceived as lost time in the eyes of sales leaders, and rightly so. On the other hand, if you are not able to recruit participants for a study, product development teams cannot benefit from user insights and cannot create those useful, usable, and beautiful products. The middle road for mutual success is mutual respect for each another’s discipline. I’m not going to discuss how salespeople should behave because you cannot control that. But you can control your own behavior. Here are a few things that will demonstrate to salespeople that you are respectful of their time:

When recruiting is done, it is time to go to the field or the lab and get all that data you are looking for. To me, interacting with study participants is the best part of my job. Why not share it with others? Especially if it helps persuade them to act upon what you find together. The next section discusses ways to involve stakeholders in field and lab studies.

Stakeholders in field studies

I consider it a personal failure if I go on a field study session by myself. It seems to me to be a complete waste of time. What’s the point? I always have company, be it a product manager, engineers, designers, other researchers, salespeople, technical writers, support people – you name it. If they want to join, I let them. If they don’t, I make them feel like they are missing an invaluable experience. I might even reschedule sessions to accommodate stakeholders’ availability.

It is a norm in our field to say that no more than three people should go on a field visit: a researcher and two stakeholders. The primary reason is to not make the participant feel overwhelmed with too much of an “audience” observing them. But what if they are perfectly fine with it? What if observers are well behaved? I say it’s definitely possible and even recommended having five people attend a field visit. Sometimes the office space is way too small to host even two people, and the study must be done in a meeting room or a bigger room. This is a great opportunity to bring more stakeholders to the field. Yes, my first priority is the comfort of study participants. If they feel uncomfortable, I am the first to notice and take care of it – even at the expense of a stakeholder. But my number two priority (very close to being number one) is my stakeholders. After all, if they are not engaged, nothing will come out of the study. So I do my best to have it both ways. I have never had an issue with it. It all depends on setting the right expectations with participants, their managers, and stakeholders.

The following story, from Israel, talks about something that has probably happened to anyone who has ever invited a salesperson to join a field visit – dealing with salespeople who try to defend the product when the user is providing feedback about its drawbacks.

One special case is a field study in a foreign country. When this is the case, and if the company you work for has the budget for it, invite even more stakeholders. As soon as they accept your invitation, they become a part of the study team. Delegate responsibility areas and get their commitment to become fully engaged in this research project. It gives me great pleasure and pride to see how people get involved and engaged in these situations. These stakeholders immediately become champions and ambassadors of UX research. I remember one engineer, shortly before we departed for an international field study, telling me that he never realized that research is such hard work. I consider these as defining moments in my career. Seriously.

Developing empathy for stakeholders is important for establishing and strengthening trust. The following story highlights a minidrama that created that bond between researchers and a stakeholder.

I am always amazed to find out that many people who write code have never met a person who is using their code. When I realize that’s the case, I suggest doing something about it. Basically, it means getting people up from their chairs and monitors and creating an opportunity for them to observe and interact with real users. When budget and time are scarce, I call these “virtual visits.” A virtual visit is my humorous name for a phone interview. I call a user, put the call on the speakerphone, ask a bunch of stakeholders to join me, ask the user a series of questions, and end up letting stakeholders ask their own questions. I repeat the same questions in every “visit” and publish short reports to the entire team (use QR code 142 to access Appendix B to see the kind of questions I ask). Once in a while (usually once a quarter), I look at these summaries to see whether some trends are repeating and whether it’s worthwhile pitching a product change. Even if nothing comes out of these visits (which rarely happens), they are precious merely because they make people realize that users are real. The questions, by the way, are basically there to better know the user, his or her environment, and company (if relevant), as well as to understand current pain points, delights, and wish list items: basic, yet extremely effective and informative. Another way to achieve these things is to hold in-person visits once in a while. I adopted an idea called “Field Fridays” from a colleague. Once every week (or two, or three), a group of software engineers and I go on a 90-minute visit to a customer office. The goal is to learn by observation and contextual interview, as well as answering customers’ questions. It is extremely beneficial to both participants and stakeholders, and this whole initiative is always highly appreciated by higher-ups on my team. That’s a classic win-win situation. I highly encourage you to try it yourself.

One thing I learned about my work is that it is extremely important to share the knowledge with team members and stakeholders who have not yet had a chance to join a virtual or in-person visit. For that, I open an internal blog where I make posts that summarize these visits. This way, other team members learn about what happened and have a chance to ask follow-up questions. It also raises interest in joining future visits. Following is an interesting example of a comment one software engineer added to a blog post for after a Field Friday visit:

The one thing that really stood out for me was how embarrassed I was to see how bad our product was and how awkward it was for me not to be able to give good answers to any of the client’s good questions.

Stakeholders in lab studies

I almost always invite stakeholders to join me in the study room. The only exception is when I expect too many of them to show up (more than three or four people). When this happens, I ask them to watch from the observation room. When I do invite them to join me, it is usually for remote study sessions or phone interviews. I make sure they are properly informed about dos and don’ts. So they know it’s not appropriate to comment on participant behavior, giggle, or even use certain body language. I try to make them aware of things that bias study participants or make them feel uncomfortable.

Additional aspects of inviting stakeholders to attend a lab study:

The following story shows how effective it is to have senior stakeholders observe several usability study sessions and how it can win them over.

Figure 4.3 Stakeholders observe a remote usability test by Bolt | Peters (printed with permission).

The following example demonstrates an effective way of documenting stakeholder observations during studies that helps establish and maintain trust between team members.

Sometimes stakeholders want to (or have to) moderate lab studies. It can happen for different reasons. They might not have a budget to hire a researcher, or the in-house researcher might not be available. With little training, these stakeholders can get better at planning, moderating, and analyzing quick usability tests. Use QR code 140 to watch Eric Ries, the father of the Lean Startup movement, talk about how his startup ran usability tests. He talks about it so naturally, and he was the company’s CTO at the time! It’s so obvious to him that this is what they had to do. To get a better perspective on roles and responsibilities of UX researchers and their teams, I also recommend that you read the controversial article “Surviving Our Success: Three Radical Recommendations” (Spool 2007). Jared’s second radical recommendation to UX people is to stop conducting evaluations such as basic usability testing. Instead, he claims, engineers and product managers should run them so that it becomes a part of the organization’s culture. It is just something everybody knows how to do.

Sometimes stakeholders conduct their own research and meet users outside of the lab. Like it or not, this is what they do. Sometimes they are even directed to do so on an ongoing basis. The next section discusses your role in this.

Help stakeholders interview users and launch surveys

I am perfectly fine with stakeholders conducting their own interviews with users, with or without my attendance – not that anyone needs my approval for this. I am aware that some of my colleagues are not comfortable when product managers talk with users. They think that product managers are asking the wrong questions and that in many cases they ask leading questions that result in proving their own preinterview opinions.

I’m a big supporter of user interviews conducted by stakeholders. It’s great that they do that. In many cases, they find what I find or complement what I find in studies I conduct. I can’t understand why people fight it. I offer my help to stakeholders. I ask if they would find it useful if I reviewed the list of questions they are planning to ask. I suggest asking more questions, I suggest asking the same questions with different angles or phrasing, and so on.

As I see it, the best help I can give stakeholders for interviewing users is exposing them to the behavior versus opinion issue. I give them the don’t-ask-about-future-opinions-ask-about-specific-past-and-current-behavior spiel. I say something like this:

When you interact with a user, I have one important suggestion. Instead of asking about their opinions about something, ask about specific behavior. Or even better, observe behavior. The thing with opinion is that we as humans are kind of weak at predicting the future. When you ask a user, “Will you need feature X?” they’ll give you an honest answer. The problem is that you and they have no idea if this answer is true or not. Instead, I always prefer asking, “In the past three months, did you ever want to do X but couldn’t?” Then I ask them to elaborate. This way I get an answer that is based on facts that actually happened, rather than opinions that try to predict the future. I must tell you that I am not perfect at this. Sometimes I slip and ask a question in the wrong way, but then I catch myself and try to ask it in a different way. Think about it the next time you interact with a user.

Some of my UX research colleagues are not supportive of this approach. They claim that humans are weak at retrospective analysis and that they tend to “beautify” reality when you ask them about their past behavior. I can definitely understand where they are coming from; in fact, I have seen this happening. To address this problem, I do two things. First, I don’t ask about more than three months ago, so people’s memory is still somewhat fresh. Second, when users do share an unmet need, I probe a lot and ask for the context and detailed reasons. So if someone is trying to rationalize their behavior, I can identify and ignore it.

Sometimes your stakeholders will prepare surveys to be given to users. Survey design is not to be taken lightly. There are many important aspects to a survey that can cause its results to be either very helpful or extremely misleading. I have seen stakeholders initiate many survey projects with users – mostly in professional services and support departments but also in marketing and product management. Two primary issues occur with these surveys. First, the sampling strategy that was picked may be mostly a sample of convenience. This means that whoever is agreeing to take the survey is picked to participate (Tullis & Albert 2008). Second, the phrasing of the questions may be problematic. Many of them are leading or too complicated.

In many cases, I hear about these surveys after the fact. When I was a young practitioner, I used to get annoyed by these and try to explain to my stakeholders why they were doing a bad job and why they should give me the responsibility of running such surveys. Now I know that this was the wrong approach. Nowadays, when I find out about a survey that has already been fielded, I try to get as much information about it as I can. Mainly, I try to get access to the survey questionnaire, the complete set of raw data responses, and the survey report. I ask to meet the person or people responsible for it and I ask them more questions about what they learned. I learn from all of these about things that can help my research. I identify opportunities to complete it with additional research. I see it as an opportunity, not a personal failure that I wasn’t informed about it in advance.

When I learn about a survey initiative prior to its launch, I offer my help. I first try to figure out the motivation for the survey and its goals. If I estimate that a survey is a wrong choice for a practical methodology, I suggest not conducting it. I usually try to be constructive and suggest an alternative. For example, when a survey is planned to find out why users behave in a certain way, I recommend conducting interviews with users instead. I also offer my help in conducting these interviews.

When I believe that a survey is the appropriate methodology for the intended goal, I offer practical help: I help phrase questions, I help with general survey tips, and I provide suggestions for sampling techniques. I give full credit to the stakeholder who initiated the survey. I don’t look for glory and recognition. My goal is to help my team do a better job and make them trust my skills. If they continue to conduct surveys on their own while consulting me, I am perfectly fine with it. If they want me to lead the survey effort, that’s also fine. I support the approach that has more chances of affecting decisions, product roadmaps, and designs. If this means that I need to take a step back and leave the stage for a stakeholder to run a survey, so be it.

The field visit debrief is a huge opportunity

When I go on a field visit with stakeholders, I always conduct a 15-minute debriefing immediately after the session. I have conducted field debriefings in many places – building lobbies, restaurants, subway trains, cabs, public parks, and street corners. What I try to say is that it is such an important opportunity to analyze the results of a study that I never skip a field debriefing. The reason relates to coloring the experience. Things are still fresh in the minds of stakeholders who attended the visit, and they didn’t have a lot of time to process and digest what they have just witnessed. This is a perfect time to add a broader, team-oriented context to everyone’s observations so that we can later draw a more balanced set of inferences. My job in these debriefing sessions is to note the things worth paying attention to, make connections to things we’ve already observed or heard in prior sessions or past research, and make sure that everyone sees what happened in the full person-task-environment context. I also take this opportunity to begin the analysis of findings by discussing four topics (IDEO 2009):

I take good notes on this short discussion and use them later when a more formal analysis is conducted.

Use the KJ technique

Contributed and written by Jared Spool, CEO and Founding Principal, User Interface Engineering, United States

Back in the late 1970s, the US government commissioned a study to look at effective group decision making. In the study, they asked 30 military experts to study intelligence data and to try to construct the enemy’s troop movements.

Each expert analyzed the data and compiled a report. The commission then “scored” each report on how well it reported the actual troop movements. They found that the average military expert got only 7 out of 100 elements correct.

Each expert then reviewed all of the other experts’ reports and rewrote their initial assessment. The average accuracy for these revised reports was 79 out of 100.

What was different between the first report and the second? The experts didn’t have any new information. All they had were the perspectives of the other experts. When they added those perspectives to their own, their accuracy increased tenfold.

Conduct workshops

Workshops are an excellent opportunity for synthesizing observations made in a study and for obtaining team consensus about future actions. Workshops take various shapes and forms. They usually last between one and a few days. Many workshops mostly focus on exercising the KJ method, but there are also other techniques. One workshop I led recently had the following structure:

I especially favor this structure because it treats stakeholders as equal members of the study team. Yes, people’s writing skills are not homogeneous, but I think it is a less critical factor than the collaborated effort. Editing can be done later in the process. In my opinion, creating a feeling of belonging to a learning team that comes up with insights from users is more important than grammar or high level of writing. When a study team is working together on analyzing results, each and every person – be it a product manager or software developer or a UX designer – becomes a champion for acting upon the study recommendations. Other workshop structures might include a couple of days when the team first discusses users and their issues, then identifies emerging themes from the study, and finally comes up with possible action items for product, more research, and design.

There are several guidelines you should consider when you conduct workshops. These should be communicated, discussed, and agreed upon by all workshop attendees and contributors:

Figure 4.4 UX practitioners and their stakeholders during a workshop held at a Google office in Brazil (printed with permission).

The following story from the United Kingdom demonstrates the power of stakeholder workshops. The researcher used personas as a tool for communicating user research findings during a workshop with stakeholders.

The following story from Germany shows how workshops that are designed to have stakeholders get excited collaboratively about research and design can reduce tension and disagreement within a team and create a fun work environment.

The next story, coming from Scotland, showcases a participatory design workshop that had a rough beginning and a successful ending.

Do not report recommendations

When I sense that stakeholders are more likely to implement their own conclusions from study findings, I sometimes choose not to share my recommendations. Here is a step-by-step description of what I do:

I have heard about UX practitioners who don’t even get involved in such meetings. They merely provide a findings report and let others deal with it. Although I see some advantage in this approach, I think that getting proactively involved increases the chances of somebody actually doing something about what was learned in the research.

Finally, we’ll discuss ways to report results jointly to other stakeholders. Benjamin Franklin once said that we must all hang together, or assuredly, we shall all hang separately. Let’s see how you write a report together with stakeholders, present together, and share your gratitude in an honest and public way.

The joint report

I’ll discuss reports in further detail in the next chapter. The purpose of this short section is to state how important it is to create a joint report for getting stakeholder buy-in for UX research. When a product manager, a research and development (R&D) manager, a designer, and a researcher appear as the authors of a UX study report, it is much more powerful than having only your name on it. And I don’t mean crediting people for work that they weren’t a part of. My point is that the people who sign on a report are the ones who were deeply involved in the study planning, execution, or analysis. Needless to say, when a stakeholder signs a report, he or she is more likely to act upon its recommendations.

The joint presentation

When your stakeholders and you collaborate on a UX study, there usually comes a time for presenting the results to others. When that time comes, invite your stakeholders to take an active part. Ask them to brainstorm on the presentation content – its outline, its emphasis, and the stories you will tell. Delegate meaningful chunks of the presentation to them. Ask them what they are comfortable working on and let them work on those parts. Then ask if they are interested in actually presenting these results. If there is one thing I’ve learned about presenting together, it’s that your stakeholders have tremendous persuasion power – a lot more than you have. Imagine that you are communicating about a problem you’ve observed and that the solution requires allocating significant resources. When you communicate this in a presentation to, let’s say, software engineering and product management leaders, it’s one thing. When one of them presents the same thing, it’s much more convincing – especially if that person was deeply involved in the study and is entirely aligned with the recommended course of action. Hence, consider the chunks you hand off to your stakeholders. Don’t ask them to talk about who participated in the study or about the methodology. Instead, ask them to present the sensitive stuff. Ask them to talk about the most important findings and suggest the solutions. When they present, provide your full support with spoken and body language (lots of positive nods).

That’s powerful stuff. Try it out.

Maintain an internal blog

Earlier in this chapter, I discussed launching a blog for collaboratively documenting Field Friday visits (see the earlier section “Stakeholders in field studies”). You might want to broaden your approach and consider launching an internal blog that covers research studies and findings, inviting stakeholders to participate – read, contribute, and debate. Let every relevant team member have full writing access and do not censor blog posts in any way. Invite stakeholders to comment on your posts as well as on other people’s posts. Ask people to post about interesting things they observed during studies, debate controversial results, and add pictures from studies and even video highlight clips. Make it a lively area where people exchange opinions about UX research. Make it everybody’s research blog, not yours. Consider it a way of sharing results along with your stakeholders. To make it easier, find at least one person – a software engineering leader or a product manager – who is willing to actively collaborate when you first launch the blog. After things run successfully for a while, invite others to join.

Provide appropriate credits

When people support a study in any way, big or small, publicly thank them. Recognize the importance of their support personally and in front of other people. If you don’t provide this recognition, you are unlikely to get help in the future. I provide credit in these three opportunities:

I know it’s obvious, but people look for recognition. Even if (in your opinion) they are just doing their job by supporting a study, they deserve to be recognized. They expect it. And you, of all people, should be extremely generous with providing this recognition. It’s one of the things that help you get more buy-in for UX research, and it’s the right thing – and a nice thing – to do.