Chapter 2
Mmm … Interesting; so what exactly is it that you want to learn?
Implementing your great participant interviewing skills on stakeholders: asking good questions, listening, saying the right things, and identifying research opportunities
Chapter 2 is about identifying research opportunities by developing empathy toward stakeholders. The chapter introduces the most important questions to ask your stakeholders as well as tactics for handling research requests and delaying early methodology discussions. It also discusses ways to become a better listener and thinker.
Joshua Ledwell
Single most important requirement for successful #ux team is senior managers who “get it.”
Introduction
As someone who studies design and user experience, one of your great skills is interviewing people to understand their behavior, needs, and opinions. Some practitioners out there wear their researcher hat only when they interview users of products and services. Others wear this hat all the time. This chapter demonstrates how to implement your awesome interviewing skills with your stakeholders, especially when you are identifying research opportunities and kicking off research projects.
Other lessons of this chapter are:
Initiation of a study
UX research begins when either a stakeholder asks for it or a researcher suggests it. When a stakeholder asks you to conduct a study, it is a good sign. It’s good because the stakeholder realizes that answers to his or her questions begin forming with the people who will use the product or service. Having said that, I am sure you have received requests for studies that did not make sense to you. Here are several such requests I have gotten:
[Using usability testing as a sales pitch] Can you go to this prospective client and run a usability test with them? We want to show them we do this kind of stuff to make them want to buy from us.
[Integrating terminology from marketing research and usability] We need a usability testing focus group.
[Picking the wrong methodology to answer a legitimate research question] Can you ask a few of our users which features they use mostly?
Sometimes I get different requests, which I deeply appreciate:
Can you help us prioritize the features we are developing?
We want to prevent usability problems from happening. We have some sketches we drew on paper. Can we get user feedback on them?
We are going into a completely new market. Can you help us figure out what people in this market need and how they compare to markets we are already in?
Sometimes I don’t get any requests. Sometimes I am the one suggesting that a certain study should be done. I can do that because I constantly listen to my stakeholders. I don’t care if they don’t invite me to an important meeting once in a while. I proactively search for opportunities to make an impact with user experience research. On the road of converting my understanding that a study is needed into an actual study, I sow seeds by constantly looking around, identifying whom I should talk with, what to ask, and how to listen.
When there is a study request on the table, I immediately ask for a short meeting with the person who requested a study and additional relevant stakeholders and ask a series of very important questions.
The most important questions to ask your stakeholders
People initiate studies for different reasons. The first thing you want to do is set the right expectations with the person who requested a study or wants one (if it was you who suggested it). I cannot stress enough the importance of starting off a study on the right foot. The quality of your study’s beginning will have a huge impact on its results. As soon as you understand that there is interest in conducting a study, schedule a 30-minute meeting with your immediate stakeholders. These could be the product manager and the lead software developer, and it’s always a good idea to also invite the lead designer and someone from sales.
These people are needed for the following reasons:
Product managers: Because they are aware of the business goals for the product and have a clear understanding of priorities. Product managers can help identify the right audience for the product and the characteristics of study participants. They’ll also be leading the development timeline and implementation for any changes that might come up from the study.
Software developers: Because usually they are the ones to be most influenced by research results. They will actually make changes to the code based on study findings.
Designers: Because they are your partners in getting things right. Designers need to understand users in the most profound, deep way, and you need their help in making changes to the design. They probably are also the ones providing you with the wireframes, mockups, or prototypes you’ll be using during the study. If the study is more on the generative, discovery side of things, designers are great at capturing critical observations.
Salespeople: Because they are in close relationship with customers and users. Salespeople can help understand the audience for a product and shape the participant criteria for a study. They might also be a critical part of recruiting study participants. The sooner they are aware of a user study, the better they can prepare themselves to help with recruiting.
The meeting you’ll be holding should not come as a surprise or as a pop quiz to your stakeholders. Include your short agenda as a part of the calendar invitation. Ask your stakeholders to put some thought into the study and come prepared to discuss it:
Meeting title: UX study prep
Meeting description: Here is a short list of questions that relate to your recent UX study request. During this meeting, we will discuss them briefly. Please take a few minutes to think about these questions prior to the meeting. Thank you!
| The most important questions to ask your stakeholders in the beginning of a research project. |
| What is the product? |
| Who are the users of the product? |
| What do you want to know? Why? |
| When do you need the results? |
| What will you do with the research results? |
Watch my interview with Gerry McGovern, the founder and CEO of Customer Carewords, from Ireland. Gerry, widely regarded as the number-one worldwide authority on helping large organizations create more customer-focused websites, calls researchers to first conduct studies on their bosses. Use QR code 135 to access the video, a quick summary of the interview, and Gerry’s biography.
Let’s drill down into each one of these fundamental questions.
What is the product?
Sometimes you are so involved with the team that you do not need any product introduction. Skip this question if this is the case. Beware, though: sometimes you are involved with the team, but people forgot to update you about a new feature they are working on, so you do need to ask this question. In any case, I suggest you ask this question or some version of it. For example, “We are talking about product X, right? Anything new I should know about it?”
If you are not familiar with the product at stake, ask for a short product pitch: “What is it for? What does it do? Is it launched yet? Why was it developed? Which user needs does it meet?” Try to keep stakeholder answers less about technology and more about the business of the product. Don’t get into too many details. At this point, you need a high-level pitch only. You’ll get all the necessary details later when you prepare the study plan.
After you get the gist of the product, it is time to find out who they think is (or will be) using it.
Who are the users of the product?
A good sign is an answer to this question that generates a discussion. It means that someone at some point actually thought about the people who are (or will be) using the product. If there’s a debate about the characteristics of these people, it’s okay. The discussion should lead to a decision about which participants are to be invited to the study.
So ask them, “Who are the users? Are there different groups of users? Why?” Try to lead the discussion toward deciding on a group of people with clear characteristics. If you notice that several groups of people with different characteristics come up, that’s natural and probably okay. Don’t push too hard to come up with one group at this point in time. You might get back to that issue later in the study planning process. Now you need to make a quick decision. As they answer, consider whether it will be hard to recruit this group of people. If it seems hard, ask your stakeholders to come up with alternative characteristics or with a different user group just in case it is extremely challenging to find the people they want for the study.
One special case for the “who are the users” question is when that is the research question. It usually happens when the answer is something like “this is what we want to find out.”
Now that you know the “what” and the “who,” it’s time to talk business and discuss the core reasons for this study.
What do you want to know? Why?
Ask about things that seem related to the study request. “What do you want to learn by conducting this study? Why do you think you need this study? What are the broad and specific business goals for the product?” Try to find the reasons your stakeholders came up with their request. If you suggested the study, you are the one answering these questions.
At this point, when you have an answer, you might be tempted to think about a research methodology that would help close the knowledge gap you are identifying. Try not to think about that just yet. Focus on listening to your stakeholders and on gathering information. You’ll think about an appropriate methodology later. Take good notes on the discussion. This step is critical for coming up with accurate research questions.
You might get an answer such as, “We want a usability test because we want to learn whether users like our product.” You may be tempted to observe that there are three challenges with this answer:
1. The stakeholder names a research methodology. They ask for a usability test. Best practices are to first define research questions, then match a research methodology that is known to answer these questions.
2. The stakeholder chooses a wrong methodology for their question. The levels at which people like a product or services are not usually determined in a usability test.
3. The stakeholder statement includes a built-in bias. They assume that users like the product and they wish to prove their hypothesis right.
Some researchers might respond to such a stakeholder request by explaining why it is wrong. I would strongly recommend that you don’t say a thing about it. The reason is that you are now trying to gather enough information that allows you to come up with a coherent study plan with crystal clear research questions. Your research questions will make the argument for you later.
Timing is everything. The next critical topics to discuss with your stakeholders are time, schedule, and deadlines.
When do you need the results?
To set better expectations, I always ask my stakeholders when the latest time is that they need the study results so they are relevant and effective. The answers I often get might surprise you:
Tomorrow. Bad surprise. In some cases, you can be flexible, but in many cases, you just can’t. When you get this answer, it’s an indication that the stakeholder doesn’t know much about user experience research processes and that you need to put some time on both your calendars to close this gap. Now, don’t get me wrong. I’m not saying research can’t be done fast. I’ve sometimes received requests for studies in the morning and provided answers in the evening of the same day after running to the nearest mall to recruit participants for a one-task, ten-minute café study (a low-cost, fast-turnaround usability study usually conducted in cafés, malls, conferences, etc.). Usually, when you provide a fast turnaround, it means that you are cutting some corners. Think about it before you say yes to a “Tomorrow” study. Sometimes cutting corners is perfectly fine. In other cases, it is inappropriate, unacceptable, and plain wrong.
Next quarter. Good surprise. You have lots of time to plan the study, gather great feedback, recruit the right participants, analyze, and communicate the results. Just be careful not to commit to too many “Next quarter” studies for the same quarter. And please don’t take me too literally. “Next quarter” means that there is time for you to conduct the study without cutting corners. It might mean you’ll be asked to provide results within one month.
No idea. An unknown surprise. This is a trick answer. They might need results next week or next month or next quarter. They just don’t know yet. They will know when X happens. Hear them out, but be sure to let them know you might not be available to conduct the study. This kind of reply on your side might make them feel they need to provide a deadline.
The timing issue can bring a whole study project down or change it completely. An A/B test might change to an eye-tracking study or a traditional usability test or a quick café study, depending on the deadline set by stakeholders. In the same way, a field study can turn into a series of in-person, in-depth interviews or phone interviews or a quick survey.
Now for a question that I do not always feel comfortable asking but that I have learned is extremely important: what’s going to happen after the study is completed?
What will you do with the research results?
In many cases, as a researcher, you need to make tough decisions and set hard priorities. You also need to decide which team or product needs research love. Sometimes you are not the one making those decisions, but when you do, you want to be sure you are making the right ones.
One of the most important things to ask stakeholders who request a study is what they are going to do with the results. At first, it might seem like a rhetorical question. Of course we will do something with the results. We are asking for this study, aren’t we? Well, it’s not that simple, and it’s definitely not that clear-cut. Every stakeholder enters a user experience research project with a set of expectations. They expect participants to be able to do this but not that, they expect a certain concept to be less understood than others, or they expect people to have one need rather than another. Now comes research that (hopefully) provides clear answers. They can do this, they don’t understand that, and most of them need a feature like this. In many cases, results are in conflict with what stakeholders expected. Some of them realize that their expectations were wrong. Others choose to ignore your findings. This is why you ask this question.
When you ask, “What are you going to do with the results?” you are actually saying, “I might choose not to work with you if you are not eager enough to act upon future study findings.” There are two types of answers you might get to this question:
1. “Obviously, we will act upon the findings.” It’s not a green light, but it’s definitely not a red light, either. They might still ignore the findings, but at least they are open to acting on what you find.
2. “Well, it’s complicated; it depends on what you find; we’ll see.” This is not a red light, either, but it’s a warning sign. Beware. Sometimes, when I think it is going to help, I immediately reply that in this case I will consider whether I should invest time in this study. They usually change their answer at that point. If they don’t or if you choose not to reply aggressively, there’s another approach I like to use: I offer to not provide recommendations after the study, only findings. I suggest that we agree together on what should be done about what we find during the study. It usually helps. Green light.
Bonus question: What do you know now?
Sometimes stakeholders internalize study findings so well that they think the research was redundant. They say they already knew what it had uncovered. They say they don’t understand why research was needed at all. It’s an extreme, rare situation that happens mostly with studies that have a goal to uncover user needs and wants. This is a dangerous situation for you, as someone who conducts studies. It is dangerous because it’s a signal they won’t be asking for your services when they wish to learn about the needs of users. They’ll count on what they find, or feel is right, or think is just working, or learn from salespeople. Intuition sometimes works, but not always. You know that in many field studies, you learn about what people really need and don’t need. You know that when you visit people at their homes or workplaces, you see things no one can have an intuition about. When I go on a field visit, the first thing I look for is what people hang on the walls that surround them. In many cases, they create things that are missing in the product at stake. When stakeholders say they knew what users need even before the study, it’s a sign that you need to do a better job in interviewing them right from the start.
To prevent this situation from happening, you might want to ask your stakeholders one simple question. Because by now you already know what they want to learn during this research project, all you need to do is ask them what they already know. Document their knowledge prior to the study (see Table 2.1). It’s as simple as listing the questions they want answered and having them answer them.
For some questions, they’ll just say they don’t know the answer and that’s why they ask for this study.
For other questions, they’ll try to guess which answers the study will bring.
And for a few questions, they will have a detailed answer based on their current knowledge. They might give you two answers saying something such as, “Big boss 1 thinks users need X and big boss 2 claims users want Y. We expect your study results to determine what we should do.” That’s perfectly fine. You’ll probably discover that none of the bosses thought of Z.
Table 2.1. Documenting stakeholder knowledge in a table prior to a research project.
| Research question | Stakeholder answer | Stakeholder name and role |
| How fast do users complete the sign- up process? | “I don’t know, that’s why we run this study, right?” | Jeff Harrison, Product Manager |
| “I’m not sure. I guess users can complete it in 30 seconds.” | Laura Bizbi, Lead Engineer | |
| “My manager thinks users should complete it in 15 seconds, but I heard that the VP of Product Management thinks they should first understand what they sign up for and that they can complete it in 4 minutes. We expect the study results to tell us who is right and what we should do.” | Chelsea Bronfman, Sales Analyst |
After you document what they know and don’t know, save this file and put it somewhere you’ll be able to find later, if needed. If after you present your findings or even after a field visit, a stakeholder claims she learned nothing because she already knew that, find this file and make smart use of it. I realize you might be asking yourself, “What am I, a divorce lawyer?” No, you’re not. You are a person who brings immensely useful information to the organization. This is a smart technique that sets you up for success. And that helps to ensure that people remember your contribution, which is business development for future engagements. Don’t act as a divorce lawyer, though. You don’t want to be perceived as confrontational. When you have proof that your stakeholders were completely (or even slightly) off prior to the study, make your case in a positive manner. You don’t have to be aggressive to make your point. You just want to gently remind them why they asked for this study, what they said they knew before it began, and how different their answers are from what was found. For more discussion about techniques for communicating results, see Chapter 5.
Now that you know the most important questions to ask your stakeholders, let’s talk about more techniques and best practices to use during these initial conversations with your team or client.
Delay any discussion about methodologies
At this point in your research project, your stakeholders and you should not be thinking about the research methodology. One of the things that annoys user experience researchers and usability engineers is when their stakeholders ask for a certain methodology prior to putting any thought into what they want to know and why. If you’ve run a few studies in the past, you probably know what I am talking about. They might ask for a usability test when they actually need a survey. They might want to be really confident with the results while asking you to interview only five users. They might ask for a focus group to test a design. They might also want to show a strategic client who threatens to leave that they care, so they ask you to run a usability study with them. These requests indicate different knowledge levels that your stakeholders have about research methodologies. Your stakeholders’ knowledge about methodologies directly affects the words they use when they ask for a study. The more they know, the more specific the request is. You want to gently and respectfully skip that part of their request – especially when they don’t have a good grasp of research methodologies. No matter what your stakeholders know or don’t know, you want to delay any discussion about research methodologies for two reasons:
1. It’s your job. Matching research goals and questions with a methodology that brings valid and reliable results is a skill researchers have, or should have. With all due respect to engineers, product managers, and marketers, it takes a lot of reading, training, and experience to become good at this. Yes, I know many user experience practitioners do not have formal education in HCI or psychology. Heck, until ten years into my career as a researcher, I didn’t have any. I am also aware that many practitioners don’t have much experience. When you think about it, even Jakob Nielsen, Jared Spool, and Alan Cooper started somewhere. In the first two or three years of their careers, they also didn’t have much experience. You get experience by conducting studies. When you don’t have the formal educational kosher stamp or when you don’t have much experience, I suggest that you read. A lot. A lot more than what your stakeholders might read about user experience research and design. Even if you do have a formal stamp, keep reading. UX is broad, and it is a hallmark of a good practitioner to be well read. All these forms of learning, individually and combined, will help you match a goal with a methodology. You do not write code, nor do you persuade prospects to purchase products. This is your job. You are a user experience professional. This is what you do.
2. You need time to think. Sometimes it is pretty straightforward to match a research question with a methodology. But in many cases it isn’t, even though it might seem easy. I think experience is really important here – primarily other people’s experience. You want to think about the methodology that will provide answers to your stakeholders. You want to be right. You want to take advantage of others (in a good way) and consult with experienced people. The one great thing about the user experience community is that it is extremely supportive, available, and willing to help you. One of the most useful resources is a secret, online list that cannot be mentioned here (contact me at [email protected] if you want details). Use it. My point is that the process takes time and a lot of thought needs to be put here.
One of the most important things you need to do is to align your stakeholders with you, and what will eventually help you delay discussions about methodologies is saying reasonable things.
Become the voice of reason
What you communicate to your stakeholders when they first ask for a study has an enormous effect on the end result. In other words, what you say now determines the level of impact of the study. Yes, it is that dramatic. When you communicate, there’s the text that comes out of your mouth, documents, or emails. There’s body language that (in most cases) unwillingly takes care of what you really think and don’t want to say. And then there’s subtext, which includes things that are implied by what you say or do. Your job is to control all of these communication channels so that they communicate one message to your stakeholders: I’m here to help us do a better job. This message means that you know what you are doing and that you are a confident professional.
Your communication now affects the end result of the study because it sets both the atmosphere and the expectations of your stakeholders. By atmosphere, I refer to issues such as:
Does the team trust the researcher to bring useful results?
Is there mutual respect between stakeholders and the researcher?
Does the researcher think this study is right or that the request is legit?
Is everyone excited with the study or are there any dark clouds over it (maybe a nonsupportive executive)?
As for expectations, I could not stress enough the importance of setting the right expectations. Imagine what might happen if any one of the following occurred in the end of a study:
Your product manager says he needed the results three weeks ago.
Engineers complain that study participants were not representative and that there were not enough of them.
The VP of sales was heard saying that user research does not give any added value. Salespeople, she said, could have provided similar insights about client needs.
Although you cannot prevent people from drawing these types of conclusions, you can minimize the chances of this happening if you make sure the right expectations for studies are clearly defined, understood, agreed upon, and met. Becoming the voice of reason means, among other things, that you make sure you set the right atmosphere and expectations.
How else can you become the voice of reason? At this point, when you are trying to learn as much as you can about a possible user experience project, you primarily want to ask good questions. Being a professional means that you do not express your opinions about what needs to be done – not just yet. What you should do now is make sure you have all the information you need to decide whether you are going to propose a research project and what this project is going to include.
One of the best examples I have for showing your stakeholders that you are the voice of reason relates to the words you use that indicate who owns the study. You can easily hear your product manager saying something like, “So, when do you think you can share a written plan for your study?” The instinct would be to answer their question, but that is the wrong response. The language that you and your stakeholders use is important. It’s not your study. It’s theirs and yours. Don’t let anyone, including you, refer to it as his or hers. It’s not about any of you. It’s about what the study reveals and what the results tell about how products can be better. When a stakeholder says something like this to me, I immediately stop the discussion and very clearly explain that it is not my study, it’s ours. I do it with a smile, not aggressively. By using language to imply shared ownership, I communicate that this matter is important.
Listening and sowing seeds
If I needed to summarize this section in one sentence, I’d say shut up, listen, then start talking.
User experience practitioners who are also excellent interviewers know that listening is a key aspect of a successful interview. How many times have you found yourself wondering, “Did I just say that?” Sometimes a word, a gesture, or even a blink or a certain body posture can bias an interviewee and add flaws to data you collect. Let’s discuss several aspects of listening to your stakeholders. You will quickly see how these are similar to techniques you apply when interviewing users.
Take notes. When you listen to what stakeholders say, take accurate notes. You will find these notes extremely useful when you move on to creating the research plan, especially when you phrase research questions and participant characteristics. Taking notes also shows your stakeholders that you care about what they think and want. It’s all about respecting each other. I once had a study participant who noticed that I stopped typing on my laptop keyboard and immediately responded, “That’s it? I’m not saying anything important any longer?” Your stakeholders are no different.
Do not interrupt. An additional way to show disrespect to your stakeholders is to keep interrupting them as they answer your questions and express their thoughts. When you truly listen to a person you converse with, you have no reason to interrupt them. Some people, when interrupted, tend to lose their line of thought, so you might also be missing important information.
Encourage a conversation. When there are several stakeholders you meet with, it’s a golden opportunity for codiscovery. When you ask a question, try creating a conversation between stakeholders. It will help you understand the sometimes many aspects and layers of answers. It might also uncover hidden political forces and tensions you were not aware of and that might affect the future of the study and its impact.
Ask open-ended questions. When you ask your product manager, “Do you want me to invite 30- to 40-year-old women for this study?” you’ve invited them to say, “yes” or “no.” What you really want to know is why. Instead, try this open-ended phrasing: “Tell me about the participants you want to see in this study.”
Look for body language signals. By the body language signals your stakeholders give you, you can tell if they truly want the study they are asking for, if they are forced to ask for it, if they are interested in the results, what they think of you and your ability to help them, and much more. Very generally speaking, if you see your stakeholders sitting back, crossing their arms and legs, and not looking you in the eye, you have a challenge ahead of you. If they sit up straight, maybe at the edge of their chair, lean forward, and wave their hands when they talk, you can tell that they are engaged, excited, eager to partner with you, and open to learning from users.
Don’t ask leading questions. You want to ask the most neutral questions, not bias your stakeholders. When you ask nonleading questions, your stakeholders talk more, give you more helpful information, and allow you to listen.
Don’t think too much. You are now in information-gathering mode. Think army intelligence. You don’t plan your moves now. You don’t think about the data in front of you. All you do now is collect and gather. Later on, you’ll think about it. If you think now, you don’t listen.
Watch my interview with Donna Tedesco, a Staff Usability Specialist in the user experience research team at a large organization located in Boston, Massachusetts. Donna talks about how she identifies research opportunities in her organization. Use QR code 114 to access the video, a quick summary of the interview, and Donna’s biography.
In order for you to truly affect people, teams, organizations, and products with user experience research, you need to not only listen but also talk, but not too much. It’s about quality, not quantity. When you hear a version of the same thing repeated in three different meetings by several stakeholders, use it. The next time you meet one of these stakeholders – or, even better, his or her manager – mention it to them. Say something such as, “I’ve been hearing that information about X has become a knowledge gap in our team. What do you think? Do you think it can be filled in with input from user research?”
Another way to sow seeds is repetition. It works for politicians and children. Politicians believe that if their message is repeated countless times, their voters (and potential voters) will eventually think it’s true or right. Children also need to experience something again and again to become satisfied. And then they want to experience it once again. Bottom line: repeat your message like a broken record. When you hear your stakeholders repeat what you have been consistently saying, it’s a sign that you planted the right seeds. Let’s say you’ve identified an opportunity for making an impact with user experience research. Create some kind of a mantra – a phrase that you keep using each time something relevant is being said in a meeting or conversation. For example, let’s say that you’ve determined that the team doesn’t know anything about what users do with information they export from the product. Do they import it into Microsoft Excel? Do they print it and hang it on the wall? Why do they export information? You listened well and kept hearing stakeholders and their managers refer to this knowledge gap. The mantra you might use each time someone mentions that they don’t know what users export and why would be: “Watching a few users export data is easy; we could do it next week.” Repeat it at every (relevant) opportunity. Eventually, it will sink in. When you hear the product manager saying that watching users export data is easy and can be done next week, it’s a sign that you’ve done well.
Watch my interview with Paul Adams, a product manager at Facebook. Paul, previously a UX researcher, talks about the pressures and priorities product managers have. Use QR code 112 to access the video, a quick summary of the interview, and Paul’s biography.
Takeaways
This chapter focused on interviewing your stakeholders at the beginning of a user experience research project. Here are the main points to remember:
1. Getting requests to conduct research is a good thing. Not getting any requests should concern you.
2. Kicking off a research project with a short meeting, during which you ask your stakeholders the following questions, is key to the success of the study:
3. Push back attempts to affect the methodology selected for the study before a study plan is ready. Matching a methodology to research questions takes some thought and doing so is your job, not your stakeholders’.
4. Words you use determine the level of impact the study will have. Use them wisely.
5. Set reasonable expectations among your stakeholders. Be honest and straightforward.
6. When you interview your stakeholders, be quiet and listen very carefully to what they have to say.
7. Use these great interviewing techniques: take notes, do not interrupt, encourage a conversation, ask open-ended questions, look for body language signals, don’t ask leading questions, and don’t overthink about the answers.
8. Attend your stakeholder meetings and try to identify knowledge gaps related to users, their needs, and their experience.
9. When you identify a knowledge gap, repeat the fact that it exists, hoping that someone will suggest conducting research to fill the gap.