One of the most important factors in determining your research approach is an understanding of what questions you want to ask. This may seem recursive, but just as asking participants good questions is invaluable to product design, challenging yourself to reflect on the same questions in defining a research plan is critical. Are you looking to understand the path customers take using your product? Have you accounted for areas of stakeholder interest that are relevant to the research efforts? Maybe analytics or contextual inquiry can offer insight and clarity. If you are asking why issues arise, diary studies or other surveys may be more conducive to your needs.

Just as important as knowing what questions need to be answered is an understanding of what your stakeholders’ goals are. For in-house teams, these goals may align with the designers’. Still, understanding if stakeholders are looking to validate their assumptions, define a new market, or prove their value within an organization helps you define how to address research. Will a quantitative report resonate with your business stakeholders, or will they be more receptive to observing experiences firsthand? How can you best inform stakeholders about the goals and outcomes of your questions?

Sample size might be a more binary choice between quantitative and qualitative methods. Based on their roots in statistics, quantitative methods often require large data sets. If you have limited participants available, qualitative methods might be more informative.

If your questions are about the desirability of a feature, you may have access to your entire customer population, and surveys or intercept testing would be most valuable. On the other hand, if you are looking to explore a specific workflow, the customer population may be smaller and qualitative methods would be more appropriate. That said, this guideline is for representative purposes only. Quantitative studies can be performed with small data sets, and similarly, large-scale qualitative studies are common for international products.

Qualitative methods are great for small sample sizes, especially when you can travel to participants in their environment. But with products spanning global markets, travel costs and time may become cost-prohibitive. While many qualitative methods can be used remotely, location is one key factor in determining a research approach.

Unmoderated validation is a great way to span geographical divides, and depending on your timing and goals, you can integrate more quantitative or qualitative approaches into your research plan.

Budget is another limiting factor in choosing methods. Just as location can be cost-prohibitive, specific methods can be as well. If your question is “How are customers navigating through a product?” site analytics provides a low-cost method of retroactively reviewing system usage. Conversely, contextual inquiries may provide more insight and allow for probing questions, though the time investment (approximately 90–120 minutes each) can be difficult to schedule.

Though the answer to “When do you need to see research results?” is often “yesterday,” this is actually a practical question.

Quantitative measures can often be prepared in a set-and-forget mode. For instance, analytics, once enabled, are constantly measured. Card sorting and tree jacking exercises can be enabled and disabled at a whim, and data can be gathered from multiple participants in parallel. Qualitative measures, on the other hand, require more hands-on involvement from researchers and often direct contact with participants, extending the amount of time needed to evaluate the same number of customers.

When choosing a research method, balance the urgency of results with the specialization of various methods. The common saying among product designers is, when considering your constraints of time, budget, and scope (often called the “project triangle”), choose the two that matter most (Figure 5-1).

Figure 5-1. There are three competing forces in any project; it is common to focus on the most important two and allow the other to be more variable

The best way to combine methods is to understand the root of your question and identify a qualitative and quantitative measure for it. If you’re interested in workflow, analytics and contextual inquiries are good options. If you’re interested in desirability and feedback, surveys and customer feedback reports are invaluable. In either case, data from one piece of research can be used to inform questions for other research. This process of asking questions, conducting research, and identifying new questions for future use is a key aspect of any successful product team.

While merging methods sounds good in theory, it is important not to combine two totally disparate methods and expect valuable results. Similarly, don’t assume the same method will be appropriate for two different research goals (e.g., card sorting is helpful for illustrating system structure but less valuable for task workflow questions). Research sessions should remain focused on specific goals and questions. If you’re interested in both system structure and task analysis, we recommend two separate research tracks. That said, don’t discount new questions simply because they cannot be answered by the current method. If you are collecting metrics and a more subjective question arises, don’t be afraid of exploring that path too.

Voice from the Streets: Kevin H. Richardson, PhD

In a few words, describe your job.

I work with clients to scope new projects, mentor my international team (and Sales), and evangelize UX by writing as well as by speaking at meetings and conferences.

How would you summarize the importance and process of research in your work?

As a consulting organization, we specialize in the design of business applications. This means that clients often come to us believing they understand the cause of their problem (poor adoption, high error rates, lengthy training needs, etc.) and have some idea of the solution—even if that solution is simply “a better user experience.”

We know, however, that client “problems” are typically indications of deeper issues rather than the root cause. The job of UX is to look beyond the symptoms and determine the underlying nature of the disease.

With this in mind, there are several things we need to know. First, the business requirements and technical constraints. These are straightforward, though we can refine the former via individual stakeholder interviews. After all, clients usually have a pretty good idea of what they want.

Next, and most important, are the user requirements. Ideally, I’ll determine the number of users × unique role needed to make a representative sample, and we’ll go out into the field, conducting interviews, observations, and contextual inquiries. These methods enable the UX team to determine not only what users say they want but also to infer, based on observation and questioning, what they need. This is incredibly powerful. People aren’t typically able to see beyond their immediate wish list, and the difference between incremental improvement (giving users what they want) and innovation (giving users what they need) lies in skilled requirements elicitation. This means understanding what users are trying to accomplish through their interaction with an application, how they work, and how the application in question fits within the larger ecosystem of other applications, tasks, and workflows that make up a typical day.

Without this information, all you have are usability improvements and educated guesses.