FOCUS GROUPS
WHEN SHOULD YOU USE A FOCUS GROUP?
THINGS TO BE AWARE OF WHEN CONDUCTING A FOCUS GROUP
PREPARING TO CONDUCT A FOCUS GROUP
DATA ANALYSIS AND INTERPRETATION
CASE STUDY: ENGINEERING COLLABORATIVE TOOLS: A DIFFERENT USE
Introduction
In a focus group, six to ten individuals are brought together to discuss their experiences or opinions around topics introduced by a moderator. The session typically lasts one to two hours and is good for quickly understanding user perception about a particular topic or concept. Focus groups are typically used by marketing professionals, but they have been used in social research since the 1930s. In our experience, the focus group is a valuable methodology when done correctly. One of the key benefits is that the group dynamic brings up topics you may never have thought to ask about. In addition, the group discussion can stimulate new ideas or encourage participants to talk about things they wouldn’t have thought about if interviewed alone.
In this chapter, we highlight the strengths of this methodology and discuss how it can be used effectively as a part of the user-centered design (UCD) process. We present a common method for conducting a focus group, as well as several modifications. Finally, we discuss how to analyze and present the data, along with some of our lessons learned. A case study by Tech-Ed is provided at the end to demonstrate a real-world application of this frequently used methodology.
When Should You Use a Focus Group?
Focus groups are excellent for the generation of ideas and for quickly gauging user impressions about a topic or concept. They allow you to capture information that might be more expensive to study directly. For example, in a focus group you can learn about activities that span days or weeks, which can be expensive to do via field studies. You can’t get information as quickly and from as large a variety of companies/users with individual interviews or field studies as you can with focus groups.
You can also discover problems, challenges, frustrations, likes, and dislikes among users. They help you understand users’ opinions, attitudes, preferences, and initial reactions. Focus groups can also be used to discover users’ priorities (e.g., must have, nice to have, what they are willing to pay for).
Focus groups are one way to gather information about a target audience that you have very limited information about. You may not have enough information to design a survey (see Chapter 8, page 312) or develop a field study plan (see Chapter 13, page 562), but a focus group with six to ten users can provide the information you need to prepare for other usability activities. Below is a list of ways you can use a focus group to prepare for other usability activities:
Identify or develop a question for a wants and needs analysis (see Chapter 9, page 370)
Identify objects to be included in a card sorting activity (see Chapter 10, page 414)
Identify potential steps for a group task analysis (see Chapter 11, page 458)
Fine-tune the wording of survey questions and identify the most frequent or likely responses for multiple-choice questions (see Chapter 8, page 312)
Identify new questions for surveys that you had not considered before (see Chapter 8, page 312)
Identify scenarios (see Chapter 2, Before You Choose an Activity, “Learn About Your Users” section, page 41).
Focus Group, Individual Interviews, or Surveys?
Focus groups are excellent if you are looking for multiple points of view in a short period. They can be used in the middle or at the end of the development cycle. They can also be a great activity to help you gather initial requirements or ones that can be of benefit at a later stage of development. For example, a focus group can help you understand surprising or contradictory results obtained via surveys (see Chapter 8, page 312) or identify reasons why user satisfaction is so poor with your product. Individual interviews (see Chapter 7, page 246) are a better choice than focus groups if you are looking for detailed information that you cannot obtain from group discussion or via a survey. You are able to follow-up on responses and clarify participants’ statements, which is not possible in a survey. Furthermore, you can spend more time understanding a single participant’s needs, thoughts, and experiences in an interview than in a focus group. You can also cover more topics and do not have to worry about participants influencing each other’s responses. In a focus group, you are typically limited in the number of questions you can ask and the depth of discussion with each participant because you want to hear from each participant equally.
One-on-one interviews can take significant time to conduct and more resources than a survey. However, there may be times when you need to collect large samples of data to feel more confident that the results you have obtained can be applied to the user population as a whole (not just the people you interviewed). This is not typical of most product development, but it may be the case, for example, in government regulated industries. Since you cannot obtain statistically significant results with a focus group (or even several focus groups), you cannot be sure that the proportion of responses to a question or topic in a focus group will match the proportion in the general population. In other words, if eight out of ten participants in a focus group state they want a particular feature, you cannot extend that to say 80% of all users want that same feature. If you are looking for statistically significant results that can be generalized to the population as a whole, surveys are a better choice.
Things To Be Aware of When Conducting a Focus Group
There are a few situations in which we recommend using caution when designing a focus group or suggest using an alternative method instead.
Avoid Predictions
Studies have found that participants are not always good at predicting the features they would find useful in practice (Root & Draper 1983; Karlin & Klemmer 1989; Gray, Barfield, Haselkorn, Spyridakis, & Conquest 1990). This is why we recommend against asking participants to “pretend” about things with which they have no experience. For example, it is completely appropriate for you to ask someone “What is the biggest challenge you face in your job?” and then follow up the user’s response with “What would make that part of your job easier to do?” You are not asking the participant to pretend – the participant does that job and may encounter that challenge every day. You can be sure that the participant has thought on many occasions about ways the challenge could or should be addressed. Going back to our travel example in earlier chapters, a travel agent might say that her biggest challenge is when people call her to book a vacation but never have the information needed, like the maximum budget, the dates of travel, and desired destinations.
On the other hand, you should not ask the travel agent whether she would like using voice-activated input, if the agent has never seen or used such a system before. She may think the concept sounds really cool when you describe it, but once she begins using it at work she hates it because everyone in the surrounding cubicles can hear every mistake she makes.
Avoid Sensitive or Personal Topics
It should go without saying, but it is unwise to discuss extremely sensitive or personal topics like politics, sex, or morals in a focus group. Obviously, these are topics that are sometimes uncomfortable or cause heated discussion between friends and family. It is not appropriate for you to ask participants to discuss them in front of strangers and on camera.
Use a Skilled Moderator
As with any group activity, you must have a skilled moderator familiar with the domain to contain overbearing users, draw out quiet users, avoid bias, probe to understand users’ underlying needs, and limit groupthink (i.e., the tendency for the members of a group to come to a consensus, rather than expressing individual ideas). The moderator must always keep command of the room, be personable, treat even difficult participants with the utmost respect, and never pass judgment on the opinions expressed during the session. (Refer to Chapter 6, During Your User Requirements Activity, “Moderating Your Activity” section, page 220 for a detailed discussion about effective moderation.) If a moderator is not successful at these tasks, the data collected will be biased and should not be used.
If you are not a skilled moderator, you could hire a professional to moderate the group (Appendix B, page 688, has a list of professional moderators and training courses). Contact your marketing department since marketing professionals typically have training and experience running focus groups. One advantage of using an outside moderator is that he/she typically will not share your biases and will be more likely to collect information in a neutral format. Unfortunately, the added expense and time to bring the moderator up to speed on your product or area of concern may not be feasible. Ask for a sample of their work (i.e., quick findings, usability report, presentation) and find out exactly how long it will take to complete the activity from kickoff to report delivery. And don’t forget to check those references. Optionally, you may choose to have a member of the product team serving as a co-moderator to follow-up with more detailed questions that the outside moderator would not know to ask.
Observation is Still Best for Understanding Tasks
In one study, participants in three focus groups were asked how they would complete the task of assessing the credibility of a website (Eysenbach & Köhler 2002). Participants provided a number of methods they would use – but when many of those same individuals participated in a usability evaluation and were asked to assess the credibility of a website, they did not use the same methods they described in the focus groups. The takeaway here is that if you want to know exactly how end users go about completing a task, the best method is via direct observation. This applies whether you are conducting focus groups, interviews, or surveys.
Focus Groups are Not Appropriate for Comparative, Competitive, and Benchmarking Data
Focus groups are not appropriate for conducting formal usability tests for your product. Conducting comparative, competitive, and benchmarking studies with a group of users results in co-discovery and doesn’t produce individual data. If you want to collect task performance information (i.e., errors, assists, task success), it is best to run individual usability tests. Unless your end users normally work with your product as a group, collecting group performance information does not provide an accurate assessment of the usability of your product. It would, however, provide you information about aspects of your product that participants think are easy or difficult and things that they like or dislike.
Preparing to Conduct a Focus Group
Before you begin any user requirements activity there are a number of things you must be familiar with. Because these elements are common to all user requirements activities they have all been covered in the earlier chapters, but now is a good time to double-check the list. If you are not familiar with one of the items you can refer to the earlier chapter to brush up or familiarize yourself with that element.
Chapter 1: Introduction to User Requirements
Getting stakeholder buy-in for your activity (see page 14)
Chapter 2: Before You Choose an Activity
Chapter 3: Ethical and Legal Considerations
Chapter 4: Setting up Facilities for Your User Requirements Activity
Create or acquire a facility for your activity (see page 106)
Chapter 5: Preparing for Your User Requirements Activity
Develop an activity proposal (see page 146)
Determine the time of day and duration of your session (see page 155)
Recruit participants (see page 156)
Develop an activity protocol (see page 191)
Pilot your activity (see page 193)
Chapter 6: During Your User Requirements Activity
Now that you understand what a focus group is and when to use it, we are going to present a fundamental version of a focus group. If you would like to make additions to the method, refer to the “Modifications” section on page 542.
Preparation Timeline
The timeline in Table 12.1 covers in detail the sequence and timing of events to prepare for a focus group. These are approximate times based on our personal experience and should be used only as a guide. It could take longer or less time for each step depending on a variety of factors – such as responsiveness of the development team, access to users, and resources available.
Identify the Questions You Wish to Answer
Begin your preparation by identifying the questions you wish to answer. You may want to have an initial brainstorming session with all of your stakeholders (members of the product team, the usability group, marketing). You will likely end up with questions that are out of the scope of the proposed activity or questions that are best answered by other activities. You may also end up with more questions than you could possibly cover in a single focus group session. If this is the case, you will either need to conduct multiple sessions or trim down the number of questions to fit into a single session. We do not recommend conducting focus groups sessions beyond two hours, however; dedicated participants or customers may be willing to stay for longer sessions provided you provide regular breaks and keep the session interesting.
Types of Questions You Can Ask
Below are several sample questions you can ask in a focus group:
A user’s “typical” day or the user’s last full day at work
The tasks that users do and how they do them
The domain in general (e.g., terminology, standard procedures, industry guidelines)
Users’ desired outcomes or goals (refer to Chapter 7, Interviews, “Outcomes Analysis” section, page 252)
Users’ reactions, opinions, or attitudes towards a new product/concept
Asking participants to discuss a “typical” day at work gives you a high-level overview of participants’ perceptions about the way they work or activities they may do in general. Asking participants to tell you about their last full day at work gives you specific examples about tasks they have recently done. You can also understand how certain tasks are done at a high level, challenges they face, and things they enjoy. Focus groups also offer the opportunity to learn about standard terminology, guidelines, and industry practices. Participants can additionally describe desired outcomes in their work or goals they want to achieve. Finally, you can gauge user reaction to concepts and brainstorm ideas for new products or features.
Wording Your Questions
Make sure that all questions are open-ended, worded clearly, impartially, and actually ask what you intended to ask. We strongly recommend reviewing the advice given in Chapter 7 before proceeding (refer to Chapter 7, Interviews, “Write the Questions” section, page 262).
The data you collect may be described as either qualitative or quantitative. We describe both types below.
Qualitative data
The majority of your focus group will likely be comprised of open-ended questions (i.e., designed to elicit detailed responses, and free from structure). These types of questions generate qualitative data. You may walk away with some quantitative data if you include ranking or polling as a part of your focus group (e.g., “85% of participants want to be able to book cruises online”), but the bulk of your data will be qualitative.
Let’s use our online travel website as an example and design some questions for a focus group. You know that college students sometimes attend professional conferences relevant to their major. You also know that college students are usually poor. Your company would like to offer “room sharing” on your website. People who are interested in sharing a hotel room in a particular hotel or city could sign up and specify preferences (e.g., females only, under $50/night, no more than three roommates, non-smoking). Since you are not aware of any travel website that offers this, you would like to learn more about people’s attitudes towards signing up on a website to share a hotel room. A focus group seems like the best way to get a general feel for attitudes, biases, and new ideas in this area. This will be followed up with a survey and sent out to a few hundred current and potential customers. With a two-hour focus group and ten participants, you realize you can probably cover only about five or six questions. Some open-ended questions you might ask are:
How satisfied are you with the kind of accommodations your budget allows?
How willing would you be to share a hotel room with someone you know to lower your costs and get a nicer room?
How willing would you be to share a hotel room with someone you didn’t know to lower your costs and get a nicer room?
What would you want to know about a person with whom you might share a room?
What would make you more likely to share a room with someone you didn’t know?
Quantitative data
During a focus group you may ask closed-ended questions. These are questions that provide a limited set of responses for participants to choose from (e.g., yes/no, agree/disagree, option a, b, or c). In a focus group you may also ask participants to rank a series of options or to vote for their preferred choice, or you can poll the participants (i.e., determine how many people agree with a statement). These types of questions collect quantitative data that can be counted or measured (but see our caution on page 517 about using quantitative measures with small numbers of participants). The benefit of collecting this type of data during a focus group rather than on a survey (where it is typically collected) is that you can ask individuals to discuss why they made the selection(s) they did. In fact, it’s best to think about ratings and polls as providing an opportunity to have a discussion rather than as a quantitative measure of a preference or attitude. You may also choose to give a survey prior to or following a focus group in order to address questions that you do not have time to address in the focus group itself.
Using our travel example, some closed-ended questions you might ask are:
How many people have a hotel budget when they travel for work or school? [ask for a show of hands]
If you had a limited budget to attend a conference/trade show/meeting, and could not stay in the hotel where it was being held, which of the following options would you choose: (a) stay in a hotel you can afford three miles from the hotel; (b) stay in the conference hotel but pay the extra out-of-pocket; (c) share a room at the conference hotel with someone of the same gender you didn’t know?
Do you agree or disagree with the following statement: “I would be willing to share a hotel room with someone I didn’t know to lower my costs and get a nicer room.”?
It is possible to generate quantitative data from open-ended questions by collecting data that reflects the number of people who agreed with a specific comment or point of view discussed. However, the bulk of your data collected from open-ended questions will typically focus on the actual responses and overall sentiments of the group. And remember that your quantitative data will not be representative anyway.
Test Your Questions
Use the data from your pilot session to practice your data analysis (see “Data Analysis and Interpretation,” page 538). You can prepare to analyze quantitative data by creating spreadsheets and entering in data from your pilot session. Basic analyses such as averages, minimums, maximums, and frequencies can be computed. (Refer to Chapter 8, Surveys, “Types of Calculation” section, page 349, to learn more about analyzing quantitative data.)
There really isn’t a lot to do to prepare to analyze qualitative data. Three common methods of data analysis are the affinity diagram, simple summarization of user responses to each question, and using qualitative analysis tools. We discuss each of these options later (see “Data Analysis and Interpretation,” page 538). If you choose to use a qualitative analysis tool, enter the pilot data into the tool to learn how the tool works and whether it will suit your needs.
Present the analysis of your pilot data to the product development team and state what conclusions you can draw from those results. Is this the type of information that the team is looking for? If not, you need to reword or create new questions to collect the information the team is seeking.
Players in Your Activity
Of course you will need end users to take part in your session, but there are also three additional roles that need to be filled to conduct a successful focus group: moderator, note-taker, and videographer.
You need to decide how many sessions to run, the number of participants per session, and the composition for each session. Below, we address those issues and provide recommendations to help you decide.
Participant mix
Some focus group professionals like to mix different user types in the same session. They feel they can get better discussion when different perspectives are brought together. We do not recommend this approach. With as few as six users per session, you cannot get a large enough sample size to know whether you are getting representative answers from each user type. In addition, the user types may be so different, that you cannot capture quality information to build upon. Finally, some user types should not be placed together. For example, managers and their direct reports should never be placed in the same group because the direct reports may not be honest or may defer to their managers. Managers may even feel it necessary to “take control” of the session and play a more dominant or expert role in the activity to save face in front of their employees. The issues are similar when you mix user types of different levels within a hierarchy, even if one does not report directly to the other (e.g., doctor and nurse). Refer to “Lessons Learned” (page 546) for a discussion of how mixing user types of different levels in a hierarchy can cause real problems.
Lastly, speaking with especially effective or expert users can provide a wealth of ideas since these “lead” users can often be a source of innovation. However, you should also recruit novice and average users since expert users may request features or services that are too sophisticated from the majority of the user population. By speaking with each user type individually, you can quickly understand the needs and issues for a broad spectrum of your population.
How many groups? How many participants?
Since the group dynamic is an important component to this method, we recommend recruiting 6–10 end users per session; however, groups as small as four can still provide valuable information. Groups with more than ten participants can be difficult to manage because you are not focused on one question – as in a wants and needs analysis where we normally have groups of 12 participants (see Chapter 7, Interviews, page 246). The questions in a focus group may be more complex and the answers are more in-depth. As a result, if you have a large group, it is unlikely that everyone will have an opportunity to speak. In addition, since you want to hear from every participant, you can cover very few questions with large groups. It may seem that two additional participants will not make a big difference, but when you are trying to get multiple perspectives to multiple questions, it will.
Although it is excellent to run two or three groups of ten participants, your time and resources may not permit it. If you are restricted on the number of participants you can recruit, it is better to run multiple smaller groups than one large group (e.g., it is better to run two groups with five participants each rather than one group with ten participants). This is because group dynamics can vary. You do not want one dominating user to influence your entire pool of participants. In addition, you may learn information in one focus group that you never thought about and would like the opportunity to develop new questions for a second focus group. If you put all your participants in one basket, you loose the opportunity to revise your questions. If you change your questions between sessions, you cannot compare the answers across groups but you do have the opportunity to cover more ground.
Only you know what resources you have available to conduct your studies and you must weigh the resources available with the questions you need to answer. There is no magic formula for determining the numbers of participants and groups to run – the final answer is: “It depends.”
Even though your idea for room sharing originally focused on college students, you realize that other people may be traveling on a budget and want to take advantage of this new feature as well. Since students are different from other travelers, you identify them as one user group and identify adult budget travelers as a second user type.
Your research identifies college students as being 18-24 years old, currently enrolled in school, and they travel at least once a year on a tight budget. Adult budget travelers are 25–40 years old. Based on some preliminary research that you have done, it is likely that males and females will have different needs and criteria when deciding whether or not to share a room with a stranger, so you decide to run adult male and female budget travelers in separate groups. So your user composition is:
The moderator
One moderator is needed per session to facilitate the activity. (Refer to “Conducting a Focus Group,” page 535, as well as Chapter 6, During Your User Requirements Activity, “Moderating Your Activity” section, page 220, for the details of being an effective moderator.) The moderator will elicit responses from the group, examine each answer to ensure he/she understands what the participant is really asking for, and then paraphrase the response to make sure that the intent of the statement is captured.
The moderator must engage all the participants. This is typically done by containing any overbearing participants and drawing out quiet participants. Finally, the moderator needs to know when to let a discussion go off the “planned path” into valuable areas of discovery and when to bring a fruitless discussion back on track.
The moderator needs to have sufficient domain knowledge to know which discussions are adding value and which are sapping valuable time. He or she also needs to know what follow-up questions to ask on-the-fly to get the details the team needs to make product decisions (see “Lessons Learned,” page 546).
If you have never run a focus group before, participate as a user in several different focus groups. You can go to online community boards or search job listings for companies in your area that are looking for focus group participants. Also, observe different moderators in action to learn what worked and what did not. You may also identify a certain style that you would like to emulate. It is essential that you practice moderating a focus group rather than just jumping in feet first. You do not want to learn “on the job” when expensive data collection is at stake.
Figure 12.1 is a moderator’s checklist. For a detailed discussion of the art of moderating, refer to Chapter 6, During Your User Requirements Activity, “Moderating Your Activity” section, page 220.
The note-taker
A note-taker is needed to help the moderator. The moderator has a big job and should not be worried about taking detailed notes. The sole job of the note-taker is to write down notes from the session. It is important for the note-taker to have domain knowledge so that he/she knows what points are important to capture and what comments can be left off, as well as to ensure that the notes make sense. (See “Lessons Learned,” page 546, to read more about the importance of domain knowledge.) You will find a detailed discussion of note-taking tips and strategies in Chapter 6 (refer to Chapter 6, During Your User Requirements Activity, “Recording and Note-taking” section, page 226).
The notes can be displayed for the group (including the moderator) to see. This can be done on a laptop and projecting the image or by writing the notes on flip charts at the front of the room. Obviously, reading someone’s handwriting is not an issue when using a laptop, but if your note-taker is not a fast or good typist, you may want to go with the handwritten notes. An obvious advantage of typing the notes during the session is that you can send out the notes to stakeholders immediately following the session.
Taking notes for the group to see has a few advantages and disadvantages. One advantage is that it can help the participants to avoid repeating the same information, and it shows the participants that their comments have been captured (otherwise participants may ramble on to be sure you captured what was said). If the note-taker captured a comment incorrectly, the participant can correct him/her immediately. Finally, the moderator can refer back to the notes during the session to follow-up on a particular comment or to direct the group to a different line of discussion.
One disadvantage to having the notes displayed for all to see is that you may not be able to use a lot of shorthand because the participants won’t understand it. Additionally, the note-taker should not include design ideas or notes about the participants, comments, or session for all to see.
Whether or not the notes are being displayed to the group, the big benefit to having a note-taker is to avoid the need to watch the entire focus group on tape for later data analysis. As soon as the session is over, you have data at hand to begin analyzing. If there are areas where the moderator feels that the notes do not make sense, he or she can go back and watch just that portion of the session. This cuts down significantly on the time it takes to analyze the data. Finally, the note-taker can help you analyze your data. Since the note-taker was there for the entire session, he/she understands the data as well as you and it is always helpful to have an extra pair of eyes and a different perspective to analyze the data.
If a note-taker is not available, you could always go back and take notes while watching a tape of the session. However, we recommend that you do this only as a last resort. A note-taker is extremely helpful and can save a great deal of time.
Some focus group professionals prefer to have their sessions transcribed by professionals. People working in highly regulated fields (e.g., the pharmaceutical industry regulated by the FDA) may be required to have precise documentation of any usability activity they conduct since the information learned in such sessions provides data for making product design decisions. Transcribing focus group sessions is extremely time-consuming and expensive. It can take an average of six hours to transcribe one hour of tape. If multiple voices are involved, it will likely take longer. Companies usually charge per hour of transcription time (not per hour of tape). In our experience in the San Francisco Bay area, we have found that they charge about $42/hour. Not only is this expensive, it can take days before you get your information back to begin data analysis. For these reasons, we strongly recommend using a note-taker to note key points and quotes for immediate data analysis.
The videographer
Whenever possible you will want a video record of your session. You will find a detailed discussion of videotaping tips and the benefits of videotape in Chapter 6 (refer to Chapter 6, During Your User Requirements Activity, “Recording and Notetaking” section, page 226). You will need someone to be responsible for recording.
In most cases, this person simply needs to start and stop the tape, insert new tapes as needed, and keep an eye out for any technical issues that arise.
Inviting Observers
If you have the facilities to allow stakeholders to view these sessions, you will find it highly beneficial to get them involved (refer to Chapter 4, Setting up Facilities for Your User Requirements Activity, “Lab Layout” section, page 123). Stakeholders can learn a lot about what users like and dislike about the current product or a competitor’s product, the difficulties they encounter, what they want, and why they think they want it.
If you take a break during the session, you can ask the observers if they have any questions they would like answered or areas they want to delve into further. You should not promise you will be able to cover them; but if time and opportunity permit, it is good to have these additional questions available. If the questions the observers suggest are clearly biased or would derail the focus group, you can always state: “Those are questions that we might want to consider for another activity.”
As always, we strongly recommend videotaping the sessions for any stakeholder who may not be able to attend, as well as for your own future reference.
Activity Materials
A basic focus group requires a few materials:
Laptop/computer or whiteboard or flip chart
Computer projector and screen (if using laptop/computer)
Blue or black markers (if using whiteboard or flip chart)
Materials for creativity exercises
Paper and pens for participants to take notes
Prototype or other artifact to stimulate discussion (optional)
We prefer to use a laptop and display each question on a PowerPoint® slide (see Figure 12.2). This looks professional and the questions are easy to read. However, a whiteboard or flip chart will work just as well. Remember to write clearly and with large letters in blue or black marker.
FIGURE 12.2 A focus group in action. The question has been projected so that everyone can read it easily. The tables are in a “U” formation so that everyone can see each other and the moderator easily
It is valuable to provide paper and pens for the participants. If multiple people wish to speak at the same time, you may have to ask them to “hold their thoughts.” Sometimes people forget what they wanted to say when they hear someone else speaking. It can be frustrating for both you and the user if good ideas are lost. Allowing participants to jot a quick note on paper can help save those great ideas.
Finally, presenting a prototype, a competitor’s product, or a video (if appropriate) can help encourage discussion and focus thoughts. You do not want to bias participants’ answers, but it is often helpful to show examples of the product (yours or a competitor’s) during the discussion. Make sure that all participants can easily see whatever you are presenting.
Conducting a Focus Group
You are now prepared to conduct a focus group. In this section, we walk you through the steps to conduct your session.
Activity Timeline
The timeline in Table 12.2 covers in detail the sequence and timing of events to conduct a focus group. It is based on a two-hour session and will obviously need to be adjusted for shorter or longer sessions. These are approximate times based on our personal experience and should be used only as a guide.
Table 12.2
Timeline of a focus group session
| Approximate duration | Procedure |
| 5 minutes | Welcome participants (introductions, forms) |
| 5 minutes | Creative exercise/participant introductions |
| 45–100 minutes | Discussion |
| 5 minutes | Wrap-up (distribute incentives, thank participants, escort them out) |
Now that you know the steps involved in conducting a focus group at a high level, we will discuss each step in detail.
Welcome the Participants
This is the time during which you greet your participants, allow them to eat some snacks, ask them to fill out paperwork, and get them warmed-up for your activity.
It is helpful to begin with a creative exercise such as designing name tents. Alternatively, you can discuss likes and dislikes of the current system or a competitor’s product. The details of these stages are described in Chapter 6 (refer to Chapter 6, During Your User Requirements Activity, “Warm-up Exercises” section, page 215).
Introduce the Activity and Discussion Rule
Once all forms are complete, introductions are made, and the participants are warmed-up, the rule for discussion should be provided. The rule is simple: “All ideas are correct.” It is important to explain to participants that their individual experiences may be different, and those differences should be expressed – but they should not critique each other’s ideas or perceptions. Write the rule “All ideas are correct” on a flip chart and have it visible during the entire session. If anyone breaks the rule, the moderator can point to it as a polite reminder.
The Focus Group Discussion
Participants should feel that the focus group session is free-flowing and relatively unstructured. You should have a list of questions or topics to be covered during the session, but you may discover interesting insights by allowing a conversation to progress down an unexpected path. Group interaction is the key benefit of focus groups, so it is important to be open to new topics and allow the conversation to flow, otherwise you may as well do a series of individual interviews. If your questions are presented on slides (see Figure 12.3), that does not mean you are locked into asking the questions in that order. As long as all your questions are answered by the end of the session, the order shouldn’t matter.
You may wish to begin the session or even each question with a “round-robin” to get everyone comfortable speaking in the group and ensure that everyone has a chance to speak.
As we mentioned earlier, you can ask ranking questions during a focus group or ask participants to vote for their preference. Rather than asking everyone to vote by raising their hand (and risk groupthink or evaluation apprehension), you can ask people to vote on paper. You can have the questions pre-printed and distribute each
question at the time you would like to vote. This prevents users from filling out all the questions in advance rather than paying attention to the group discussion. Or worse, users may look at the questions in advance and this could possibly bias the discussions. Alternatively, you can have a survey available for users to complete at the very end. It should be noted that voting on paper, whether it is during the session or on a survey at the end, removes the benefit of group discussion. A compromise is to vote on paper during the session and then ask volunteers to talk about how they voted. If no one feels comfortable sharing his/her vote, you can move on.
A possible component of focus groups is brainstorming. Chapter 9 provides tips for conducting an effective brainstorming session, and ways to ruin a session (refer to Chapter 9, Wants and Needs Analysis, “The Brainstorming” section, page 385, and “Modifications” section, page 401). We recommend reviewing those sections if you plan to have brainstorming during your focus group.
Data Analysis and Interpretation
Data analysis and interpretation can be divided into three steps: debriefing, analyzing quantitative data, and analyzing qualitative data.
Debrief
Within 24 hours of the focus group session, we recommend getting together with your note-taker to have a debriefing session. Review the questions asked and note the key points from the session. Were there any unexpected findings? What did each observer identify as the key take-away from the session? Were there any trends that can be identified at this point? Fill in areas of the notes that may not be clear while everything is still fresh in your mind. You should also decide whether another session is necessary with the same user type. If so, determine if you would like to change the questions based on what your learned in the previous session.
Analyzing Quantitative Data
As we mentioned earlier in the chapter, you can collect quantitative as well as qualitative data during a focus group. If you prepared a spreadsheet in advance, you can quickly input the data from your quantitative questions. Complicated statistics are
not appropriate with such a small sample. With small numbers of users, all you want to demonstrate is the average response, and perhaps frequency of response, and minimum and maximum responses. The details of these calculations are discussed in Chapter 8 (refer to Chapter 8, Surveys, “Types of Calculation” section, page 349). You should be very cautious about making too much out of the quantitative data from a focus group. As we mentioned earlier, if you ran one focus group and eight out of ten participants said they would buy a product, that data is not the same as having 80% of 100 participants in a survey say the same thing.
Analyzing Qualitative Data
Several software tools are available for purchase to help you analyze qualitative data. Even so, when we surveyed several usability experts, we found that none of them had ever used such a tool. If you are interested in exploring qualitative analysis software tools, refer to Appendix G (page 722) for a description of each tool and the pros and cons of using such tools.
Most professionals who conduct focus groups (including ourselves) find affinity diagrams (see Appendix F, page 714) and simply summarizing users’ responses more beneficial. Because this analysis technique applies to a variety of methods, it is located in an appendix for all applicable methods to reference. You may also find it helpful to develop a coding system as you analyze the data to identify trends across a single session or multiple sessions. For example, you might want to code each instance a participant mentions a shortcoming of your current product as “negative” and each positive statement as “positive.” This will make it easier to quickly scan your notes and summarize all the positive and negative statements participants made about your product.
There are several sources of information for you to draw upon as you analyze the results:
Notes taken by observers (they can have different insights on the participants’ comments)
Notes taken during the debriefing session(s)
Audio/videotapes of the sessions
You may wish to write a summary of the responses to each question and provide some supporting quotes. Use the participants– actual words whenever possible. This will likely require watching the tape(s) but it is worth it. You can group similar responses by user type, age, gender, or other factors you collected during recruitment. (Refer to Chapter 5, Preparing for Your User Requirements Activity, “Recruitment Methods” section, page 173.) Identify themes or frequently made statements. Look for data that support or contradict the thoughts and research already gathered by the development team or marketing. If the purpose of the activity was to create a basis for future usability activities, you should also demonstrate how the focus group results would be used in the next activity. You may also be able to provide guidance to the development team on new areas to investigate further.
Communicate the Findings
Preparing to Communicate Your Findings
The specific data that you communicate to product teams can vary depending upon the activity you conducted, but some elements of how you communicate the results are the same regardless of the method. Because these
strategies are common to all user requirements activities, they are discussed in detail in Chapter 14, Concluding Your Activity, page 636. We recommend that you read that chapter prior to relaying your results to your stakeholders. Topics discussed include:
Depending on the complexity of the data you collected, you can create a simple bullet list of information you learned during the focus group, a table comparing the answers between two or more user types, or a summary paragraph for each of the questions you asked users. Figure 12.4 is an example of a portion of a report.
Modifications
Because focus groups have been in use for so many years, there are a plethora of different modifications that you can try. Many of them involve providing users with exposure to the product or concept under development in different ways so that users have some base of experience from which to draw. For more information about a particular modification, check out the reference provided.
Task-based Focus Groups
In task-based focus groups, participants are presented with a task (or scenario) and asked to complete it with a prototype or the actual product. Several participants may be working in the same room at the same time but they should be working individually. If your product is software or a web application, this will obviously require several computers. After participants have completed their task(s), they are brought together to discuss their experiences.
It is best to give the participants the same set of core tasks so they can share common experiences. For example, in one study, participants were asked to look up information in a user’s manual and describe how confident they were that the answer they gave was right (Hackos & Redish 1998). Similar focus groups have been conducted for car owner’s manuals, appliance manuals, and telephone bills (Dumas & Redish 1999). Keep in mind, however, that a focus group is not a substitute for a usability test.
Similarly, you can present participants with multiple activities or artifacts during a single focus group. Participants may start off with a brief group discussion but then they work individually for the majority of the session. Multiple facilitators are needed for this activity. Each facilitator works with a participant one-on-one and completes a different activity (e.g., brainstorming new ways of solving a problem, view a prototype). After all the participants have gone through all of the activities, the group reconvenes to discuss experiences. Ideally, all participants will have completed all of the activities.
An excellent example of this methodology began with each of five focus groups photographing participants holding phone handsets to assess gripping styles (Dolan, Wiklund, Logan, & Augaitis 1995). Participants then rank-ordered six conventionally designed handsets and six progressively designed handsets according to several ergonomic and emotional attributes (after having experience with each). Finally, participants critiqued the handset designs according to personal preference, and each built a clay model of his/her ideal handset. By giving participants exposure to a wide range of experiences with the potential product or domain area, participants do not need to “imagine” what a product would be like. They can discuss their actual experiences with the product or domain and determine whether or not the product would support their desired outcomes or goals. The activities can also spark new ideas for you to draw upon.
The same type of questions can be asked in task-based and non task-based focus groups. The benefit of task-based focus groups is that follow-up discussions are richer when participants have the opportunity to actually use the product than when they must simply imagine it or remember when they used it last. With this technique, participants can reference the tasks they just completed to provide concrete examples, or the tasks may trigger previous experiences with the product.
The cost of doing such a study is that you must have several sets of your product available so that participants can work with them simultaneously. You will also need several facilitators to help, and the prep time to develop such a focus group will be longer than for traditional focus groups since you need to create the materials for several activities. It can also be more expensive because more materials are needed. However, the added benefit of giving participants something to experience and work with is worth the cost and should be done whenever possible!
Day-in-the-Life
You can create a “day-in-the-life” video to demonstrate how someone would use your product and then discuss participants’ impressions of what they have seen. Participants can get a better idea of what the product is really like without having to imagine it. This is perfect for situations where you cannot give participants direct exposure to the product or domain (e.g., requires significant training, safety issues, financially infeasible).
It is important to show someone realistically using the product (warts and all!) since providing an idealized image will not result in valid user impressions. This allows users an opportunity to determine whether what they are seeing would address their desired outcomes. It takes additional time to create the video and may even require you to develop a prototype of the product to work with if a real version does not exist. The benefit of getting all the participants on the same page and enabling them to share your vision is worth it!
Iterative Focus Groups
With iterative focus groups, you begin by presenting a prototype to participants and getting feedback from them. This can be specific design suggestions or just general impressions. Once the prototype is updated to reflect the feedback from users, the same participants are called back to participate in a second focus group. The new prototype is presented and additional feedback is collected. You could continue this iterative process as long as participants return, until you feel comfortable with the design, or until you run out of money.
The benefit is that you can see whether you are on the right track and understood the participants’ requests. It also takes less time to recruit since you are not looking for new participants each time. On the other hand, this modification is useful only in cases where you have enough information to build a prototype and you have the time and resources to make regular changes and conduct additional focus groups.
Obviously, it takes time and resources to make changes to a prototype and run additional sessions.
The Focus Troupe
The “focus troupe” is an interesting twist on the focus group (Sato & Salvador 1999). Members of the usability or development group – or even the participants themselves – perform dramatic vignettes demonstrating the new product, feature, or concept in use (participants would follow a script). The play should demonstrate the implications, operations, and expectations of what the product would do. A discussion among the group is then initiated.
As with some of the modifications listed earlier, participants can gain experience with the product or domain and this will limit their need to “pretend.” They can think of concrete examples of “using” the product while they provide their reactions, opinions, and alternative suggestions. The additional cost of using this modification is the time it takes to write the script(s), and – if you use co-workers in your troupe – the additional people involved. However, if you are unable to create a working prototype, this is the next best option.
Computer-aided Focus Groups
Computer-based brainstorming tools can be used for focus groups. If you are interested in these tools, please refer to Chapter 9, Wants and Needs Analysis, “The Brainstorming” section, page 385, and “Modifications” section, page 401.
Telefocus Groups
Smaller focus groups (i.e., five or fewer participants) can be conducted over the phone. This is more convenient for participants and you can recruit people from outside your geographic region. You can also save money because participants will often participate for less since they do not have to leave the comfort of their home or office. However, you need a high-quality phone meeting system that allows multiple people to call and clearly hear everyone speak.
There are several disadvantages to this modification. Social loafing (i.e., the tendency for individuals to reduce the effort they make toward some task when working together with others) is potentially high because participants are not held accountable by their peer group or the moderator if they do not contribute. A participant can surf the web or read a book without others seeing it. Doing a “round-robin” can help because everyone knows they will be called on to contribute to the question so they are less likely to slack off. It is also helpful to ask all participants to say their name before responding. It will be more obvious if, during an hour-long focus group, you never hear Bob announce his name. In that case, you call on Bob directly to reply.
Another problem is that the moderator cannot read the body language of the participants. There is no way to know whether a participant is silent out of disagreement, boredom, or because he/she has nothing to add. Queue-jumping by overbearing participants can also be a problem. It is hard to know who is speaking (even if you ask people to announce themselves before the speak) or if someone wants to speak but can’t get a word in edgewise!
Figure 7.3 on page 261 can help you further determine whether it is better to conduct the session in person or over the phone. Overall, we recommend telefocus groups when you have geographically diverse users, when you do not need to demonstrate a product, when you are tight on resources, and when you have a high-quality phone conferencing system available.
Lessons Learned
Below are three lessons we have learned with regard to user composition, moderating, and note-taking for focus groups.
We advised earlier not to mix user types of different levels in a hierarchy. The first lesson learned demonstrates why. Also, as we indicated earlier, it is important for a moderator to have good domain knowledge to follow-up on user comments, but it is also important for the note-taker. The moderator also needs to know how to handle difficult users, even to the point of removing a user to save the session.
Mixing User Types
A series of focus groups was conducted to learn about the use of “Patient Problem Lists” (PPL) by nurses and physicians. The goal of a PPL is to bring a provider up to speed regarding a patient’s healthcare or medical history without having to review the patient’s entire chart. Both doctors and nurses use it, so the development team wanted to combine the two user types in the same session to reduce the number of sessions needed and to hear a discussion of their different perspectives. Because the type of work they do is different, and because the relationship between doctors and nurses can be antagonistic, we insisted on separating the two user types.
During one of the sessions with the physicians, the participants began insulting nurses over the way they maintain their paperwork and the notes they take. A development team member who happened to be a Registered Nurse was watching from another room and nearly leapt through the one-way mirror when she heard the physicians’ comments! Although both user types would have likely been on their best behavior if combined, we would not have gotten their honest opinions. The physicians’ comments may have seemed mean-spirited at the time of the session, but the point was clear that the notes the nurses take (and find valuable in their job) clearly do not meet the needs of the physicians. Everyone on the product development team realized that it was far safer and more enlightening to keep those user types separated in future usability activities, even if it meant running additional sessions.
Having Sufficient Domain Knowledge
This may sound obvious but, when conducting a focus group, you need sufficient domain knowledge in order to know which questions to follow-up on and how or when to probe deeper into participant responses. This is also true in a wants and needs analysis (see Chapter 7, page 246) and a group task analysis (see Chapter 9, page 370). We witnessed one colleague’s painful two-hour focus group in which he did not have enough knowledge of the discussion topic. The moderator learned a great deal about the domain during his moderation; but because he did not know what information was important to follow-up on and what he should ignore, he could not provide new information and useful insights to the product development team as a result of the focus group.
What is different from a wants and needs analysis is that a focus group note-taker must also have domain knowledge to take notes in a focus group. On a separate evening this same individual took notes for a co-worker who was running a focus group on the same topic. Unfortunately, his domain knowledge was so insufficient that the notes were meaningless to his co-worker. He documented statements that were not noteworthy, missed points that were actually of value, and inaccurately summarized other comments. His notes were worthless and his co-worker had to review the session videotape to take proper notes herself.
Not only can you harm your credibility with a product team, you can lose credibility with the participants you are interviewing and your co-workers if you do not know what you are talking about. You need to do your homework before running any usability activity – including user requirements gathering activities. (Refer to Chapter 2, Before You Choose an Activity, “Learn About Your Product” section, page 29.)
Removing the Difficult Participant
During the same healthcare focus group discussed in the first lesson learned, we covered an amazing ten questions in each session. Each question was presented and then we completed a “round-robin” so that each participant had adequate time to contribute. After everyone had a chance to speak, people were free to add additional thoughts. This worked extremely well for the nurses. Once we reviewed the ground rules for participation, we never had to refer to them again.
The physician sessions were more difficult to moderate. The participants did not stick to the “round-robin” format, interrupted each other frequently, disagreed with each other, and by the end of the session were speaking quite loudly in order to be easily heard over the others. The ground rules had to be referred to regularly. At one point, the physicians even mocked the way nurses maintained and used the PPL. One outspoken physician in particular insulted the comments and ideas provided by another soft-spoken physician. Despite continued instructions of “everyone is right” and “do not critique the thoughts of others,” the domineering physician only got worse and the soft-spoken physician stopped speaking altogether. The amazing thing was that the negative vibe rubbed off on the other participants and they all began criticizing each other’s ideas.
We realized that, even with a great deal of experience moderating groups, it is easy to lose control of a session. Once you let one negative or overbearing participant take control of a group, the rest of the group will either behave similarly (e.g., insulting each other) or bow out of participating all together. Sometimes humor and polite reminders will not work. It takes a strong hand to get a session back on track. In this case, it meant taking the participant to the side and asking him to either quit criticizing others or to leave altogether. This is uncomfortable to do; but for the well-being of your participants and for the sake of good data collection, it is critical that you intervene as soon as you realize the session is headed on a downward spiral.
For more tips and information on moderating activities, see Chapter 6, During Your User Requirements Activity, “Moderating Your Activity” section, page 220.
Pulling It All Together
In this chapter, we have discussed what a focus group is, when you should conduct one, and things to be aware of. We also discussed how to prepare for and conduct a focus group, along with several modifications. Finally, we have demonstrated various ways to analyze the data and used our travel example to show you how to interpret and present the results.
Engineering Collaborative Tools: a Different Use
Stephanie Rosenbaum, Tec-Ed, Inc.
The Internet is the catalyst for the design of many new products. Innovation Chain (formerly Analytics Channel) developed collaborative tools for publishing and using engineering design information on a website called an Exchange. The company then engaged my research firm, Tec-Ed, to:
Learn how engineers currently save and share design information such as models and simulations
Observe how engineers search for and publish models and simulations
Gauge engineers’ reactions to the Exchange tool and elicit feedback.
To collect this data, we conducted usability focus groups with members of an engineering professional society, the IEEE (Institute of Electrical & Electronics Engineers), that was considering partnering with Innovation Chain to offer the collaborative tools. Innovation Chain approached us on the recommendation of the IEEE, for which Tec-Ed has performed many user experience projects.
The Innovation Chain marketing director had used focus group data at a previous company and suggested this type of research. After exploratory planning discussions, Tec-Ed agreed that usability focus groups were an appropriate method for the project, which sought target audience feedback not only on an early prototype of the Exchange, but also on its underlying business process model. The social quality of interactive discussion in focus groups makes them a useful research method to explore the potential value of a product to its intended users.
Our Approach
In a focus group, people with similar characteristics who do not know one another discuss selected topics with the assistance of a facilitator. The focus group facilitator creates an atmosphere in which participants feel free to express diverse points of view, with no pressure to agree or support particular ideas. Participants develop perceptions and make choices in part by interacting with the other people in the group, just as they do in real life.
Tec-Ed’s methodology for usability focus groups differs somewhat from traditional focus groups in that ours are task-based whenever possible, so we can observe the participants’ actual behavior with products or websites. These observations add validity to the data, because participants base their opinions on real product use, not on an imprecise (and invisible to researchers) image of how they might use the product.
After an initial group discussion, we typically “break out” into smaller clusters of two to four people who explore a product in a co-discovery manner, each cluster with a facilitator; then we reconvene the full focus group for discussion of the participants’ experiences. (In co-discovery, two or more people use a product together; they take turns handling controls and advising the “driver.”) Alternatively, we plan smaller focus groups of three or four people, which is the approach we used for Innovation Chain because of budget constraints.
Although these usability focus groups provide the opportunity to observe user task behavior, the findings cannot be treated like individual performance data from usability testing. In the group situation, although one person “drives” the computer, the other focus group members make comments and suggestions, so the task behavior is collaborative. Quantitative task findings (such as “Six out of eight participants had difficulty completing the task”) are not an outcome of focus group methodology, although we can and do say things like “The participants found task 1 more difficult than task 2.”
Because usability focus groups produce qualitative rather than quantitative data, the data can be challenging to analyze. Differences among the groups – both audience type and discussion content – can further complicate the analysis. Finally, the quality of the results depends heavily on the facilitator’s skill.
Participant Selection and Recruiting
In all user research, the value and the validity of the findings depend in part on how well the study participants represent the actual target audience. If the participants do not have the same characteristics as the intended audience for the product or website, we cannot know how well the collected data apply to the real situation. Therefore, Tec-Ed invests thought and time in first identifying the participant criteria and then recruiting people who meet the criteria. Recruiting for specialized areas of expertise, as we did for this project, can take two to four hours per recruited participant.
While Innovation Chain and the IEEE explored their partnering opportunity, we had access to the Boston-area membership list of the Signal Processing Society, one of the IEEE’s special-interest groups. We began recruiting using this list, and expanded our efforts to include other IEEE members in Tec-Ed’s database of participant candidates. Another reason why participant recruiting is so time-consuming is that customer lists are often out of date, requiring follow-up efforts to track down appropriate candidates; however, the IEEE membership list used in this project was more accurate than most corporate customer lists Tec-Ed has used.
Our goal was to have three or four people in each of three focus group sessions, so we recruited 15 people in all, to provide one or two back-ups for each group in case of no-shows. In studies with engineers and system administrators – or any busy, highly skilled professionals – last-minute cancellations are common. In fact, we had only one no-show; we conducted two focus group sessions with five people and one session with four. All participants were paid honoraria of $200, a typical incentive for technical professionals. However, most technical professionals agree to help with user research out of intellectual curiosity or a desire to improve technical products, not for the incentive payments.
All participants had to be IEEE members and comfortable with the web (use it at least five hours a week). We recruited participants with varying amounts of engineering design experience, ideally three to five participants each with less than 5 years of experience, 5–10 years, and over 10 years. We recruited three to five people each with industry, government, and academic jobs. We screened for diversity in size of employer, from individual consultants to large companies like Nokia, Verizon, and Texas Instruments. Finally, because we began with a list from the Signal Processing Society, we also screened to obtain diversity among the technical areas within that engineering specialty.
Tec-Ed employs participant recruiters on our staff, which enables us to provide training and ongoing feedback to build their recruiting skills. For this study, we expected the professional society involvement to reduce the time required for recruiting, but that was not the case. We believe the IEEE connection did influence candidates to accept calls from our recruiters, and it may have contributed to our excellent no-show rate. When candidates agreed to participate, they actually did so. However, we had the usual difficulty in coaxing busy engineers to leave their office or lab for a user research project.
Focus Group Activities
Conducting usability focus groups requires a team of two researchers: a facilitator and an observer/note-taker. The facilitator must concentrate on drawing out quiet group members, eliciting explanations of ambiguous or incomplete comments, and making sure everyone’s opinions are respected. Therefore, the attentiveness and perception of the note-taker are very important. In most focus groups Tec-Ed conducts (including this one), both the facilitator and the note-taker sit in the room with the participants, although the note-taker takes an unobtrusive position.
Tec-Ed also videotapes and audiotapes our focus groups to create an archival record, or to produce “clips” for client executives. Time and budget constraints usually discourage us from reviewing entire recordings during data analysis, although we use them to crosscheck our notes as needed.
The facilitator conducts the focus groups from a scripted outline that lists high-level questions and issues, with supplementary “probing questions” for each. However, because of the dynamics of the group environment, focus group scripts are not as detailed as those for laboratory testing. The facilitator must remain flexible, which in turn puts more demands for in-depth note-taking on the observer.
To ensure that the outline addressed the issues of most concern to Innovation Chain, we created and revised several drafts of the outline before the focus group sessions began. The facilitator and note-taker also used the prototype extensively, both to refine our formulation of the questions in the outline and to practice the tasks participants would perform (so that we could provide appropriate remediation during the sessions).
Each two-hour focus group session for Innovation Chain had four parts. We began with a 30-minute group discussion of how the participants currently develop and exchange engineering solutions such as models, simulations, and program code; this discussion provided insights into the environment in which the Exchange tool might be used. Some of the questions (with follow-on probing questions in brackets) the facilitator asked were:
Do you tend to develop models and simulations from scratch, or modify existing ones from other sources? [What other sources?]
Once you develop or modify models and simulations, how often do you need to re-use them? [Or do they tend to have limited use?]
Where do you save your models and simulations? Does your company or organization have a specific place where you save them so that they’re available to your coworkers? [If so, do you use it? If not, would you use one if they had one? Names of software? How likely? How often?]
What other ways do you share models and simulations? [With your co-workers? With clients or vendors? How often do you use them?]
What are some of the concerns you have when sharing models and simulations with peers, clients, or vendors? [Pitfalls/problems?]
Could you give an example of an engineering model or simulation that you’ve developed or used recently? I’m not an engineer, so try to describe it briefly in simple terms.
The participants then used a prototype of the Exchange tool to download an engineering model from the Internet. Different participants performed the tasks in this activity, while the others looked on and made suggestions. After a short break, the participants used the prototype to upload and publish a model, again trading places so that all had a chance to perform some tasks. The downloading and uploading activities were fairly complex, yet they illustrated the power of the Exchange tool.
In the wrap-up 30-minute discussion, the participants were probed for their reactions to the Exchange tool. The facilitator asked questions like:
How does this process compare with the way you currently share and exchange solutions? [How much would this tool save you time?]
How do you like the idea of exchanging interactive solutions over the Internet? [How much confidence would you have in a solution that you downloaded from this site?]
How would you feel about paying to download solutions from this site? [How would you feel about being paid for solutions you submitted? Would you prefer a flat-rate or per-use fee?]
How likely would you be to use this tool if it were available to you?
What other comments do you have about the engineering solution exchange tool you’ve seen today?
Finally, after each facilitated session was over, we invited the Innovation Chain developers to join us and answer technical questions from the participants, as well as ask their own questions. We often use this technique for focus groups and usability testing of highly technical products. It addresses developers’ concerns that usability researchers will not understand their products well enough, and it enables us to use session time to learn about agreed-upon research goals rather than field participants’ questions about product features. (We say “You can ask the developers after the session.”)
In this study, all participants stayed at least 15 minutes after their session to ask technical questions of the developers and answer developers’ follow-on questions about the engineers’ experiences with the prototype. Most participants indicated that they would be willing to be contacted again by Innovation Chain for further input.
Lessons Learned During the Focus Group Sessions
Innovation Chain originally envisioned its engineering collaborative tools being used over the Internet, through the websites of professional societies. The participants’ behavior and comments in the first focus group session caused us to question this usage model; and after the second session, we and the Innovation Chain observers were convinced it had problems.
Although participants liked the idea of using the Exchange tool to access program code or multimedia components of published articles, they said they would not submit models to the Exchange tool on a public website, because most worked for companies with concerns about proprietary information. Even within their companies, they rarely shared models and code with peers outside their own work groups.
One of the benefits of focus groups for the early stages of product design is their flexibility to adjust topics between sessions, unlike controlled surveys or performance-based usability testing. Although this approach reduces comparability between the groups, if the participants’ behavior clearly indicates the need for a change in product positioning, we can revise the script outline and collect more relevant qualitative data from the remaining sessions.
Therefore, between the second and third focus group sessions, Tec-Ed’s team held an intensive working meeting with Innovation Chain management and then revised the wrap-up discussion questions as follows:
Under what circumstances would you be interested in sharing models or simulations with your co-workers through the Exchange?
Who else might you share models with?
If you encountered a design challenge you did not have the answer to, when would you search the Exchange for a model or simulation that might help your efforts? [How would you apply that model or simulation to your design? How much confidence would you have in a model you found on the site?]
What projects within your organization do you feel would benefit from the Exchange? [Why?]
If you were reading a technical brief in an IEEE journal and learned that you could access an interactive component of the article through this Exchange, would you take the time to register and download it? [Why or why not?]
What would you call this tool? [Model and simulation exchange? Something else?]
In the third focus group session, we introduced the discussions with a scenario of using the collaborative tools on corporate intranets to share models and simulations with co-workers, rather than on the Internet. The questions thus elicited new issues and concerns about using the Exchange as an internal collaborative tool.
When a company like Innovation Chain considers a major change in product positioning, user research is typically only one of several determinants. Innovation Chain management had been considering the pros and cons of positioning the Exchange as a public publishing tool versus an internal tool for intranets. Discussions with the IEEE had motivated Innovation Chain to explore a public Exchange first, but management was already sensitive to indications – like the data from our first two focus groups – that an internal tool would be more acceptable to their target audience.
Lessons Learned After the Focus Group Sessions
After the focus group sessions, the user research team compiled its notes, supplementing them as needed by consulting the videotapes and back-up audiotapes of the sessions. For qualitative research projects like focus groups, Tec-Ed creates a text database of notes (including participant quotes) that reflect various aspects of the answers to the research questions. This method both facilitates the initial data analysis and makes it easier to “mine” the data to address follow-up questions from the design team.
The results of the usability focus groups benefited Innovation Chain in two key ways. They learned about engineers’ current use and reuse of models and simulations, which differed considerably from their expectations. And they learned that the collaborative tools needed extensive usability improvements, even for the target audience of engineers. Our findings included:
After working 50 minutes with the prototype, participants did not gain a good mental model of the Exchange process; they had difficulty understanding how the Exchange tool worked.
All groups required assistance from the facilitator to successfully download a model and submit a model.
Engineers do not generally exchange models with peers outside their work group unless the company culture specifically demands it. They tend to reuse their own code, or use portions from established engineering sources and programs.
One of the main reasons engineers do not share models and code with a wider audience is that it takes time to document a model well enough for other people to understand it, and to figure out how someone else’s model works.
Most engineers do not reuse an entire model or simulation. They are more likely to reuse and modify segments of code in order to create a model that applies directly to their problem.
Overall, while the focus group participants could envision the Exchange being a useful tool in some circumstances, they had reservations about its value in practice. As they worked with the Exchange, they didn’t easily discover many of its features and benefits. Although they thought sharing models was a good idea in theory, they had many concerns and were unsure how or whether the Exchange addressed them. Some of the perceived barriers to using an Exchange tool included:
Time investment to document models well enough for others to understand them
Time investment to understand how other peoples’ models work, when considering whether to use them
Need for models to be very specific and thus only applicable to a narrow or limited audience
Need to adapt the source code, because most models apply to a limited audience and cannot be used “as is”
Proprietary constraints on sharing models, even within a company.
Tec-Ed noted that the Exchange tool needed a more supportive user interface; in particular, we made several recommendations for improving the usability of the Exchange home page. Innovation Chain was implementing the recommendations – and planning further rounds of focus groups targeting technical and business audiences in corporations – when the dot-com recession began and the company failed to receive funding for further development.
The case history remains a good example of the flexibility of focus group research and the kinds of data usability focus groups can collect. The realistic tasks helped participants experience a complex new tool and envision using it in their own workplaces, so their discussions gave more in-depth insight into the tool’s strengths and weaknesses. When participants reacted negatively to Innovation Chain’s initial positioning of the Exchange as an Internet-based tool, we were able to change focus quickly to describe an intranet context, while continuing to observe the same task behavior.
As always, we learned other lessons that improved our methodology:
We resolved (again) to allow more time for recruiting participants who are technical users rather than members of the general population.
We confirmed our decision to assign an especially skilled observer to the project. Note-taking in focus groups is always more demanding than in studies with one participant at a time; and the highly technical subject matter made this project more challenging.
The user research team obtained coaching in design engineering terminology before conducting the sessions, another practice we will continue in the future. User researchers need not be experts in the product domain, but some familiarity with the users’ language helps us collect richer qualitative data.
Above all, these usability focus groups helped Innovation Chain better understand the needs of its target audience while exploring their opinions of its new ideas for engineering collaborative tools.