Chapter 12. Debrief Sessions
There are three classes of people: Those who see. Those who see when they are shown. Those who do not see.
—Leondardo da Vinci
Research is about observing tasks, questioning hypotheses, and validating concepts to define a more successful product. No research is complete without sharing observations. And sharing starts in the field with debrief sessions. While the remainder of this book looks at the organization and communication of research data, this chapter focuses specifically on communication that occurs continually during research.
What Are Debrief Sessions?
In Chapter 6, we described the pacing and cadence of schedules, including the “recharging” time between research sessions where prototypes are reset and the team grabs a cup of coffee. Debriefs, simply put, are conversations you have during those times. These debrief sessions don’t need to be planned, though their purpose and goals are consistent.
Planned Debrief Sessions
Planned debrief sessions may be as short as five minutes after a research session, or as formal as an end-of-day discussion with all observers and business stakeholders. In either case, the goal of the session is to quickly go around the group and identify what went well, what could be improved upon, and actionable next steps for the duration of the research initiative.
Ad Hoc Debrief Sessions
Ad hoc debrief sessions can take place at any time, though we recommend waiting until after a research session is over. More impromptu debrief sessions may take place after a particularly good or bad session, or if new business requirements are identified and need to be brought to the project team’s attention.
Why Debrief?
Nobody can be everywhere. And nobody can remember everything. And no single perspective is perfect or right. With project teams consisting of multiple skill sets—from design, research, development, and strategy—each observer has a different perspective with which they interpret their observations (see Figure 12-1). Each perspective carries different concerns, needs, and questions. Add to this the variety of business stakeholders, and the complexity multiplies exponentially. A familiar metaphor for this is the multiple accounts that different eyewitnesses bring to a case; their personal interpretations vary depending on their perspective, biases, and focus. Debrief sessions allow project teams to align on findings and use a common language.
How do you debrief with your team throughout research initiatives?
When we launched Reebok.mx, I was so focused on not repeating the mistakes that happened when we launched adidas.mx that I took for granted the things that went well and lost sight of them. I was so focused on the trees that I lost sight of the forest.
Our biggest hurdle when launching Reebok.mx was the marketing campaign: it was go-live day, everything was set up, the site had been tested thoroughly, teams had traveled, product was online, and product campaigns were on...what could go wrong? The marketing campaigns were not fully active, which meant traffic was scarce despite weekly meetings for two months to make sure everything was ready.
I was expecting for everyone to do their part. By meeting with my team afterward I learned that my expectations were not communicated properly. The people collaborating with me were not aware of the responsibilities I had placed on them but didn’t clearly communicate to them. I also learned that sending emails does not equal communicating expectations.
Had I not met with them, I would not have seen this. It’s not about being Captain Hindsight, it’s about creating a space for the people that walk with you to share their perspective of an experience. Despite being on the same project, each of our perspectives is a different one. Their perspective will open up your boundaries and your confirmation bias to realize what role you could have played that could have made the difference.
For me it was also important to be in listening mode. When receiving feedback, my team didn’t want my justification on why things went the way they did. They wanted their side of the story to be heard and understood. And from my side, listening to their stories allowed me to see where I took wrong turns.
In the end, having this debriefing session helped us understand and identify where the gaps were, united us as a team, and prevented the snowball effect of placing blame on others instead of accepting responsibility on the role each of us played in the project. It’s a place where we learn from each other and ourselves, allowing us to grow individually as well as a team.
Validate Goals
In Chapter 2 we discussed how research involves asking good questions. And in Chapter 6 we discussed the importance of crafting a hypothesis. One of the greatest benefits of the debrief session is getting to see in real time how those questions and hypotheses are measured.
When crafting questions, you might think you have an open dialogue, only to realize that your questions are leading or limited in scope. Debrief sessions help you step away from your own perspective and, by playing back tape or simply discussing with your team, fine-tune questions in real time. Discussion guides are exactly that—guides—and while there are sometimes concerns about changing tactics midway through research, it is better to realign questions than to gather less-than-ideal feedback.
Your hypothesis can also be tested in real time. For those with scientific backgrounds, this differs from a double blind study, where observers do not know the goals of the research and therefore cannot impact data. UX research is not clinical, and does not pretend to adhere to strict protocols. Being able to see how results unfold in real time gives you insight into how much more research is necessary. If you have 10 participants scheduled and have enough answers after 4, a debrief session could save you time and money. On the other hand, realizing after 8 participants that there are still unknown questions can give you enough lead time to schedule additional sessions.
Keep the Home Office Informed
It is rare that everyone on a project team can travel for research. You also have the constraint of having only two to three people in the same room as the participant. Thus, it is very important to keep your project team and business stakeholders informed throughout a research initiative.
Moderated research initiatives
Moderated research initiatives, especially for research involving travel, require keeping the home office informed. This provides a sense of security that research is happening and the process is moving forward. The following popular methods of communicating with the home office are listed from easiest to more complex:
A simple email between sessions can keep your project team informed (see Figure 12-2). A message like “We just met with Bill, head of accounting, and got some good validation on the task flow about procurement. Something interesting came up around liability we should ask the client about later” is more than enough.
- Phone
Email is great for brevity, but sometimes a phone call is preferred. This may be a scheduled call or a “drop-in” to a project manager or team lead. The information doesn’t need to be any more detailed than the email. If you are looking to confirm timely information, such as scheduling, phone is preferred for its speed.
- Video recap
For longer initiatives, it is sometimes preferred to “see” your project team, even remotely. With the advent of screen-sharing and video conference tools, you can schedule end-of-day recaps with your project teams. This is particularly helpful with teams located across multiple time zones, as it can help build rapport. Figure 12-3 illustrates a remote debrief session.
Remote research implications
When performing remote, unmoderated research, meeting to debrief is often easier because travel is not involved. The challenge lies in the multiple, parallel paths research can take. With remote research, you are no longer limited to two researchers for each participant; multiple participants can walk through a remote test simultaneously.
For remote research, it is important to have team members tasked with reviewing incoming content as quickly as possible. Then, by scheduling daily meetings, the team can quickly verify the information gathered is appropriate and raise any concerns or proposed changes in a timely manner.
Quantitative research implications
Debrief sessions for quantitative research are very similar to those for remote research. In quantitative research, data is being gathered passively and across many streams. Daily, or even weekly, touchpoints allow for regularly checking in on the data and sharing it with a project team. Constant communication is critical in high-volume environments, such as commerce platforms or sales tools. In these situations, A/B tests can provide feedback in a matter of hours or days (depending on the test’s scale) and can validate a decision or prevent a larger problem from arising.
Keep clients informed
Ultimately we all work for a client. This may be our own organization if we work in house, or it may be a third party. Keeping the clients informed of the status of the research is just as important as keeping your project team informed. Think back to the challenges you may have faced selling research to managers and clients. Debrief sessions offer an opportunity to see the value of this work in real time and can make it easier to sell future research.
The tone of a client debrief may change slightly. Whereas with your project team you may discuss findings, observations, and updates in real time, you often want to provide a higher-level view to clients. Despite this, clients will ask what results you are observing, and there is a fine line between being too vague and too detailed while a research initiative is under way. A common way to ease client debrief sessions is by inviting clients to observe research in real time. This way, they become more active members of the research team and can act as your champion with the broader client team.
Keep the team informed
For research initiatives involving many teams working simultaneously, daily debrief sessions keep everyone in sync. It allows teams to identify challenges that are localized to a specific region because of technology constraints, cultural differences, or something else altogether. If a challenge, observation, or opportunity arises across multiple locations, it allows the team to address the hurdle in a uniform manner.
Track Highlights in Real Time
For any research initiative you will collect a lot of data. For a single 60-minute contextual inquiry, it is not uncommon to leave with 75–100 individual data points, or individual pieces of information. Chapter 13 will go into more detail on the analysis and organization of this information. Multiply this across a number of participants and a pool of observers, and you can be faced with thousands of data points from a simple five-person study. With this in mind, it is critical to start tracking data early and often. Next we offer a few methods that help with this process. And while specific tools are mentioned, these are meant to be representative of the type of tools available, not prescriptive recommendations.
Spreadsheets
Spreadsheets can be so cumbersome that many designers are hesitant to use them. But when you’re looking at multiple participants and observers, spreadsheets are often the gold standard of organizing separate notes. While there is no wrong way to set up a data collection spreadsheet, Figure 12-4 illustrates a sample spreadsheet.
Affinity diagrams
Switching from digital to analog, sometimes a good wall and stack of sticky notes is sufficient for real-time tracking. Taking time each day to write down key findings and observations is often enough to stay on top of research analysis later on. While one challenge with affinity diagrams is the requirement for a shared, physical space, this also acts to your benefit, because business stakeholders can drop in and engage in the process in a curated manner. Figure 12-5 illustrates a project room covered in sticky notes.
Digital affinity diagrams
Technology allows you to adapt traditional affinity diagrams in more digital ways. Trello and other digital tools (Figure 12-6) enable remote sharing and collaboration.
Mural.ly is another collaborative tool that enables the creation of poster spaces for sharing inspiration (Figure 12-7). And for those familiar with the Google tools, Google Drawing is one that can be adapted for remote collaboration. Each of these tools has its pros and cons, and we encourage you to try as many as possible to see what works for you and your team.
Highlight reel
For the truly adventurous, a highlight reel (Figure 12-8) collects video and audio snippets from a research initiative to tell a piece of the story. This requires recording and the necessary consent forms, and is often used in larger projects to sell the research upstream to senior stakeholders. During research, it is common for observers to annotate where something interesting or exciting happens. This makes it easier to find that clip later for editing and integration into a highlight reel. Reels don’t need to last longer than five minutes. Tracking the content early is key to being able to consolidate information later on.
Debrief Informs Iteration
So far we have talked about the value of debrief sessions for keeping your project teams informed. Equally important are the iterations in process that come from frequent, open communication. Research, like design, is iterative, and this process of evaluating and change happens only through sharing findings and concerns with team members and stakeholders.
Research in the Open
Research is part of the collaborative design process. Collaborating in research starts by sharing your approach and findings early and often. While it can be frightening to share raw, unfiltered content with clients, providing access to this material early and often can help build relationships and accelerate synthesis later on.
For a client, access to video or an end-of-day email summarizing activities brings transparency to the process. This may highlight new opportunities for future validation and expose gaps in your or the client’s understanding of the approach. Some practitioners shy away from sharing raw data because information out of context might create more questions than it answers. While that outcome is possible, framing the information with the goals can mitigate the risk.
For your project team, communicating your research allows checks and balances. Whether you are performing multiple parallel paths of research or simply communicating to other team members, open communication allows an escalation of concerns early and often, mitigating risk.
Mitigate Risk
Nobody wants to go through a weeks-long research initiative only to find out something went wrong. Debrief sessions allow for iterating on technology and process to avoid larger issues down the line.
This issue may be something as minor as the microphone not recording audio or a poor phone connection. It may also be something more significant like the wrong order of questions or a persistent bug in a prototype. By having these conversations early and often, you can adjust the protocol on the fly and have more actionable results later on. We recommend you keep track of when and what changes are made to a research protocol, as any sudden spikes or changes in data should be related back to changes in how research was performed.
Exercise: Discussing Observations
The information that gets discussed during debrief sessions is best sourced from the team that was involved in observing the research. Getting comfortable with sending out debrief notes involves gathering everyone together to discuss what each person saw. The following exercise describes how to facilitate this meeting.
Gather the team.
Gather your team to discuss the observations from research, stakeholder meetings, or team meetings that recently occurred.
Write down observations.
Each team member should write down three to five observations that stood out during the sessions being reviewed. This should be done in silence to avoid influencing one another.
Gather and discuss.
Organize the observations using a whiteboard or online collaboration space. Go around and have each person explain what made them write down their observations. With one person as the note taker, ask for additional feedback for each observation to see if others noticed anything that stood out.
Share.
Save the notes to a shared network. Sharing within your organization and other teams on related projects is suggested, though in practice is often difficult.
Parting Thoughts
Debrief sessions are not difficult, but they do require a degree of intent and planning. When thinking of a debrief session, don’t try to make it a formal presentation or review. Instead, think of it as a “How was your day?” discussion. In this way, you have a conversation with your team and stakeholders. Everyone will feel more engaged and more motivated to contribute to the project as a whole.
With this in mind, we will now move into Part IV, where we will explore the nuances of how to communicate your research findings to teams and stakeholders alike.