Chapter 12: Testing Your Prototype

Testing is often the ultimate goal of prototyping. Even if it’s not the ultimate goal, then it should be one of the major milestones along the way.

Usability testing is an entire industry. Usability has its own professional organizations: the UPA (Usability Professionals Organization) and the ACM’s SIGCHI. Many books have been written on usability testing, notably Usability Engineering by Jakob Nielsen and A Practical Guide to Usability Testing by Joseph Dumans and Janice Redish. There’s even an entire U.S. government site dedicated to the subject, http://usability.gov.

With all this information readily available, it’s a wonder that so many mistakes are commonly made during usability testing. Well, not really. Usability testing isn’t one-size-fits-all.

Common Mistakes

There are different methods, such as in-person and remote testing. There are different tools, like Silverback, Morea, or just a simple video camera mounted on a tripod. And there seem to be more ways to run and report a study than Ben & Jerry’s flavors of ice cream.

While this chapter won’t turn you into a seasoned usability professional overnight (only experience can do that), it will highlight a number of common mistakes and tell you how to avoid them.

Mistake #1: Usability Testing Is a Process, Not an Event

The process of usability testing has a number of equally important pieces, including planning, recruiting, test moderation, analysis, and reporting.

Just sitting someone down in front of a product or service and watching him use it isn’t that difficult. Getting quality results from testing, however, that’s another story.

Understanding that usability testing is a process and that each piece has an important role to play will go a long way toward improving the quality of your results.

Mistake #2: Poor Planning

When planning your usability test, it’s important to understand the who, what, when, where, why, and how of testing.

The first question to ask yourself is: Why are you doing usability testing? Usability testing is great for finding out how something performs, based on measuring time, effort, and satisfaction.

If, on the other hand, you want to get feedback on something less objective or measurable, like whether they like visual design option A better than visual design B, usability testing isn’t the best method. This could be something you included at the end of your usability test, but it isn’t a reason to choose usability testing as a research method.

Determine whom you want to test. Who uses the product or service? What are their behaviors? Why would they use it in the first place?

Set a testing date and work backward from that. Our usability testing cycle at Messagefirst is typically three to four weeks from initial planning to producing a report. A typical timeline might look something like this:

• Week 1: Recruit screener and test scenario development.
• Week 2: Recruit participants, finish testing scenarios and prototype.
• Week 3: Test prototype and begin analysis.
• Week 4: Finish analysis and report findings.

Mistakes in planning often result in underqualified test participants, issues with the prototype, and ultimately poor quality research results. Give yourself enough time to plan properly. It’s really that simple.

Mistake #3: Not Recruiting the Right Participants

We have a saying in the field of research: “Bad data in, bad data out.” The whole point of usability testing is to see how your design works through the eyes of the people who will use it. If you recruit the wrong people, or don’t recruit the right people, then you’re getting the wrong data.

What’s worse than not seeing how your design works through the eyes of your end users? Seeing it through the eyes of someone who isn’t your end user.

A few years ago, I was conducting usability testing on a large content site focused on finding something to do in LA for the weekend. We were looking for participants who were regular users of social network sites and had created some type of user-generated content (e.g., they blogged and wrote reviews for products online).

Most of the participants the recruiting firm sent us fit this profile perfectly. And we learned a great deal from them. One, however, didn’t even have an email address. As you might imagine, seeing the site through this person’s eyes was essentially useless to us.

The best way to ensure highly qualified participants is to either do your own recruiting with a very detailed screener or hire a professional. Currently, at Messagefirst, we do all of our own recruiting. You can download an example of the screener we use at http://rosenfeldmedia.com/books/downloads/prototyping/Sample_Screener.doc.

Mistake #4: Poorly Formed Research Questions

Formulating research questions correctly is one of the most difficult challenges of usability testing. The key is to get the answer to your question without actually asking the question. Confusing enough?

Let’s take the example of the site focused on helping you find something to do in LA for the weekend. The prototype we were testing had content for finding movies, live music, local events, and restaurants.

Now, we could have asked participants to plan dinner and a movie this weekend using the site. That would have been direct and leading. But it wouldn’t map to how they plan doing things with friends. It also wouldn’t tell us how they used the site, but rather how we wanted them to use the site.

During the initial discussion with participants, we found they “Looked for something to do” with friends. That left it open to going to a movie, seeing a live music performance, having drinks, going to dinner, or checking out a music festival.

Instead of asking them “How would you plan dinner and a movie with friends using this site?” we asked “How would you use this site to find something to do with your friends this weekend?”

See the difference? The first is very direct, dinner and a movie, with a fairly predictable outcome. The second is a more open-ended question, allowing them to show us how they would use the site. This is a true test of how the site will perform for them when they are at home, away from us, in the real world, day-to-day.

Mistake #5: Poor Test Moderation

A good usability test moderator is worth his or her weight in gold platinum. Moderating usability tests is a learned skill. In fact, I’d go so far as to call it a craft. Like any craft, it takes a great deal of training, practice, and time. It’s not something you pick up overnight. While I believe most people can moderate usability tests, brilliant moderators are rare, and frankly, some people should never be allowed to moderate.

What makes a good moderator? Balance.

A good moderator knows how to balance being a silent fly on the wall with just enough dialogue to keep things moving. They know how to extract the right level of detail without going too deep into the weeds. They know when to let the participant explore and when to pull them back. They know how to get the answer to the question you want without asking the question.

The biggest mistakes made by moderators are talking too much, answering for the participant, and asking direct and leading questions. Sit through a few usability-testing sessions and the skill level of the moderator will become immediately obvious.

The best way to improve your moderating skills is to find a mentor, practice, listen to their critique, and practice some more. You can also try taping yourself and then playing back the audio or videotape to evaluate yourself. You might be surprised what you see and hear.

Mistake #6: Picking the Wrong Method to Communicate Findings and Recommendations

Nobody is going to read that 10–20-page research report you’re thinking of writing. Its sole purpose in life is to act as proof that a research study was done. It will remain isolated in the virtual world of bits and bytes. That’s it. And yet, you still have to write it.

If they won’t read the written report, then how do you communicate the findings? That really depends on the environment and the team you’re working with.

Quick review sessions between participants at the end of each day work especially well. These quick review sessions can be used to discuss key takeaways from each participant and to identify patterns as they evolve over the course of the day.

A PowerPoint or Keynote presentation with a summary of the report findings works well. Video clips highlighting key interesting moments during testing are especially effective. It’s one thing to hear someone tell you what a participant did during testing. Watching it on video, however, is the next best thing to experiencing it firsthand.

Preparing for a Usability Test

Remember the first guiding principle of prototyping from Chapter 4: “Understand your audience and intent?” That’s also the first step in preparing for usability testing. Just like prototyping, this tenet drives every other decision.

Work with your product team to determine the key characteristics and behaviors you’re looking for in participants. It’s equally important to determine what characteristics and behaviors you don’t want. Use this information to create a detailed screener.

A sample screener can be found at http://rosenfeldmedia.com/books/downloads/prototyping/Sample_Screener.doc, which you can revise for your own needs and use freely.

If you’re going to audio or video record the sessions, it’s a good idea to provide participants with a waiver and consent form. This gives you the freedom to review the recordings later, while protecting them from their recording ending up on YouTube.

A sample waiver and consent form can be found at http://rosenfeldmedia.com/books/downloads/prototyping/Waiver.doc, which you can revise for your own needs and use freely.

Knowing the intent of your test will inform the scenarios, research questions, and the prototype. These three elements—scenarios, research questions, and prototype—tend to feed off each other.

You might start by designing the scenarios and research questions and using those to drive the prototype. At some point, the prototype will create some additional research questions or possibly even an entirely new scenario you want to test. Just make sure they’re in sync with each other in time for testing.

A good rule of thumb is to limit the length of testing sessions to 45–60 minutes. This should be enough time to test five to six key scenarios and not exceed the attention span of your participants. We typically conduct 45-minute sessions with 30-minute breaks in between. Using this format, we’re able to conduct six to eight test sessions per day.

Having a 30-minute break between participants provides enough flexibility for participants who show up late (which will happen) or sessions that run longer than 45 minutes (which will also happen).

Additionally, the 30-minute break provides enough time to do a quick debriefing among participants and make any necessary adjustments to the test scenarios or prototype.

Design Test Scenarios

Test scenarios can be either very specific to determine whether or not a participant can access a particular feature on the site, or exploratory in order to gain insight into a participant’s overall approach toward solving his or her goal.

In the example Web site used to find something to do in LA for the weekend, the participant’s goal wasn’t to find a restaurant or even a band. Rather, the participant’s goal was to find something entertaining to do with friends. Finding a restaurant and live music was just the process to achieve an enjoyable evening with friends.

Good test scenario design focuses on the goal, while accommodating the activities and the process required to accomplish the goal. A good way to do this is use a combination of an introduction to a scenario with a list of observations or key questions.

If the goal is to find something entertaining to do for the weekend with friends, you might frame the test scenario like this:

“You said earlier that you and your friends were looking for something to do this weekend that might include going out for [dinner, drinks, live music...] and that you wanted to check out that new club [...]. Show me how you might plan that using this Web site.”

The [...] are key points where you will insert important contextual information gathered from your opening discussion with the participants.

Next, you could include a few key questions or observations, like the following:

• How do they start (for example, search, browse)?
• Do they look for a place to eat or entertainment first?
• How do they determine which restaurant to go to?
• How do they select what kind of entertainment they want?
• How do they plan (for example, phone calls, SMS, email, twitter)?
• Do they see the map showing locations? Thoughts?

Finally, you’ll want to include some type of scoring system for the scenario. We tend to use a sliding scale of 1–5, which is used by both the participant and moderator to rate the scenario.

Test Your Prototype

Soliciting feedback from participants is easier when they’re comfortable. One of the tricks I often use is to have the moderator greet the participants, introduce himself (or herself), offer them something to drink, and start a conversation while walking them to the testing room. This gives the moderator a chance to build some rapport with the participants and get them comfortable as quickly as possible.

I’ve found remote participants are pretty comfortable from the beginning. I imagine it helps being in their own environment instead of coming into a strange place and sitting in a room with a camera pointed at them, while someone watches from another room.

Once they’re comfortable, I start with an opening dialogue asking them to describe their experiences related to whatever we’re going to test that day. For example, if I wanted to see how they would use a site to find something to do in LA for the weekend, I might ask them to tell me about the last time they planned an evening out with friends.

I might be interested in listening for details like how the planning occurred and if they used email, text messaging, phone calls, or other methods. I might also want to listen for details on what types of activities they considered, such as a movie, dinner, drinks, or going to see a live band.

After you’ve gathered some information from the opening dialogue, use that information to provide context.

Pretend you have a design challenge to help make their planning easier. Imagine you came up with a concept you want to test called “What’s Nearby.” The idea is that once someone has found a place of interest, like a restaurant, you show them other places to eat, events, and activities that are nearby. Items nearby would be displayed on a map with star ratings and the number of reviews.

During the opening dialogue, the participant told you he was heading to In and Out Burger and then over to see a local band play. Perfect. You can use that information to frame the scenario and provide real context for him.

I might start off by saying something like, “Remember how you said you had planned to go to In and Out Burger with your friends and then see a live band afterward? Well, I’m going to show you a page and ask you to show me how you might find the In and Out Burger location you want along with anything else close by you might be interested in.”

The framing of the scenario includes a couple of key details: real context based on their life and an activity that I can use to test our design concept. Additionally, to keep from being too direct, you’ll notice that I asked him to find something “close by” rather than “nearby.”

While a subtle nuance, it’s a careful balance between having him perform his activity of finding something of interest close to In and Out Burger without giving away the farm.

Tip Don’t Touch This

If you’re testing a prototype that is on a testing or development server, make sure that the development team doesn’t touch it the day of testing. I’ve experienced this once or twice. It’s not only frustrating, but also a waste of time for the test participant and the client.

Record Observations and Feedback

Playing the role of moderator and note taker is a daunting task. The best solution is to have one person focus on moderating while another person takes notes remotely. This allows the moderator to focus his or her attention on the participant, while the note taker can focus on taking notes.

When recording notes on your observations, it’s best to over-record rather than under-record. If you over-record, you can filter that out during analysis. If you under-record, you may miss a very important piece of information. So, record everything.

As I mentioned earlier, we use a sliding scale of 1–5 to rate each scenario. Both participants and moderators use the same scale. At the end of each scenario, the moderator will ask something like “On a scale of one to five, where one is very difficult and five is very easy, how would you rate finding something to do this weekend in LA?”

The participant’s rating is recorded, and the moderator scores the task separately. The moderator’s score is not shared with the participant.

The moderator’s score should be based on measurable elements like time and effort. An example is something I refer to as pogo-sticking. During pogo-sticking, the participant bounces back and forth, down one path, then back up, then down another, then back up again. Specific metrics like this should be established before testing.

The participant’s score is more subjective, qualitative, and a measurement of his or her satisfaction with completing the task. The moderator’s score is more objective and quantitative, as it is based on something more measurable, such as time and effort.

Using some type of recording system, like Silverback, Morea, or even a camcorder mounted on a tripod is a great way to ensure that you won’t miss anything. If you miss an important detail, you can always go back to the recording and review it. It’s also a great way to double-check your notes.

Note-taking methods vary greatly from paper, to Excel spreadsheets, to custom research frameworks.

Morea’s has built-in capabilities for note-taking and flagging keyframes during recording. While these features are nice, I’ve always found the interface and interaction of Morea a bit clunky and awkward. Additionally, Morea doesn’t work on a Mac. If you’re only concerned about Windows participants, then it’s a good solution. Our work often requires testing both Mac and Windows participants. So a non-Mac version of Morea is a deal breaker for us.

Since the quality of our findings is paramount, we try to filter out any variable not directly related to the system we’re testing. The experience of Firefox on Windows is not the same as Firefox on the Mac. So we put Mac participants on Macs and Windows participants on Windows.

We use Intel-based iMacs and Silverback to record our audio and video. This solution allows us to run both Mac and Windows on one machine. The screen sharing built into Mac OS X allows us to view the remote machine without the need for additional hardware or software.

When performing remote testing, screen sharing can be done with a number of solutions, including NetMeeting, WebEx, and Adobe Acrobat Connect.

When taking notes, I’ve found tracking the following three pieces of information make analysis easier:

• The observation: Something seen or heard.
• Time stamp: At what point in the video did it happen.
• Tags: Keywords to describe the observation.

While we’ve created our own custom framework, which makes capturing and analyzing this information easier, you can record these data points on paper or in a spreadsheet.

Analyze and Determine Next Steps

Most of the time, when I start the analysis process, I have a short list of significant issues that are already at the top of my mind. These are typically things I’ve seen repeatedly in testing. This short list is just a starting point.

We recently did a number of on-site and remote testing sessions for a client. We knew we had a great deal of information recorded, but weren’t really aware of how much until we printed each observation out on a Post-it note and stuck it on our walls. It turns out we had recorded over 1,000 data points.

Seeing 1,000 points of data surrounding us on every wall made it much easier to look at not only the entire set, but also to dig in and identify themes and sub-themes.

It took us several days to work through 1,000 data points, but I don’t believe we could have been as effective analyzing both the depth and breadth any other way.

Figure 12.1 (Open in Flickr)

1,000 points of data.

After you have identified your themes, you can determine the significance of those themes and individual findings within. How do you determine how significant a finding is?

I’ve seen a number of ways to determine significance. Probably the most common is to base it on frequency (how often it occurred) and severity or learnability (was it a problem every time, or did the severity decrease with continued use?).

I prefer a method I developed a few years ago, which combines the measurement of significance to the customer, value to the business, and technical feasibility of solving the issue (assuming a design solution is available). This method provides a more holistic measurement and a greater ability to prioritize findings.

A Final Word

Several years ago I had a discussion with a usability consultant who referred to summative usability testing—testing that happens at the end. I’ve never really believed in summative testing, as I don’t believe that product development is ever finished. It’s an ongoing, living cycle.

Some might find this discouraging—the notion of never being finished. I see it as a great opportunity to continue the learning process, building on past knowledge and experience, iterating and refining my craft by integrating that knowledge and experience.

I hope this book has helped you learn how prototyping is an iterative process. You generate design concepts. You prototype them. You test them. You discover what works, what needs to be refined, and opportunities for new ideas.

I also hope that whether you are just starting out prototyping, or consider yourself a seasoned veteran, you’ve been able to learn a few tips and tricks to help step up your game, making it easier to sell prototyping to your boss or clients and improve your prototyping skills.

And finally, I’m always interested in learning new methods and techniques myself. If you have a method or technique you’d like to share, I’d love to hear about it. Who knows? Maybe your tip will find its way into a future edition (hint, hint).

Summary

Testing your prototypes is a key step in the prototyping process. Next time you’re testing your prototype, keep the following in mind:

• It’s a process, not an event.
• Your results are only as good as your participants. Make sure that you’re recruiting the right people.
• Good test moderation is a skill that takes years to develop. So practice—a lot.
• Test your prototype before your first day of testing to make sure it works.
• Pick the right method to communicate your findings.
• Video is very powerful.