This chapter addresses the following questions:
- Given all the work invested in research, experience modeling, and design, what is the purpose of testing?
- How and when to test?
- What to test and who should perform the testing?
- What to do with the findings?
Launching a new product or a product redesign, is a significant investment for most companies. Because design plays a pivotal role in the success of many products, testing its effectiveness is considered an important activity before, during, and after the product is launched. And that is why most companies invest in testing.
Testing focuses on two primary dimensions of the design:
- Strategic (macro) testing: This type of testing validates the entire design approach at a high level to ensure that it matches agreed design principles, high-level business requirements, and target audience expectations. This is a macro-level approach that is mostly concerned with marketing aspects, such as competitive edge and market domination.
- Tactical (micro) testing: This type of testing is concerned with the details of interaction patterns. It includes a structured examination and micro-level validation of usability, efficiency, and the effectiveness of and satisfaction with the user experience under various conditions of use. In some industries, such as aerospace, automotive, or healthcare, design testing is literally a matter of life and death.
However, the effectiveness of macro testing as a predictor of end-product success is often challenged. Sometimes the experience approach fails the testing, and yet, both designer and company share a strong belief in the appropriateness of their approach. The choice is between taking the safe path and modifying the experience, or following instincts and taking a risky bet.
As was discussed in previous chapters, design is a mix of subjective qualities derived from the arts, and objective qualities derived from engineering and science. The objective-subjective balance makes design challenging to test. People respond to experiences in unpredictable and sometimes irrational ways. Consequently, while testing is unquestionably effective when it comes to improving design engineering and usability, the overall power of testing to predict the contribution of the design to the success or failure of a product is disputable.
The introduction of the iPhone in 2007 launched a usability controversy. Typing on glass--how usable is that?
To judge by the enormous proliferation of mobile devices equipped with glass screens and by the fact that few have a physical keyboard, one might easily conclude that typing on glass is not only possible, but even superior to typing on a physical keyboard, the experience previously championed by Blackberry.
Getting rid of the physical keyboard, which has been an integral element of cell phone design since the inception of the iPhone, certainly has had advantages. Steve Jobs outlined these in the 2007 iPhone-launch keynote speech:
And they all have these control buttons that are fixed in plastic and are the same for every application. Well, every application wants a slightly different user interface, a slightly optimized set of buttons, just for it.
And what happens if you think of a great idea six months from now? You can't run around and add a button to these things. They're already shipped.
So what do you do?
...Now, how are we gonna communicate this?... We're gonna use the best pointing device in the world. We're gonna use a pointing device that we're all born with--we're born with ten of them. We're gonna use our fingers.
We're gonna touch this with our fingers. And we have invented a new technology called multi-touch, which is phenomenal.
It works like magic."
Despite these words, typing on glass can be an unpleasant tactile sensation, and slow compared to physical keyboards. Blackberry devices have been especially popular with business people due to the ease and convenience of typing long emails. Typing long documents on a glass screen has been described by some as "torturous." And yet, here we are: Not going back to keyboards, but witnessing the rise of conversational interfaces as a direct response to the typing-on-glass problem and in an effort to bypass typing all together.
What this story highlights is an inherent tension between experience design and usability testing, between strategic and tactical testing. Sometimes, when tactical testing shows that a design is wrong, strategic design testing forcefully claims the opposite, or vice-versa. Of course, both types of testing may also align in their assessment of the design.
The iPhone experience, for example, had to be considered holistically. From a wider perspective, typing is only one of limitless entertainment and productivity activities a mobile user can perform by engaging directly with beautifully designed apps. Direct and instantaneous interaction between the device and one's fingers further enhances perceptions of having fun and freedom. Compared to this, the input from the keyboard is associated with activities that are more formal, businesslike, and work-related.
In a company different from Apple, if a designer came up with an idea for a device like the iPhone, it is quite possible that after rigorous usability testing, the entire concept would have been scrapped. Indeed, our world today might have been very different.
The example of the type-on-glass screen may raise a concern that experience testing is potentially an innovation-busting activity. For the most part, however, testing is a foundational pillar of the user-centered-design methodology. When integrated into the design process, testing provides a practical and reliable means of improving the design, and the experience it delivers.