Chapter 13. Releasing Multimodal Products: Validation and Learning
AFTER DOZENS OF ITERATIONS, ruthless prioritization, countless cups of coffee, and whirlwind factory visits, your completely operational Death Star multimodal product is ready to meet the Rebel Alliance your users. It may feel like discovery was completed many phases ago, but a new kind of discovery may be just beginning.
Release methodologies like dogfooding, alpha, and beta play a larger role in hardware product releases: they give designers a great opportunity to validate real-world usage that can be difficult without both functional products and the real-life contexts of users. Running acceptance tests to validate multimodal designs can provide valuable insight that can only be found “in the field.”
During full releases of multimodal products, initial user experiences reveal unexpected adoption behaviors. When Siri first came out, there were countless stories of how users tried to “stump” the service only to discover funny Easter eggs in her repertoire. Further releases broadened this playful aspect of Siri, and continue to both delight users, and also deepen their learning of the service through play.
This is one of the most rewarding parts of designing multimodal interactions. People will always manage to surprise you. As designers, we get to try to surprise and delight them back.
Release Is About Process
The industrial designer Eva Zeisel said, “As a designer, it’s your job to give people gifts every day.”
One of the ways to make sure that your users are able to receive those gifts is to have a product that is easy to learn and easier still to live with. Releasing products is when you get the chance to see if you’re really doing that. There are many different approaches, and they are generally tailored to both the product type and organizational profile of the product teams. There are also many books, and much expert knowledge about launching products; clearly this book isn’t going to substitute for that. What we hope to do is point out a few of the aspects of launching product that matter the most to multimodal design.
The release process is, well, a process. It’s not a one-moment event. A common approach is to start with smaller, more familiar groups of users, and gradually scale up that group of users to better represent the target user audience. Earlier stage launches are a chance to validate the core ideas and value of products as well as identify critical defects or gaps in the product experience. Later stage launches test the appeal of the product to wider, more diverse audiences and identify stresses and defects of system integrity across contexts and scale. It might also show you whether or not people will understand and use it as intended, and possibly even surface interesting new uses.
Alpha Release
The first step is the Alpha release, which might start with the appetizing name dogfooding (coming from “eat your own dogfood”—i.e., if it’s so tasty, you try it!). Not everything is polished, and not all features are complete. That’s OK—this is the time to get initial feedback and observations, as well as getting your team primed for responding to those as you move forward with later releases. Generally speaking, this is where you start to ask, does the product do what you’d hoped? What do the users expect and need? Does it fail at any points?
The earliest alpha stage starts with the project team but quickly includes as many other colleagues as possible. Once a level of initial confidence is reached, the circle is expanded to friends and family. This may be a tough hurdle, but making it happen as soon as possible is valuable. This is also often the stage at which many product partners are given access, and coordinating that may be an effort in itself.
Organizing alpha
Create a list of all participant groups, including any legal agreements for each type.
Employees are probably covered, but there may need to be releases and non-disclosure agreements for friends and family. If you are working in a compliance-heavy field like health or finance, this legal part is especially important and likely to include other regulatory hurdles. Your team better have a robust plan to address those.
Define testing goals and success metrics.
You should start with a list of success criteria for features overall, as well as a list of use cases. In some cases, lifestyle fit may be an important factor.
Develop a set of objectives for each type of participant.
This is the time to test user experiences and general functionality. Since multimodality is so context-dependent, try to test in real-world conditions or as close as you can get. If there are particular features that need to be understood, it helps to create a combination of direction and education for first-time users.
Create a system for soliciting and processing feedback.
Make sure that participants are motivated to use the product as much as possible and are given a clear path to submit feedback. Explicitly calling for particular types of feedback may help prompt users, though the fact that the feedback was prompted should be noted. See “Validation and Feedback” for more ideas.
Once the feedback starts to come in, it’s important to parse and prioritize it. Interpretation is the better part of research.
Learning from Alpha
Identify critical user errors, or main weaknesses that could jeopardize product stability and viability.
If something sticks out as critical, make sure that it is flagged, communicated, and given the right weight. These are larger issues, at the level of product requirements and definitions.
Prioritize bugs and assign them to responsible parties.
No doubt your team has a bug tracking system in place; that’s where these go. It’s probably a good idea to split the issues into addressable fixes; known issues for support; and challenges for user education and support.
Organize efforts with partners by addressing any needs and concerns that lie outside of bug fixes.
You’re now starting efforts that probably fall outside of your bug tracker, and dealing with partners’ needs is likely to include some overlap between bugs and contractual agreements. Be aware of any pitfalls.
Develop educational priorities.
You should be learning a lot about where misunderstandings or interface difficulties occur. This is great insight for developing any support materials that you will create for later releases. You can even begin to draft and test those.
Understand what users like and don’t like.
This is a great time to see if the features that your team likes and has prioritized are aligning with users. This can help with prioritizing bugs, driving any revisions, and it can also feed into marketing efforts and user communications. You probably also want to add to use cases if it turns out that people are doing things that hadn’t been considered. Do you want to support these? Discourage them? A good first step is knowing about and documenting them.
Have a responsive team ready to take on what will hopefully be a nice stream of new product insight. And get ready for beta.
Beta Release
All concerns about product definition and core features should be resolved before entering beta release. This may mean that there are a few alpha iterations before you get there. Beta releases focus on functional feedback, start to inform customer service protocols, and open product usage to wider audiences and real-world conditions. Learning from this stage of release increases reliability, flexibility, and team and product responsiveness when faced with unpredictable external factors. Some product teams prefer to think of product release as “perpetual beta,” because many software and now hardware development processes can support a continuous stream of testing, iteration, and refinement. This mindset is especially useful for automated or assisted products where user behaviors can provide insights or data used to improve or train product behaviors.
Organizing Beta
Set participants, licensing terms, and beta release agreements.
Some betas are open to anyone, while some are invitation only. Commonly, an invitation is requested, which can create a sense of excitement and buzz around a product launch.
Decide how to announce the release.
Many people shun the major PR blitz kind of releases these days, especially in the realm of Lean Startup. The idea is to let the bugs work out and let the user base develop momentum before making a larger push for awareness.
Excite and educate users, explain nuances.
Develop content that conveys the product benefits. Whether printed, on a website, video, or all of the above, you should be able to use the release announcement to get users excited to use your product, and educate them in the best ways to do so. No matter how you announce the release, you might consider ways to get the word out from trusted sources. Are people having a great experience? Find ways to amplify their sentiment through a word-of-mouth or influencer strategy.
Build and engage the product community.
Certain kinds of products can be supported through community forums or other types of social engagement like user-generated content. Others may require higher touch forms of support or service to accompany the product. There may be a range of engagement dynamics between users, like friendly competition, different forms of support and guidance, or collaborative usage. This is a good place to take cues from the audience members themselves, and to approach the relationship with them on their terms.
Refine system for soliciting feedback.
This may include channels through the user communities, surveys or other research methods to understand user attitudes and satisfaction, direct channels within the product itself, as well as more traditional methods of customer support, such as phone or chat help.
Implement real-world quality assurance strategies.
Real users will now have their hands on the devices and in the software. If you are making physical products, there can be layers of quality assurance and calibration standards. This is especially true for medical devices, financial services, or any others that rely on more protected groups of user or forms of user data (in which case, you’ll also need to manage compliance issues and standards).
Learning from Beta
Many of the approaches for an alpha release are the same here, just on a different scale, and more public facing.
Identify critical user errors, or main weaknesses that could jeopardize launch.
This is still the top priority.
Prioritize bugs and assign them to responsible parties.
Continuing this effort is ongoing.
Refine educational priorities.
Are there nuances that people are just not getting? Are there features that are so great they need to be celebrated? User education and communication should help prioritize these.
Understand what users like and don’t like.
You may have shaped your product based on the input of staff, families, and friends. Get ready for a whole lot of people who are not predisposed to go along with your vision or forgive your dumb mistakes. Try to learn from them, and respond without getting discouraged even if there is negativity. Some communication channels are really lightning rods for negative feedback. It’s important to discern inappropriate dissatisfaction from reasonable complaints and unvarnished feedback.
By now, your responsive team will be familiar with the ins and outs of the product and the audience, and ready for that nice stream of new product insight to become a flood. Because now it’s public release time.
Public Release
For many companies, the distance between the later phases of beta release and the early stages of public release is very small. “Perpetual beta,” in which a product is constantly improved, has become a key approach in the success of many of the most innovative companies like Google. While it can be harder to re-create the fluidity of software updates in physical devices, approaching releases with that same spirit can be valuable, especially when the software components are an ever-increasing part of devices. Hardware development processes are evolving to replicate this tighter, faster feedback loop between product teams and their users.
Validation and Feedback
It doesn’t do much good to get products in the hands of users if you are unable to learn from the experience. Feedback can take several forms: quantitative versus qualitative; and observed versus self-reported. There are benefits to each of them.
Start with the most important things to learn. You probably already have a good idea of the kinds of things that you want to validate. Depending on the design and build methodology you have used up until now, you may even have a headstart in organizing, prioritizing, and resolving them.
Important metrics
Success should be considered from both the user and business perspective. You are likely to have clear goals overall, as well as steps along the way. If you created user journeys or models, this is a good chance to validate those.
Focus has been an important theme in this book, and testing it with users should be a top priority. Validating the focus of the user’s intended tasks is a good way to get at what’s working and what isn’t. Are users able to focus on the task at hand when using your product? If there is a learning curve, and are they able to eventually overcome it and reach a state of focus? How long did that take, and what were the main hurdles?
Time might be very easy to quantify, less easy to interpret. Time spent with a new product can indicate pleasure, interest, or confusion. It is therefore often useful to pair time with other factors, whether qualitative or quantitative to arrive at workable insight.
Frequency of use can be a big positive indicator that the product is enjoyable, especially if it is sustained.
Modalities can be tricky to assess through self-reporting, especially when they are subtle or preexist product use. The same goes for things like transitions, shifts, and other aspects of multimodality. So you probably have to work backward a little and see how well the system as a whole is working, and then dig into any problems with more detailed observation. Field research, outside of labs and more automated forms of user testing, is returning to prominence in the development of multimodal products. An informed, anthropological eye for context is invaluable.
Safety is a higher concern with many multimodal products than others, by virtue of their use beyond the desktop, out in the world. The world is a busy place, and distractions can lead to bad results. Anticipate where this might happen, but also stay alert for signs that less obvious dangers might arise.
Feedback methods
Surveys are an easy place to start. When getting at the right questions, try to get input from many people on the team, but use one editor to ensure consistency.
Video diaries can be very useful, if complicated by the fact that many of the products we’re discussing are very immersive, so videos will likely be about users talking about their experiences. Videos of actual usage might be accomplished with the help of friends, but it’s hard to count on that. There are several user research companies that offer apps and hosting systems for creating these kinds of diaries.
Support is a great source of insight into what works and what doesn’t, as well as the steps necessary to get a user up and running. In an alpha release, you may not have a dedicated support staff yet, but if possible, set someone up as point on that, ideally someone that might help make a smooth transition to the official support staff for beta release.
The Out-of-the-Box Experience
The experience of buying, receiving, unboxing, and using a product for the first time can be clear, easy, exciting—even magical—and lead to adoption, word of mouth, perhaps even buzz. Or it can be off-putting, confusing, frustrating, and lead to abandonment or worse, product returns. As consumer sophistication and marketing finesse have both developed, expectations for all points in this experience have risen. Orchestrating this out-of-the-box experience has become a desired skill for interactive designers.
The experience is generally divided into a sequence of days:
Unboxing is when the product is opened. Apple famously set the bar high with its consumer products, starting with the iPod and later the iPhone. Other companies have followed suit, with a similarly themed message: this is a special moment, when a new, somewhat alien but exciting and human-centered piece of technology is entering your life. Marketing awe aside, a clear emotional as well as experiential benefit is now expected to be a part of the packaging.
First-time experiences include that moment of awe and purchase excitement, but should quickly move to getting hands on the product and understanding what it does.
First 9 minutes: Setting up the product by following one of those Read Me First booklets has become the norm, but as much as possible the packaging and product itself should do the talking.
First 90 minutes: Trying the key feature is the agenda, or in many cases, trying whatever happens. Hopefully that’s the key feature, or an exciting Easter egg at least.
First 9 days: Trying all the key features is a likely scenario, though if the first days, or even hours, fail to impress, many of today’s fickle early users may abandon the cause and return or discard the product.
First 90 days is when driving frequency of use becomes very important. Creating moments of use that in turn become habits is key to product adaption. Enabling or prompting ties to the product by making useful connections with other activities and platforms can create value for the user and make the device part of regular routines.
That’s a lot. Managing to tie together good design, ongoing research, and responsive marketing (and to do it all on time) is impressive—and that usually takes a good balance between coordination and accommodation.
Summary
Releasing multimodal products follows the same general schedule of phases as other interactive products, moving from alpha, to beta, and then public release. Understanding that it is a process, and working to organize each part of it to communicate and learn helps to manage and make the most of each step. Learning from each phase is particularly critical given how likely it is that complexities—either from users or real-world contexts—will add unforeseen challenges and opportunities. Creating hypotheses and setting the metrics for gauging success and analysis is an important part of each phase. Leave room for your users to surprise you with new challenges, opportunities, and ideas. Multimodal design is a new lens on delivering delightful product experiences that reflect how people really experience products and the world.
Speaking of which, we’d love to hear your experiences with designing, launching, and using multimodal products. You can reach us at the following email addresses: