Chapter 7. The Opportunity: Transforming Existing Products and Developing New Ones
Processors that used to be the size of a football field are now the size of a shirt button. Batteries are portable, small, and now they’re flexible. Technology has reached a tipping point in power, size, and cost to enable the next wave of computing: the Internet of Things. Like the Cambrian explosion, the IoT will bring a diversification of products designed in different shapes and sizes to suit all kinds of environments.
Nature created many different forms of locomotion: wings for flying, legs for walking, flippers and tails for swimming. Similarly, the wings of an airplane, the wheels of cars, and the turbines of boats and submarines emerged during the mechanical age. There can be many different solutions to the same problem: birds have feathers but bees have exoskeleton wings. We have helicopter blades, jet engines, and hot air balloons. As computing expands beyond screen-based devices, there will be waves of morphological divergence and convergence across hardware products—not by evolution, but by design. We will be able to blend these new and existing technologies with our natural environments like never before.
Just look at the IPV6 protocol internet addressing scheme, a plan to accommodate the tremendous number of connected devices expected to come online within the next 20 years. It creates over a trillion internet addresses for every person on the planet. One of the well-known slogans for IoT is “Anything, anywhere, at any time.” “Anything” is a pretty big word all by itself. Even so, it keeps expanding. The category of wearables has branched out to include “hearables,” “ingestibles,” and “embeddables.” New verticals are emerging, and they are evolving rapidly. Existing products are being updated with new computing abilities. The range of possible products and the ways to interact with them can make it hard to know where to begin.
Luckily, existing design practices and deliverables for product development can be adapted to develop multimodal experiences. During early-stage product definition, multimodality introduces an expanded set of user considerations. Product and business capabilities are leveraged against user needs in new ways. This chapter helps identify the unique challenges and opportunities that multimodal design introduces.
Key Applications of IoT: Monitor, Analyze and Decide, Control and Respond
It’s not a coincidence that the key IoT applications are inspired by human abilities. What is valuable about the IoT is that it partially mirrors human adaptability and intelligence: devices can go wherever we can go, and like us, they can directly interact with the environment. These devices have sensors, like we have senses, though they lack our robust multisensory integration. They have new analytical and decision-making capabilities through technologies like big data and AI. This gives them deep but narrow capabilities to understand and decide. Advanced mechatronics and robotics give them the ability to take action within highly structured or focused contexts. The technologies can be used to complement a wide range of human behaviors and interactions. Products that already exist can be optimized for human use.
While it might seem like giving devices capabilities similar to human perception, cognition, and action would put them on a collision course with human evolution, it’s the differences that make the IoT powerful. Ingestible technologies like nanobots blend chemical and thermoreceptive capabilities to conduct automated biopsies. It’s like combining the sense of smell and touch into a single sheet of material the size of a pinhead. Driverless technologies use a navigation system called lidar, an order of magnitude more powerful than radar, sonar, and even our own hearing and vision. They have to: transportation speeds will not be bounded by the limitations of human reflexes for much longer (see Figure 7-1). These are transformative technologies, and in many cases, products will have powerful new behaviors and capabilities of their own.
Figure 7-1. With augmentation or automation, the responsibility for parts of the experience shift from user to device. Rather than complementing a human modality, they run parallel with our experiences, or take over certain aspects altogether.
Functional Categories
In humans, the abilities to sense, understand, decide, and act are intertwined. The same capabilities can be kept together or broken apart across multiple devices. It can be helpful to measure these capabilities against our own as a starting point (see Figure 7-2). Other times, the comparison might be an unnecessary limitation.
Monitor
Dedicated monitoring of environments and events is at the heart of many IoT products, whether it’s taking your home’s temperature, measuring the beats of your heart, or keeping track of the whereabouts of your puppy. Many sensor technologies can detect a wider range of physical stimuli with greater precision than humans. They can also do it without getting bored or sleepy.
Analyze and Decide
From simply binary conditions to massive server farms that use AI to analyze the greatest mysteries of the universe, computational processing is remarkably scalable. Computers have already demonstrated an ever-increasing ability to augment data analysis and decision making. Connectivity puts that power into any device almost anywhere, and sometimes everywhere at the same time.
Control and Respond
The ability to automate product behaviors and mechanical capabilities is allowing devices to take matters into their own, uh, flexors. Has the temperature gone above the range of user preference? The Nest can turn on the air conditioner. Is the dog bored when you’re not at home? Push a button and the Furbo will toss treats. Self-monitoring sensor capabilities, such as in humans, are called proprioceptive systems and allow robotic devices like the Roomba and driverless cars to avoid or mitigate collisions.
Figure 7-2. The building blocks of interface are present whenever device activity replaces that of humans
“Disruptive” Technologies
Early IoT products focused on making existing products “smart”—the phone, the watch, the home, and the car. The term described adding computing functionality that ranged from improved interfaces to complete automation. The term “connected” is now being used to describe the addition of an internet connection, using WiFi, Bluetooth, and other technologies. These transformations give products valuable new features, but sometimes their negative impact is less understood. They can challenge social norms, result in government regulation, and the longterm impact to users, especially children, is under increasing scrutiny.
The user experience of new products can be trial by fire. These technologies can disrupt existing industries and markets, but they can be pretty disruptive to user experience as well. Both design flaws and impossible to predict real-world situations contribute to this, so a public product launch becomes a living phase of user research.
Removing Sound—And Putting It Back
As mobile phone use grew in the mid-1990s, the need for another way to alert users to incoming calls without disturbing the peace in social contexts led to the Motorola SmartTac being given a silent mode that could be switched to haptic vibrations (see Figure 7-3). This has become standard across all smartphones.
Figure 7-3. A Motorola StarTac phone could be muted by the user, otherwise known as being put into silent mode (Source: ProhibitOnions, Creative Commons Share Alike)
The opposite of the noisy phone intrusion occurred with electric cars. The silence of the Toyota Prius in all-electric mode proved dangerous to nearby pedestrians—who, in a twist of fate, were probably distracted by their smartphones (see Figure 7-4). In 2010, Toyota found that its electric car had caused that type of danger, and responded. They created an onboard sound generator that mimics the sound of an electric motor. According to Toyota’s news release at the time, “the sound—aimed to alert but not annoy—rises and falls in pitch relative to the vehicle’s speed, thus helping indicate the vehicle’s proximity and movement.”1
Figure 7-4. The Toyota Prius added sound to accommodate social (and safety) norms
Studies of electric vehicles confirmed the difficulties caused by the removal of a key sensory input, in some surprising ways. In the United States, the National Highway and Traffic Administration found that “pedestrian and bicyclist crashes involving both HEVs (hybrid electric vehicles) and ICE (internal combustion engine) vehicles commonly occurred on roadways, in zones with low speed limits, during daytime and in clear weather, with higher incidence rates for HEVs when compared to ICE vehicles.”2 Interestingly, it was in the moments of least predictability, like slow driving, or backing up, that the lack of sound increased collision. If the cars were going in a predictable straight line, having sound made no difference. In moments of uncertainty, sound has an important function as an alert, one that has been taken advantage of historically. It must be adjusted periodically as contexts and needs change. A car moving around is a legitimate reason for all nearby people to receive an alert. On the other hand, a bus passenger getting a phone call or message isn’t quite as material to personal safety. Violations of social norms or etiquette should be identified early in user research. Safety issues have significant impact, but may be harder to identify and measure, and more complex to resolve. Product teams should be vigilant about unintended impact when introducing or modifying interface modalities—especially in broader social usage contexts.
Mapping Apps Know Who Is in the Driver’s Seat
A thoughtful approach to user needs and active modalities can make or break the success, safety, and adoption of new products. Mobile mapping applications are extremely popular, with usage of Google Maps and Apple Maps in the tens of billions of requests per week. When human safety is at stake, multimodal human–machine interaction must be taken seriously.
Hands-free and eyes-free operation are important modes, as they prioritize the driver’s sensory, cognitive, and physical activities, while the device serves a supporting role. While knowing where you are going is pretty important, the safe operation of a two-ton vehicle at highway speeds can be life or death. To enable adequate response time, the just-in-time verbal directions vary based on speed and driving conditions. The directions can be easily ignored and reroute fairly quickly to accommodate both user choice or error.
Beginning Inquiry
When is a multimodal design approach valuable? The answer is all the time. Most personal computing devices are multimodal already and are very often connected to additional devices that enable even more modes. It’s simply becoming a standard user expectation and experience. Mobility and specialization of devices means that they are also increasingly used across a wider range of contexts and user behaviors. If someone is unable to look at a screen, they increasingly expect a sound or vibration to help or alert them. On the other hand, voice assistants are now “always on”—they have a special wake up command word, because voice is often used when the device itself is not within arm’s reach. Voice, by itself, allows the creation of interactions at farther range from the device—something that haptic modes cannot provide. Blending multiple modes isn’t just mashing a whole bunch of screens and sensors together into a minimal glass and metal case. To be effective, it requires a tailored approach that accounts for shifting user expectations and device capabilities, and anticipates the cognitive state of the user and their physical context.
Developing multimodal products can also have many different starting points. It can mean developing a completely new product from scratch within a product category that does not yet exist. It can also mean adding multimodal capabilities to a mature product ecosystem with an international user base and a network of complex platform and partnerships. From either extreme and in between, product teams should start from the same place: the users.
Workflow to Identify Opportunities
Multimodal design addresses the way people experience physical information and how they begin to apply it back to their own decisions and actions. There are many aspects of a user’s needs, behaviors, and contexts to consider. Empathy exercises begin by asking designers to step into the shoes of their users. Multimodal design asks you to step into their eyes, their ears, and their skin. What kind of physical information is available within an experience? How will people need to use it to accomplish their goals? What kinds of previous experiences will shape their expectations and aptitudes? Will they need to develop any specific skills? Develop hypotheses about these aspects of the experience and explore how individual or integrated modalities enable different user responses. Because multimodal design is about products that fit across the shifting contexts of people’s lives, much thought and imagination should be centered around focus. Take inspiration from other work. Think experientially. Think conversationally. Think cinematically.
During these initial stages of product development, there are several areas of the user experience to explore for multimodal design considerations. The user, their context, and existing product modalities in the product you intend to replace, or a comparative example, are all good places to start.
Assessing user needs
Does the context of use raise the likelihood of interference? Or could the user be experiencing situational or permanent disability? Interference can include the temporary disruption to the user’s sensory or physical abilities. Stress, emergency, and even simple daily inconveniences can introduce situational disabilities, both physical and psychological. Noisy environments, like a crowded work or public place, bar, market, or public transportation, can make ringtones difficult to hear. Carrying groceries can make typing a challenge.
Interference can be overcome by amplifying the primary mode, like raising the volume or making letters bigger and brighter. It may also call for the introduction of a secondary substitute mode, like enabling haptic vibrations when ringtones are inaudible.
Would users benefit from having optional modes in certain contexts The ability to switch from ring to vibrate, or shut off alerts with the “Do Not Disturb” setting, makes sense for the way people use smartphones, especially newer features that can detect when someone is driving. They are now always with us, within arm’s reach, even as we sleep. Other features like color-shifting screens take cues from our body’s natural reactions to daylight, darkness, and sleep. The choice of multiple modes can allow people to use products in ways that are more responsive to their context. Products that support additional modalities within the same focal sense, like both visual scanning and close-up examination, not only strengthen perception but aid in fluid completion of a task.
Would it feel more natural to users to involve another sense or modality Some existing product technologies have now been surpassed. For a long time, lightweight, portable paper was the optimal material for cartography. Then we figured out lamination, and people could trace their planned routes with erasable markers. Now, digital maps can show the entire world and pretty much read algorithmically tailored routes to us. This enables the interdependencies between navigation and driving to be more seamless.
Do users need to create or maintain focus within another modality? Because people constantly use their hands throughout the day, the hands-free modes of home assistants like the Amazon Echo and Google Home are a natural complement to home activities. Shopping lists can be created while looking through the fridge, or just as you spread the last pat of butter. The addition of another mode can also be used as a reassurance when users need to maintain a flow state in a related activity. The red light on a microphone or tape recorder is meant to be seen peripherally as a sign that it’s on and recording. It’s an ongoing equipment check that helps users feel confident that they will not miss a thing.
Would the support of an additional modality strengthen perception, deepen understanding, or aid learning and retention Multimodality can play an important role in activities that require a high level of sensory resolution or where sensory information can be unreliable. People cross-match between modalities to more finely attune their sensory, cognitive, and physical abilities. This is very common in manual tools like drawing or musical instruments and surgical tools that require a high level of hand–eye coordination. Tools like data visualization and physical modeling can powerfully advance our understanding of phenomena like mathematics or chemistry, which exist outside our powers of direct observation. By embodying this information within our sensory modalities, they become more approachable and more likely to be grasped and remembered.
Are there human limitations where technology would be helpful? Thermometers allow us to tell the temperature of things that would burn us. Visual interfaces for sound editing make some auditory properties much more understandable. Infrared cameras are often used at night for security, where human vision fails. The human umvelt is a pretty narrow slice of reality. There is a whole bunch of universe out there.
Are there variations in personal preference for modalities? When checking a melon for ripeness, some people check its color. Some thump it like a drum. Others swear by pressing on the navel, while others go by smell. All of these melon interactions are used to help predict the one sensory characteristic that matters the most: its taste. Some preferences in task modality are straightforward, but there’s always more than one way to make a bed. Modalities are living structures within human experience, and while products can be designed to support existing modalities, we are also modifying them or creating new ones throughout our lives.
Would adding novelty to usage be interesting and appropriate? Sometimes an interesting detail, or even surprise, creates excitement, delight, and appeal. Play, leisure, and rest are crucial to our well being. Play, in particular, is too often overlooked or underestimated. All intelligent animals play; it is important for developing cognitive abilities, not just for children but throughout our lives. The Wii’s gesture-based controllers are an easier interface to learn for casual gamers. When launched, it was a surprising breakthrough that convinced many non-gamers of all ages to try it out. It may be tempting to eschew originality in some areas, particularly where playfulness can harm credibility, but it may be surprisingly compelling and beneficial.
Is there an easier, better way for a user to understand and do things? Sometimes you just have a better idea for how to accomplish a task or convey information. Whether through research, observation, or intuition, ideas happen. This list is not exhaustive, and maybe you don’t even know the answers to any of these questions yet, but you simply have a hunch that it might be useful to explore additional senses or modalities. Sometimes just noodling around and plain old trial and error can reveal new design solutions. Because human modality is such an embodied experience, physical prototypes, and open-ended ideation and exploration can reveal a great deal.
Assessing user context
Developing an understanding of product context is critical. What are potential users accustomed to experiencing in specific moments, what are they accustomed to doing, and how do they typically decide to do it? Is there a current object whose role your product will replace? An activity that it will subsume? These are important questions, and because they are so fundamental, there are many design research methods that are used to answer them, usually based on some combination of interview, observation, and empathy.
Context of use analysis and contextual inquiry are broad terms for this type of assessment that also includes context mapping. These can be expanded upon to include multimodal considerations. Scenario planning and use cases can also be extended in this way. These can blend concept testing with research that aligns with key modalities. As with any effort that aggregates observation, inquiry, imagination, and empathy, the quality of all the inputs is important. Of course, the time, money, and effort required to have top research will vary greatly between types of work, so developing the approaches that best fit the project scope and resources is necessary.
As we see with the Prius example, it can be important to not only account for product users, but for non-users who share the environment. Not enough people had thought about the noise of an engine as an interaction between driver and pedestrian, but for all these years it functioned as one, playing a definitive role in pedestrian modalities and behaviors. And while it may not matter much to the driver of a car that an electric motor doesn’t project the noise of an engine, it sure matters to the pedestrians who need to avoid a moving hunk of metal. Products can have broad and unexpected impact on social relationships and public contexts, and vice versa.
Assessing changes to existing product modes
A quick assessment of multimodal impact can start with asking which of a few types of changes you will be making: adding a new mode to existing product, augmenting existing product modes, replacing a mode in an existing product, or innovating a new type of multimodal product.
Adding a new mode to an existing product is probably the easiest change to grasp. The new device doesn’t just blink—now it also beeps! This can be done for a number of reasons. Additional modes can extend the types of response loops and interactions available to a user. They can extend the range or resolution of an interaction to increase the quality or richness of an experience. Substitution modes provide accommodation around situational or permanent disability or support modality preferences.
Augmenting an existing mode in a product can require some comparison between human modalities. Cognitive load, learning periods, and response times can change significantly. For instance, when street crossing signs went from simply Walk/Don’t Walk to offering a countdown timer, they granted extra information to pedestrians and drivers (see Figure 7-5). However, this little bit of information introduced more complex decision making for everyone as well, which may not have been a net benefit. A simple “yes” or “no” decision—“Should I keep going?”—became one that required pedestrians and drivers to calculate their speed and trajectory, increasing sensory and cognitive load. It’s still yet to be decided conclusively whether this one addition has made for safer crosswalks, as test results vary from city to city.
Figure 7-5. The information in a countdown timer at a crosswalk informs decision making but adds cognitive load
Replacing a mode completely can be a dramatic shift. Because of the time and effort invested in creating existing skills and how effortless they feel as they become more non-aware behaviors, replacement modalities need to offer immediate value or at least clear potential. They may even need to be “insanely great,” depending on how invested people are in their preceding experiences. These shifts may not require a great deal of product development effort. However, it’s user effort in behavioral change and learning that can be the decisive factor.
New modes can be developed for a number of reasons, but they can be powerful drivers of innovation. The GUI introduced a new type of haptic mode to users and unlocked the personal computing revolution. The touchscreen introduced gesture-based haptics and enabled the mobile wave of computing. Human modalities shape our ability to adopt new technologies. Harnessing their power has played a major role in how quickly new technologies become a part of our lives.
Summary
The pace of technological development means that identifying new opportunities and deciding which are worth pursuing can be dizzying. Starting with an assessment of user needs, user contexts, and current product modes presents a reliable way of finding opportunities to develop or update products. Similar to the ways humans function, many new IoT technologies can be categorized by abilities to monitor, analyze and decide, and control and respond. Conceptually matching technologies with human functioning helps us assess their new possibilities in the probable context of human use and make good design assessments.
1 Christopher Jensen, “Toyota Prius to Get Sound to Alert Pedestrians,” New York Times, August 2010, https://wheels.blogs.nytimes.com/2010/08/25/toyota-prius-to-get-sound-to-alert-pedestrians/.
2 U.S. Dept of Transportation, Incidence of Pedestrian and Bicyclist Crashes by Hybrid Electric Passenger Vehicles, https://crashstats.nhtsa.dot.gov/Api/Public/ViewPublication/811204.