Different Maps Cover Different Scopes and Details

The difference between types of maps is often a question of story, scope, and fidelity. What is the experience you’re describing, how much detail is helpful, and which variables are most important to the experience? The following are common ways to map user experiences that are also used in multimodal design, with a few variations. Multimodality calls for a few additional facets to be incorporated, kind of like additional pins or layers on a Google map.

Some design maps and models don’t change very much and should be familiar. The following can be used for multimodal interactions in the same way they would normally be used, with the addition of some multimodal considerations. Multimodal design adds a few others that aren’t as commonly used (see Table 9-1).

Modalities and Senses

What are the modalities of an experience or activity and which are dominant at any given point? Knowing this can tell you a lot about user expectations, habits, as well as what is needed to support the experience. We can start to think about the category of activity, like whether it is orientative, social, athletic, expressive, or analytical. Those are non-exclusive categories, and you may go much further in your definitions of them, but starting high level and then digging deeper is often a good way to proceed.

Focus

What is the most important sense or cognitive activity occurring? How do you manage the amount of information, choice, or activity possible? What are the consequences of minor or major disruptions? If a typist loses track of what they were typing, they might lose an elusive thought; but if a driver loses track of the road and cars around them, they might just lose their life. Whether the interface itself should be the focal point of an experience becomes one of the most important questions a designer should ask. It’s a profound shift from designing for engagement or “stickiness.”

Despite designing some of the most iconic objects of the 20th century, Dieter Rams believed that, “Space does not exist for objects, space exists for people.” We can apply the same humanist principle to preserving mental space.

Level of Focus and Engagement Depth

Every type of product requires varied depths of engagement. At the dead simple end, it’s just on and off, engaged or disengaged. On the more complex end of the spectrum, there are always-on devices, ready to subtly adjust their contribution and accommodate varied user engagement. Their useful presence may shift from ambient to deeply engaging, depending on the moment’s needs.

Even the dead simple end has a twist: when someone is using a product for the first time, and learning how to use it. Take a hammer. Most people get that you hold it from the light end and whack things with the heavy end. It still takes a little time to figure out the right swing. For more complex products, the learning curve can be steeper and require even more consideration. It might be worth modeling this learning curve as a way of understanding and possibly reducing the initial complexity.

Where will the high and low points for your product engagement occur? What kind of attention does each state call for from the user, and in which modalities will they occur? Are there points where the user will be have difficulty keeping focus, and is this caused by internal or external factors? What are the demands on a user’s attention at the deep end? Will they need to stop whatever else they are doing? Will the engagement require focus throughout product use, or can it become a more routine and effortless habit over time? This is often the case with tasks like setting preferences, initiating activities like phone calls, or deciding which music to play. Hence the popularity of default settings, stored phone numbers, and playlists or automatic music stations: they ease some of that cognitive load, and thus the strain of engagement depth.

Continuity

A sense of continuity, that sequential events are tied together, is essential to most experiences. It helps us feel in control and allows us to focus within our thoughts and activities. Causality, an understanding of why and how events unfold, contributes to continuity by assembling information into events and events into narratives. When experience counters expectation, we may feel surprised or disoriented and have difficulty responding appropriately. Repeated disruptions to continuity can cause a person to abandon an experience.

Sequence

As we’ve seen, a baseball pitcher executing a standard pitch follows a sequence of windup, early cocking, late cocking, acceleration, and follow-through. As with a pitch, the sequence of activities in many multimodal experiences is important. One step leads to the other, especially in that realm between mental and physical, where habits and non-aware activities (“muscle memory”) reside. In these kinds of procedural activities, later steps depend on those that came before them to make sense or work effectively.

Shifts

When circumstances call for changes in focus, it’s known as a shift. Providing ways to mentally or physically shift out of a sequence, and back in, can reduce friction enormously. Even though shifts occur naturally, and more frequently in our era of connected distractions, it’s not ideal. Attention switching has costs. “A state of continual partial attention is basically a state of reduced cognitive activity, and that has a large cognitive cost,” notes professor of computer science and author Cal Newport, whose work has focused recently on “deep thinking” and how to overcome distractions.

Flow and Habits

Flow is the state of being immersed and focused on an activity. Habits are how people approach an activity in repeatable ways so that doing them becomes nearly automatic. How much does your device support an activity that is likely to be best experienced as a flow? One main determinant is how expert the user already is or will become. If they are already expert, or have strongly developed habits, it may be important to consider how willing and able a person might be to change existing behaviors. It’s true what they say: old habits die hard.

Interruptions

Sequences may be disrupted beyond the user’s ability to shift smoothly in and out of them. Required sensory input may be impeded, or events requiring attention may pop up. The user might remember something suddenly or simply become tired and unable to maintain a flow state. Whether from external factors or from the internal state of the user, unexpected breaks in a product experience call for smooth recovery, exit, and re-entry. In some cases of surprise or danger, this smoothness is not just a buffer for the quality of the experience. It can help reduce certain kinds of startle response, allowing users to respond to sudden change in a calmer and more rational way.

Substitutions

There are many cases where the most common way to interact in an experience may not be available. Situational, temporary, and permanent disabilities are experienced in one form or another by everyone fairly regularly. A person’s hands are filled with groceries. The majority of people have some form of vision loss. We all need alternative methods of doing things when we hit up against our own limitations or when contextual factors are working against us. Anticipating these kinds of experience blockages, designers can create options to provide flexibility and avoid complete disruption. Many devices handle this by creating redundancy of information: flashing alerts and buzzers; a vibration, screen message, and alert sound. (That last triple threat can sometimes be an interruption more than a substitution.)

In many cases, a well-developed substitution strategy can address both variations in preferences and situational, temporary, and permanent disability. This hybrid approach is especially useful for products with large and diverse user groups. Closed captioning works great whether you can’t hear your favorite show because the upstairs neighbors started tap dancing, or because your ears have been damaged from too much rock ’n roll, or because of permanent hearing loss. The discoverability of alternative modalities can prove challenging, depending on how often it is needed by varying user groups.

Specialized Integration

There are special combinations of multiple modalities that are well understood and are deeply innate to human behaviors. There are also specific categories of activity that call for unique modality combinations. In these combinations, like audiovisual, or haptic tool use, the way that sensory information, decision making, and action are so tightly interwoven means they need to be considered together. Take the example of language—some people mouth words while they are reading or writing, which enhances comprehension. Taste and smell combine to create our experience of flavor. Our sense of touch extends into tools when we use them frequently.

This integration can often occur on a non-aware level—especially around highly developed skills, which can also be difficult for people to describe, separate, or even remember correctly. This can require user research techniques that learn more heavily on observatoin than on self-assessment, like surveys or interviews.

Knowledge and Skill

Is there any special knowledge or skill necessary to perform a task in the experience? Is the device designed to accommodate all level of expertise, or does it bias toward a certain range? Are there different products altogether for the novice and expert? Does it reward improvement or reflect growing expertise by unlocking additional possibilities or removing some guidance? If it was made for users who are already experts, what are their habits, preferences, unique skills, and expectations?

Key Contexts in Multimodal Experiences

Context is a broad concept, and it’s up to designers and their teams to understand which part of any situation is relevant to a product experience. It can be helpful to think about context as layers of an experience. Designers often need to consider physical and environmental contexts, social and institutional ones, and device and information ones.

Experience Map: Transitional Flow

When there is are transitions between modalities, a diagram of those transitions is helpful. This is a time-based, sequential map. Task modalities can appear as swim lanes, with the main types of sensory data feeding them as the user moves through the experience. Your device may or may not be in all the swim lanes; it might be the star at any moment, or it might not, as it might play different roles at different times, allowing people to focus on aspects of the task they perform alone and those which are supported by the device. A tighter variation on this is an internal map, which describes a user’s decision points. This could also be a starting point from which a more comprehensive map is built that also reflects external factors (see Figure 9-2).

Ecosystem Map

When it’s important to get a sense of the information, processing, devices, and other kinds of resources utilized in the experience, an ecosystem map can help make those very clear (see Figure 9-3). The important goal is to illustrate a functional symbiosis, in this case drawing out dependencies and information flow. These are sometimes drawn similarly to architectural programs, mapping the key systems resources and user needs and behaviors to each other. These focus less on designing multimodal experience, but are used to help inform device, platform, or service architectures of multimodal experiences.

Context Map

A context map is a way of visualizing and resolving the dynamic external factors in which your product may operate (see Figure 9-4). At the start of this chapter we listed some key contexts, including physical, social and institutional, and device and information contexts. Depending on what you’re hoping to illustrate, you might include one or more of these. One might play a stronger more prominent role than another in shaping the experience. For instance, resources needed and their availability spans both device and physical contexts. Does your device require a regular recharge? How will that work? Are you making a device that includes voice commands? How will that work in different physical contexts? Will people feel awkward engaging that function with others around them? Will those others around them feel put upon?

Focus Model

Where is a person’s attention within an experience, and what are the details of the senses they are using to achieve that focus? The dimensionality, range, and resolution of the senses, as described in Chapter 2, are important points to consider for the practicalities of a focused sense experience. It is equally important to document the points of focus necessary to achieve the overall goals of an experience, even when they don’t align exactly with your product. In other words, if it’s crucial for a user to stay focused on a separate task, either fully or intermittently, then charting just what they need to focus on will help you best understand your device’s part of the experience and its restraints (see Figure 9-5). Early makers of mobile apps were probably not considering what happens when someone wanted to use their app while driving. But as they grew in popularity, it became clear that they were likely to be used by drivers.

Figure 9-2. Experience map

Figure 9-3. Ecosystem map

Figure 9-4. Context map

Figure 9-5. Focus model

Storyboards and Keyframing

Used during the process of creating films, storyboarding gained popularity as interactions became more complex and were expected to maintain a fluidity that matches real life. It’s even been adapted by software designers using Agile methodology as a way of capturing interaction nuance that user stories don’t allow. It’s basically like a comic, and it can be as detailed or sketchy as required. Ideally it is image-based, but it may start out with some words. The way that stories are told has a strong relationship with the way that we perceive meaningful events; storyboards can help “block” the pieces of your experience appropriately and really help develop the user’s point of view and sometimes even the device’s itself.

What we’re talking about isn’t filmmaking; you won’t have dialogue between superheroes and villains in bubble quotes. For multimodal design, storyboard annotations include technical details as well as the sensory and interaction dependencies that underpin each part of the story. The idea is to bring your vision of an experience to life in a way that communicates both the user’s intention and the thought behind the product’s affordances. Noting the user’s modality and focus—as well as why, when, and how those shift—is important.

Cognitive scientist and comic author Neil Cohn blends cognitive event models with visual storytelling techniques, which we have adapted for design storyboarding:

Release

• Illustrates the effects of the interaction, demonstrating how it fulfills the user’s need, and addresses the contextual and conditional requirements

Cohn’s work suggests that like speaking and writing, visual language has a grammar that is structured similarly to the way our minds understand events. This approach to storyboarding, using the elements that Cohn identifies helps designers ensure that the interaction both fulfills user needs and follows an event model that users can recognize, easily learn, and remember.

Keyframes focus on describing important events to quickly get a larger story across, and to identify phases rather than individual events. They often include several interdependent interactions, rather than focusing on just one (see Figure 9-6). These accented moments give a sense of the important moments of a flow, but without having to connect all the dots. It’s a lot like outlining a story or essay: it lets you make sure you’re getting the point across and pulling the users along for the ride. This kind of document also emerged from filmmaking, to show key moments of character development or plot. They are used to inventory and sequence multiple interactions or events. Once the order and number of keyframe events is right, they can be fleshed out into storyboards, which provide more detail about each interaction and the details that will make them effective.

Figure 9-6. Keyframes

Update as Needed

The artifacts of the design process should always be looked at as provisional. This is especially true at the beginning of the process: the activity of diagramming expected uses, and possibly working with others on the team to workshop, enhance, and align will undoubtedly spur more and better thinking, as will testing your assumptions through prototyping activities. Never be so impressed with your map that you become precious with it: remember it’s just a tool.