Using Physical Information

There are many different ways to tell when someone is attracted to you. Their pupils may dilate, making their eyes appear darker or more deeply colored. They may stand up taller or blink more often. They may stand up taller or blink more often. They may stand closer to you, or touch you (refer back to Figure 1-18, back in Chapter 1). They may touch their hair, smooth down their clothes, or other preening activities. Whether they like you or not, people recognize faces by the proportions between features. We can tell milk is sour by smell—we don’t even have to taste it.

Each of these bits of sensory information is a cue. When we are trying to find out a specific aspect about what is going on around us, we use different kinds of cues. For depth perception, we use several types of cues. Occlusion, or whether one object blocks our ability to see other objects, is one type of distance cue (see Figure 8-2). Some studies suggest that the vestibular system—providing our sense of movement and balance—is also critical to depth perception. Our ability to feel gravity may affect our visual perception of distance.¹

Figure 8-2. Occlusion informs the perception of distance

Graphic designers have intuited this for decades, but it has been confirmed in perceptual psychology studies: we use the breaks between letters and words to pace our eye movements while reading. While the glyphs themselves contain linguistic information, the whitespace between them enables our reading modality; we use it to establish the rhythm of reading (see Figure 8-3). This is not just a matter of personal taste or aesthetic. It’s how we turn on and off visual processing during eye movement to prevent motion sickness in our close-focus visual activities.

Figure 8-3. Whitespace, even when it’s gray, helps establish the rhythm necessary for reading

Similarly, in spoken language, the stressors—an element of prosody characterized by emphasis through volume, enunciation, or rhythm—help listeners break apart strings of sounds into individual words.² Poets and dramatists knew this and used poetic meter to improve the flow of spoken poems or theatrical dialogue. It’s now a part of the markup language for designing speech interactions too. We use outlines to recognize familiar shapes and objects, which iconographers and illustrators understand very well to create simple graphics that are still recognizable. Across all of our senses and modalities there are cues that apply to specific activities and behaviors. This means that certain types of sensory cues are more important than others, depending on modality and the user’s purpose. When creating icons, earcons, and hapticons—and any design element for that matter—there are certain sensory details that people really seek out and need to successfully sense and analyze information.

Across many of our senses and modalities, time is an important piece of information. Within vision, time helps us understand movement. In hearing, the difference between when each ear receives a sound helps position its source. In touch and proprioception, it helps us guide our own movements and our position relative to objects around us. It’s often an overlooked design detail, but one that traverses all of our experiences in critical ways.

Constructing Knowledge, Interactions, and Narratives

Like with Legos, we use different types of sensory stimuli to construct our experience of reality. The scent of freshly cut grass, a warm breeze, and the dazzlingly bright sunshine tell us that it’s going to be a beautiful summer day. We grab our swimsuits and head to the beach. The smell of smoke wafting from the oven, the blare of the smoke detector, and the rumble in our bellies may tell us it’s time to order takeout for dinner—and fast. We hear the alarm clock, feel a little disoriented, but the bed is so warm and the pillow is so fluffy. So we hit the snooze button. Each moment of our lives, we are taking a cross-sample across all of our senses to figure out what is going on and what to do next.

But these sensory stimuli are always relative to what we have experienced already and what we are trying to accomplish. For one person the smell of freshly cut grass might mean that it’s time to take allergy medicine. In another context, a smoking dinner might mean barbeque. How we use the information defines its value, and there are several different ways to use it (see Figure 8-4).

Figure 8-4. The elements of multimodal design are used to inform, guide, and reinforce how users sense, understand, decide, and act

Cues are a type of sensory relationship that help us understand what is happening around us and to recognize objects and environments. We can look at a house plant and see that it is a bit droopy. It probably needs some water. We can look at dog, and by its size, coat, and shape, we can tell it’s a Yorkshire Terrier and not a Great Dane. We can listen to water and tell the difference between rainfall, waterfall, and ocean waves. Through repeated experiences, we learn to use sensory information in different ways. We use cues to recognize our car from similar cars. They help us to construct new knowledge and understanding, and to remember that knowledge when we repeat an experience.

Affordances are a type of sensory relationship that is perhaps most familiar to designers. Introduced by Donald Norman to design, affordances are perceived action possibilities, letting us know the different ways in which we can interact with different objects and environmental features. There is some overlap of course, between affordances and cues. If we want to know if we can lift a dog, we might look at how large it is. But we can also use an animal’s size, as well as other characteristics, to determine their age—if they are still young or full grown (which is then a cue). Affordances help us develop our abilities by guiding the way we interact.

Feedback lets us know that something has happened. Feedforward is real life foreshadowing. It tells us that something is about to happen. Humans are pretty good at confirming what just happened or predicting what will happen next when given the right information. These kinds of sensory stimuli are used to help us construct the narrative aspects of our experiences—especially those around cause and effect. In activities where response times or attention levels are important, the sequence and relationships between connected events are powerful drivers of human behavior. We hear the crash of a wave, and we know we are about to be splashed. We feel a foot slide out from under us, and the rest of our body braces to fall. We hear the latch of a door click, and we know it has closed properly. When people are developing a new physical skill, feedback and feedforward can play a big role in ensuring that people develop that new skill correctly.

Prompts are specific to shifts in agency, interactions that require turn taking in communication, control, or activity. In human conversations there are many different kinds of prompts that let a person know when they are expected to speak and listen. We also have all kinds of nonverbal language to manage how to hand objects to each other, lift sofas together, and pass each other the ball during a game of hoops. Because humans are social creatures, we have a wide vocabulary of prompts around these kinds of exchanges. These are beginning to become more important in the design of assistive or automated products or interfaces like voice where turn taking is required.

Summary

Designers use different types of sensory stimuli to help users to understand, decide, and act. These elements can be categorized as cues, affordances, feedback, feedforward, and prompts. Thoughtful use of them is the basis of good multimodal design.