Vision Dominant Activities

Because vision is our dominant sense—even within multimodalities—we have developed a great deal of technology for it. The most fundamental technologies are of course our written languages and image-based technologies, including visual and film-based media. We have been writing and drawing for millennia—by definition, all of recorded human history. It is somewhat unique, because with stereoscopic vision, we can use it for sensing both 2D and 3D forms. Along with hearing and smell, it is one of our abilities that we rely on for perception across distance. This is why the technology used to create visual experience runs a very wide gamut, across multiple types of engineering and design. In addition, our eyes, heads, and bodies move, allowing us to focus on specific visual depth ranges or objects within our visual field, to track objects moving across it, or to steady what we see when we are moving ourselves. We can change our view, perspective, and proximity to objects and environments.

Many of the modalities within vision provide sufficient information to support understanding and motor skills. This is why they can also become building blocks of multimodal behaviors. Many experiences that we cannot directly experience are mapped to visual technologies, using learnable visual languages and systems or specialized ones requiring a legend or key for the user. Vision can often be a decision-making factor for how we apply sensory abilities as well. The sight of a vibrant red apple increases our desire to taste it. Seeing and hearing water boil in a pot warns us not to touch it. Seeing a big pile of garbage spilled across the street perhaps convinces us to hold our breath while we walk past it.

Immersive Activities: Screen-Based Experiences and VR

Screen-based activities are among the most visually immersive experiences available, requiring that we focus our attention away from the physical world around us and to the events and images on the other side of a piece of glass. When we are looking at them, we focus and fill our visual field and may not spend a lot of time perceiving visual stimuli outside that frame of focus. We are most often reading, observing, or examining the visual information in front us. When we are watching a film, the experience is largely passive. We are simply observing a story unfold before us. Filmmaking has a great many techniques for maintaining visual focus, and for drawing the eye into the frame, where the movements of a camera simulate the way we ourselves would look at an event unfolding. Combined with sound and in some cases, haptic vibration through subwoofers, we are immersed in a story. We are empathizing with the emotions and decisions of the characters in front of us. In this case, a screen need only be the sufficient size, distance, and resolution to engage our visual field and focus.

Experiencing the story for ourselves takes the experience to a deeper level of physical embodiment, engaging our proprioceptive and in some cases, our haptic modalities in the experience. Like many advanced interactive technologies, VR was developed largely for gaming, though other applications are being rapidly explored. Gaming itself extends the techniques in filmmaking in terms of world building and creating characters that the user can easily step into and embody, becoming an actor in what we see. (It would be a bit more challenging to step into the body of a Pegasus with four legs, and learn how to flap our wings, though perhaps really fun!) The goggles use small stereoscopic screens a short distance from our eyes, making it easier to fill our visual field and provide 3D vision. They block out visual stimulus external to that screen experience, forcing us to rely on the screens for our visual reality. Proprioception and vision validate each other for spatial orientation and balance. When they do not align—because of timing or because they are sensing two incompatible experiences, people can suffer from motion sickness or dizziness. Even in just 2D film, unsteady camera movement can also cause motion sickness. This is aggravated when, for example, a screen is showing that we are moving, and our bodies are seated. When playing a video game, the stationary edges of the screen anchor our spatial reality, to offset that physical reaction. In VR, that spatial anchor is deliberately suppressed, and so a great deal of technology is used to anchor spatial reality to the virtual environment.

The speed and tracking of user movement is essential to VR experiences. The Oculus Rift uses a great deal of mathematical computation to build 3D virtual worlds. However, it takes a great deal of computational effort to keep our bodies from rejecting it. To prevent motion sickness and to accurately sync user movement to 3D rendering, the Oculus team turned to motion capture techniques more commonly used in the visual effects realm of filmmaking. The Constellation motion tracking system uses the position of infrared LEDs on the headset to map the movements of our heads quickly to its 3D world building, stabilizing the movement of the images in front of us.

Augmented or Auxiliary Activities: Visual Indicators for Peripheral Information

We already “augment” our physical realities with graphics and text that don’t require head-worn gear. LEDs, printed labels and icons, and small are very common in physical devices. They are often used as an indicator for power level or for other factors, like connectivity in a router, or letting you know which elevator in a bank has arrived. Lately, the usage of LEDs has become very sophisticated, especially with the type of housing and material used around it. A common technique is called counterboring, where small depressions are molded or drilled into the back of the housing material. This allows the light to be seen and focused when it is on, but the surface of the housing remains visually uninterrupted when it is off. This is related to punch-cut screens, which only allowed light to pass through certain areas, using predetermined shapes to spell out numbers, letters, or special characters, like old digital clocks and cross walk signs. Some examples of this include indicators on Apple laptops, and the Areaware wooden clock. Jawbone created custom shaped icons for their counterbores, and used color, indicating states for specific functionality.

In addition, color and animation play a strong role in creating visual systems or codes. “On” or “off” is a binary, usually mapped to “yes” or “no” of some state. Colors can mean a few different things, if people learn them. This idea has been around since stop lights, with red, yellow, and green. Increasingly, some of the technologies that are applicable to LED usage can be applied to other light sources, using “smart” technologies. One example of this is the Phillips Hue, which can be used to emulate sunlight as a kind of wake-up alarm clock. Home lighting has always been an indicator of sorts. When a fuse blows, or there is a blackout, it’s always reassuring to see the lights come back on, which lets you know that the power is running again. It also lets others know that someone is home.

Augmented Reality Versus Augmented Products: Visual Arrays of Control and Choice

For a long time, control panels and dashboards have been a very important part of our interactions. We use dashboards in cars, planes, and other forms of transport, but they are not just for heavy equipment. Artists use their palette to review the colors they have available for drawing an image—both with physical paint and in software such as Photoshop. We have signage to help us determine where in the grocery store we would find a can of peas, or where to find a book in a library. In these cases, we are given an array of information that we can use to inform our focal action, whether it’s driving, drawing, or shopping. These kinds of visual arrays are most commonly used for activities that require a high level of focus or decision making. But the most important experience attribute of dashboards is really not how much information we can cram into them. It’s the fact that we can look away or ignore them, and then easily remember where to focus our eyes for a specific bit of information or control. It is on a need-to-know basis, over which we develop effortless control, after much repetition. On a car dashboard, all the radio controls are in one place. All the heating and cooling systems are in another. After a time, we can use them without looking, because the spatial position and haptic information is sufficient.

With augmented reality technology, particularly for vision, that ability to look away is not as effortless, and there isn’t always a learning period. It can be difficult to apply the technology to complex activities correctly.

It may be better to use sensory substitution or to add extra steps to the focal experience, to prevent distraction. Splitting cognitive focus has limits, and may not be quickly repeatable or easily sustainable over long periods of time. This is why hands-free technologies are receiving increasing scrutiny. When we divide our attention across two activities, it’s not split 50/50. Our performance in both activities degrades significantly. The more activities we add or the more focus and cognition one activity requires, the more rapidly our performance on any of them deteriorates. People seem to know this to some extent. You wouldn’t try to explain a complex mathematical equation to someone who was driving, and expect them to understand or remember later—especially in difficult driving conditions.

Over time, however, the gap between 2D and 3D screen-based media will be bridged, allowing a more contiguous experience between traditional screen-based UIs, AR, VR, and the physical world.

Automated Visual Capabilities

One of the more interesting aspects of recent technology is re-creating vision through machine learning. People can do a great deal of visual analysis, drawing many important ideas and information with sight. We can recognize who a person is, how they are feeling, and if they are paying attention to us or if they are distracted. We can estimate how fast a car is moving toward us, how many people are in that car, and most importantly, whether we need to get out of the way. We can do all of them at the same time to a certain extent.

This ability to analyze what we see, and to parse it into meaningful information is a bit more challenging for technology. There isn’t one specific technology that can encompass all of human visual ability. Instead, it’s broken across several technologies and applications, and different companies have different capabilities within them. Driverless cars have special emphasis on avoiding obstacles: determining the speed and position of other cars and objects that will enter the projected path of the car. Facial recognition however, focuses on the geometry of our face, and cross referencing it against existing catalogs of faces to make a positive ID. Image tagging focuses on recognizing objects, colors, and other visual attributes within a specific image. In the Google Vision API, there are algorithms that will even try to recognize the breed of dog, if it recognizes one in an image.

The Google driverless car uses a blend of sonar, stereo cameras, lasers, and radar for obstacle detection. “The lidar system bolted to the top of Google’s self-driving car is crucially important for several reasons. First, it’s highly accurate up to a range of 100 meters. There are a few detection technologies on the car that work at greater distances, but not with the kind of accuracy you get from a laser. It simply bounces a beam off surfaces and measures the reflection to determine distance. The device used by Google—a Velodyne 64-beam laser—can also rotate 360-degrees and take up to 1.3 million readings per second, making it the most versatile sensor on the car. Mounting it on top of the car ensures its view isn’t obstructed.”²

Facial recognition requires cameras facing the person, usually at eye level, and in some cases infrared, to determine the contours of the face, and where to focus, as in autofocus cameras. Image tagging requires scanning of image files that are already data files, and so does not require particularly specialized hardware at all.

Creating Focal Experiences with Audio and Speech

Creating auditory and speech experiences entails a stronger emphasis on social experiences. Talking is something we do with other people, and so the social aspects of voice interactions will always play a strong role in their design. While several people in a room can choose to look at a screen or not, sound vibrations carry through the air in all directions—at sufficient volume, everyone can hear them. They also travel through liquies and solids, which makes possible warning sirens, sonar, and sadly, annoying neighbors. Hearing through solids, however, generally requires direct physical contact. Solids can conduct lower tones better than air, so our voices sound lower to us than they are, because we are hearing them through our jawbones and skulls. Our sense of hearing depends largely on the frequency or pitch and the volume of the sound. This is relative to the distance between the listener and sound source, because soundwaves have physical properties like decay (they lose energy as they travel), echo (they bounce off solid objects), and the Doppler Effect (they can get compressed or stretched out depending on movement), which all impact the sound quality. Those qualities also help us to roughly position the source of a sound in space. Like all of our other senses, we can detect patterns in sounds, especially in time and frequency, resulting in our experience of rhythm, harmony, and melody.

Personal Sound Experiences

Typically, the larger the speaker, the lower the pitch and louder the sound it can create. Lower pitched sound waves are physically larger and therefore require more physical volume. That is why headphones can sound tinny (it’s difficult for them to create low frequencies at sufficient volume), and why subwoofers are usually so big. This can be a physical challenge for sound-based devices. Headphones and earbuds are designed to keep sound personal. They may cover our ears, trapping sound against the side of our heads. Earbuds are funnel shaped to fit securely into our ears and take advantage of the little bump in our ear, called the tragus, to hold them snugly in place, trapping both the buds themselves and their sound inside our ear canals. There may be externally facing speakers to detect environmental sound, used for sound cancellation, where the peak and trough of the detected sound wave are inverted in a second sound wave, thus neutralizing the first sound wave. Increasingly, the directionality of sound is being explored.

The downside to all of this innovation, however, can appear when we are actually having a phone call or other conversation. We can look like we are talking to ourselves, or seem like we are addressing someone when we aren’t, creating awkward situations. For now, the social stigma of talking to yourself is still pretty strong. As the technology for auditory experiences advances, many related services are being integrated into the earbud (see Figure 10-5) or headphone form factor.

Figure 10-5. Google Pixel Buds integrate services like translation and assistance

Parametric speakers, on the other hand, take advantage of the ultrasound—mechanical waves that rise above our threshold of hearing. These waves can force audible sound waves to travel along a specific path and distance—in effect targeting where an audible sound can be heard. This is still a somewhat experimental technology.

Social Experiences: Broadcast

Ancient Greek and Roman auditoriums were designed for both hearing and speaking. Their name literally means “a place for hearing.” They were shaped in the same way sound travels from our mouths: facing from one direction with a small starting point that radiated outward and grew larger. Seats were arranged to fall within that radiating shape, so that those sounds could be heard. They were smaller than those we have now because the human voice alone can only travel so far.

Modern technology allows us to amplify sound to be heard over longer distances, but some of the fundamentals still apply for social auditory experiences. In some cases, many small speakers can be used to replace one large speaker. They sound the same within a specific spatial range, but can be more directionalized, preventing sounds from carrying too far. This can be helpful for public spaces like stadiums, or museums with multiple auditory displays in close proximity to each other. The Amazon Echo is cylinder-shaped to allow for a 360° microphone array and speaker configuration to be able to “hear” and “speak” to anywhere in a room from any place in the room.

Conversation Experiences: Speech

On the flip side of social auditory experiences is conversation. The often-quoted number of how 75% of communication is nonverbal is correct in spirit, though perhaps not mathematically accurate. Conversation is a multimodal experience, and we use many visual, auditory, gestural, and spatial cues when we are speaking to one another. We are very good at predicting when it is our turn to speak or listen or what a person is about to say, and use many different behaviors to communicate emotion, attention, and other aspects of dialog and empathy. We are so good at conversation, in fact, that our turntaking response time in conversation is unbelievably fast. The pause between turns can be as little as 200 milliseconds, literally faster than the blink of an eye. This can make creating speech interfaces a bit challenging, as longer than that breaks expectations of dialog. We can also process large quantities of verbal communication. We can listen to a person speak for hours at a clip. Alexa, on the other hand, can handle up to ninety seconds of sound file, and its response time varies on a number of factors, including the speed of your household internet. Despite these technical limitations, its interaction design took cues from how people really converse. A ring of LEDs on the device responds as quickly as a human does in conversation, but it tries to point at the direction of the person speaking, like it’s “looking” at you. This makes the delay feel like thoughtful listening and consideration of response, rather than latency.

The multimodal cues around conversational speech are translated into experiences that may not have the same physical experience of chatting with someone, but echo the timing, conversational cues, and other social qualities of conversation. Other perceptual attributes of conversation may help increase the acceptability of the experience for users.

Creating Haptic Experiences

Our sense of touch is the most multidimensional, detecting mechanical, electromagnetic, and chemical stimulus. It is our most direct sensory ability, requiring physical contact with objects and environments. It’s also tied to some of our most non-aware and our most engaged activities. We feel the smooth, cool cotton of a shirt when we put it on in the morning, but then stop noticing it touching our body almost immediately. We can feel the shape and cushioning of a chair, but quickly forget when we are watching a really great movie. When we do notice, we get tremendous pleasure from touch-based experiences: the fluffiness of down pillows on our faces, the satisfying crunch when biting into a crisp crust of bread, the funny jiggle when poking Jell-O, a warm bath after a long day. We laugh when we are tickled, and we recoil with horror if something fast with a lot of legs touches our feet unexpectedly. Touch can be our most experiential sense—the one that most quickly and effectively immerses our bodies into being in the physical world.

While the cosmetic design of a device may have tactile material characteristics when being handled, it’s quite another thing to simulate them using hardware componentry. All of our other senses have physical focal points: the eyes, ears, nose, and tongue. We have touch receptors all over our bodies, and deep in our muscles, bones, and organs—even in the previously mentioned eyes, ears, nose, and tongue. There are sensors that can detect moisture, but there is no technology that can re-create the feeling of wet without the immediate presence of liquid. Just think about what it feels like to be in a jacuzzi: warm jets of bubbles and water all over our skin, steam and splash hitting our faces. Because of the range and degree of sensitivity in our haptic experiences and how deeply tied they are to our bodies, they are difficult to re-create with digital technology. For now, a full haptic body suit seems a bit farther off into the future.

There is, however, a wide array of sensors that can detect the same kinds of information we can through touch. Sensors can detect many of the physical stimulus we can, like wind speed, temperature, and the presence of moisture. This helps us avoid direct exposure to unpleasant weather conditions, like tsunamis, tornadoes, and raindrops falling on our heads. On sunny days with a cool breeze, birds singing, and flowers blooming, the weather report can be a tease if you are cooped up inside. It is about as experiential as reading the list of ingredients on a chocolate wrapper versus actually tasting one.

Common haptic technologies fall within four dimensions: our ability to detect force, contact, movement, with surface texture sometimes playing a supporting role. They focus more on sensing what we are doing, and are limited in creating a meaningful range of haptic stimulus for us to sense. Helmets can detect force of impact during football, and determine whether the force was significant enough to cause a head injury. Fitness trackers can differentiate between fitness activities like biking, running, walking, and swimming. Between sensors and cameras, game consoles can create 3D avatars of our bodies and approximate many different kinds of interactions, from your golf swing to shaking up a bottle of champagne. Touchscreens can tell a great deal about the way we touch our screens, and used with peripheral devices like styluses, they can simulate many different types of hand tools and instruments, from pencils and paintbrushes, to pianos and steering wheels. Because hand-eye coordination and tool use are so fundamental to human interaction, there is still probably a great deal more we can still do with haptic interfaces.