Chapter 3. Sensing
HUMAN BEINGS ARE MARVELOUS perceptual, cognitive, and biomechanical creatures. We filter and process millions of pieces of sensory information every moment. We weave that information into a constantly updating view of ourselves, the people around us, and everything else in the world. What we think of as the five senses—vision, hearing, touch, taste, and smell—are not what we think. They are actually multiple forms of perception working together. Riding our little round spaceship through the universe, we are bombarded with energy and atoms across four dimensions. And yet, human experience is not as elemental. We feel warmth and see color. We taste pizza and smell jasmine. The bridge between the realm of science and what we actually experience is our senses.
The Three Main Categories of Stimuli
Our bodies evolved to perceive things that had some significance, and therefore needed to be accounted for and possibly acted upon. The things that trigger the sensations we experience are varied, but fall into three basic types (see Figure 3-1).
Figure 3-1. There are three basic categories of stimuli that our senses perceive
Electromagnetic
Vision is the ability to detect frequencies of electromagnetic waves and interpret them as light and color. The strength of the wave, or amplitude, is perceived as lightness and darkness. The span of the electromagnetic spectrum that we see is called the visible spectrum. Ultraviolet and infrared fall just outside either end of what humans (and most other creatures) are able to see. Heat, experienced through touch, is also a form of electromagnetic energy.
The photon is the component of light as well as all other electromagnetic radiation. Its properties, such as the quality of behaving sometimes like a wave and sometimes like a particle, continue to amaze and perplex scientists. (A fun fact is that humans and other animals carry a photochemical response that is similar to how solar panels work. The difference: we use it to get a tan, by creating melatonin.)
Chemical
We are able to detect the presence of many chemicals as they pass through the nose or over the tongue. This response is based on direct contact between molecules and receptors. We have plenty of delightful perfumes and delicious foods, but no significant interfaces based on chemical senses. Our skin can detect some chemicals to a degree, though not in a very broad or sophisticated way. This is mainly through hot, cold, or tingly responses to stimuli such as chili oil, menthol, eucalyptus, and so on. This is caused by temperature receptors, normally stimulated by hot and cold, being triggered by compounds in plants that probably developed them as protection.1
Mechanical
Physical touch and hearing are mechanical events. Touch is direct contact with our body, while sound is generally indirect. Waves of air are funneled by our ears to bones that strike the eardrum, causing tiny hairs inside to wiggle. Other bodily senses are mechanical, such as balance, the feeling of pain, and proprioception, a lesser-known but very important sense that tracks the position of body parts, movement, and level of effort.
No matter which type of stimulus triggered a sensation, our sense apparatus converts it to electrochemical signals, and those signals are processed by the brain, where the information, if noticed, is shaped by the mind into coherent experiences.
Defining the Senses: Dimension, Resolution, and Range
Touch is made up of several types of sensory receptors that detect information about warmth and coolness, texture, pressure, and more. It generally requires direct contact, or even movement against something, in order to work. Vision, on the other hand, detects only light, and at a basic level requires only that the eyelids remain open to receive it. Despite sensing just one thing, it can detect many aspects of it in fine detail, such as millions of colors, precise movements, and complex forms. While color and intensity fade away across distance, and outlines get smaller until they disappear, this might happen very slowly. We can see something across thousands, even millions, of miles. When there is little light pollution, a good portion of our entire galaxy is visible to the naked eye. We can see stars, nebulas, and other objects that are light years away. The sense of touch, on the other hand, drops instantly to nothing when we lose direct contact (see Figure 3-2).
Figure 3-2. Our eyes can see flowers a few feet from us, or stars billions of light years away. To untrained eyes, the flowers and stars may appear very similar; it takes time for us to learn how distance and size affects the way things look.
These examples illustrate dimension, resolution, and range. Dimension describes the types of stimuli that a sense detects. The many dimensions of touch include warmth, texture, and pressure, whereas light is the single dimension of vision. Resolution is the level of detail and amount of information within the stimuli, which is low for most of our sense of touch (it varies across our bodies, with fingers being among the standout exceptions) but incredibly high for vision and hearing. We can process a large amount of visual and auditory information with very fine detail. On the other hand, it is difficult to tell the exact temperature through touch. Range is the variation in the stimuli that can be sensed. For example, we are able to see many colors and differentiate subtle grades of light and darkness. Though our precision in detecting temperature is low, we can experience a range from between freezing up to about 140° or 150° Fahrenheit, where we stop feeling heat and start feeling pain.
[TIP]
The word focus is used to describe how we delegate our attention to different aspects of our experience. When it is used to describe sensory abilities, it will be described as sensory focus. When it is used to describe modal or multimodal abilities, it will be described as moda2 focus.
Sensory Focus: Selecting, Filtering, and Prioritizing Information
Each of our sensory abilities can process different kinds and amounts of information. Together, our senses provide us with a superabundance of information. To use it, people need to figure out which bits of information are important or urgent, which are unreliable (like trying to figure out colors in low light conditions), and which are most relevant to our immediate decisions and actions. In design, this is incredibly important, because it determines what people notice and what people miss, what people can remember and what causes confusion. Within each sensory ability, we delegate attention, called sensory focus, figuring out which things to apply and to remember, and we ignore the rest. How we do this is very much dependent on the sense, the context, and our immediate objectives. We have many different ways to focus within vision: we can track a movement with our eyes, focus on specific objects at various visual depths, or examine a particular detail very finely, and we do so with a high level of awareness. Our sense of touch is less aware: we use it automatically for regular physical movement but tune out a great deal of haptic information unless we experience discomfort. We tune out the sensation of clothing on our skin, ambient temperature, and even the chair that we are sitting on, because it is not immediately pertinent. Within hearing, we can focus to identify connections and groupings across sounds like harmonies or rhythms, or to isolate one sound source in particular. Sensory information surrounds us constantly. While our senses receive all of it, we filter out most of it.
Reflexes
We have many patterns of response that require no awareness and which happen immediately, known as reflexes. Many of these are innate, but they can also be learned and refined over time.
When hit on the knee, your leg responds with a kick before you’ve fully “sensed” the blow and certainly considered it. This is an example of an innate reflex, stimuli that send signals on a different path to speedier action than traveling through the higher functions like the brain for cognitive processing. In some cases, it may work its way to cognition through a secondary, slower path. That can sometimes leave us with the feeling, “How did I do that?” as if for a moment, we had been granted superpowers. This quicker path is known as a reflex arc—for instance, going directly to the spinal cord to send a reaction signal to motor neurons. Some reflex actions are also affected by cognition. A flash shined in your eyes will automatically trigger your pupils to constrict, but if shined in one eye and not the other, the level of constriction will depend on which eye, whether seeing light or dark, is topmost in consciousness.3
When creating sensory experiences, it’s a good idea to pay attention and not accidentally trigger an unwanted reflex or automatic response. Fight or flight, also known as the startle response, can be brought on with a sense of danger, like from too many loud noises. Many of our reflexes can be traced back to the way our senses and nervous systems evolved. In an odd twist of evolution, our most automatic functions are sometimes our most primitive. In contrast, automation technology is usually a more advanced capability.
Our Senses and Their Unique Properties
Each sense is a combination of stimuli, plus physical and mental processing, that provides a specific stream of information about our experiences. It’s increasingly common to examine the amount of information (resolution) and range of the senses for interaction design. These factors determine the types of design elements used in an interface mode, as well as the characteristics of these elements.
Vision
As bilateral creatures, we very often have mirrored pairs of anatomy. We have two eyes, two ears, and two nostrils, but hopefully not two left feet. The small distance between our eyes working in sync allows us to perceive spatial distance and to see in three dimensions. Two types of photoreceptive cells in the retina, cones and rods, detect photons and react by sending a signal along the optic nerve and then deeper into the brain for processing. There are usually three types of cones sensitive to lightwaves that roughly correspond to red, green, and blue. While our eyes are fixed inside our head, they can move inside their sockets, allowing us rapid tracking and precision movement. This is important for reading, scanning, spotting and following objects, and having a consistent view of the world despite any other moves our bodies make. We have a blind spot, where the optic nerve enters the retina. Our vision curves at the edges much like a photograph taken with a fisheye lens, because our eyes are spherical. We also lose color perception away from the center of our vision. The tip of our nose is visible to us all of the time. Yet we don’t often perceive these discrepancies in our visual field; we think we see seamless and distortion-free visual field with an even distribution of color (see Figure 3-3).
Figure 3-3. The visual field that our eyes see is different than the resulting image our mind creates by smoothing over the blind spots and uneven distribution of color
Some important characteristics of vision include the visual field, which is what our eyes take in, usually measured by degrees. The field for healthy human eyes is typically 60° toward the nose, 107° outward, 70° up, and 80° down. You can measure your visual field by looking straight forward and stretching your arms out horizontally until your hands disappear. Do the same vertically and you have a personalized measurement. When it comes to film and TV screens, the aspect ratio describes the visual field created for viewing (see Figure 3-4). Some popular formats are 16:9 (HDTV), 4:3 (NTSC video and “four thirds” photography), 3:2 (35mm film), or 2.35:1 (Cinemascope). For obvious reasons, screen design is informed greatly by the visual field, and the overlap between the visual fields of each eye inform the way 3D and stereoscopic technologies work.
Figure 3-4. Different theories about how to simulate the visual field, efficiences in film and video usage, as well as overall device experience have led to the proliferation of aspect ratios
Vision also helps us to know which way is up, and plays a big role in balance. It is the foundation of written and printed communication; the visual lexicon of letters and numbers enable imaginative and precise description of our most complex thoughts. Some nuanced abilities are made possible by visual processing: facial recognition, spatial and volumetric measurement, estimating the speed of a moving object, and calculating our own trajectory through space by using environmental anchors like the horizon or the sun.
As our dominant sense, at least 50% of sensory processing is used for vision, equal to all the other senses combined. Neurons devoted to visual processing take up a full 30% of the cortex.4 Because we process so much, vision takes longer to perceive, even though light is the fastest moving thing in the universe. In return for that cognitive expense, we get an incredibly rich moving picture of the world around us. We can see brightness, colors, shapes, and movement—four parallel image processing systems that work in unison. The range of shades we see from light to dark can also allow us to perceive depth and form. Though the number of colors we can see varies between individuals, as well as over lifetimes, it is in the millions. We can tell what time it is by gauging the quality of sunlight, view the trees to stay on the path where we are walking, and keep watch over a fading campfire by pale moonlight.
There are many reasons that vision takes so much processing. A big one is that we’re doing a lot more than just detecting shapes, light, color, and movement. We are analyzing, matching, sorting, recognizing, remembering, judging, reacting, and more. While it’s true that we might tell a lot by the character of a voice, or quality of a particular smell, we simply see with greater nuance and breadth, both because of the body’s equipment and through cultural training. Being set to process so much information so well, vision is very useful for absorbing abstract data, as we do with reading, data visualization, and iconography. We can also close our eyes, thus stopping most visual stimulus. None of our other senses has such an immediate on/off switch. Table 3-1 contains the human factors of vision.
Visual Interfaces
As our dominant sense, most design tends to be based on vision. The bulk of media is made to be seen, whether with words, images, shapes, or all three. Screen-based interaction design reflects that reality. The field of graphic design is one of interaction design’s most prominent influences. It addresses typography, colors, visual space, and images, bringing with it subdisciplines like iconography and data visualization. Motion-based media like film and animation bring the element of time into visual design. Interaction design extends these techniques in many ways. Much of interaction design could be described as the addition of haptic interfaces to visual media: the keyboard, mouse, and touchscreen greatly expanded the direct actionability of words, images, and graphics beyond just being able to turn a page.
Hearing
Hearing takes advantage of the fact that the air around us carries pressure in waves of vibrations. We perceive the variation of those vibrations in interesting ways. We perceive pitch and harmonics, and we can selectively focus listening, whether on a bird in a forest of activity, or one person speaking in a room filled with voices. (This last one is called the “cocktail party effect.”) Because we have two ears separated by a distance, we also have binaural hearing, allowing us to position the source of sounds in 3D space. While air is by far the most commonly experienced vibration, we can also hear waves conducted through most any form of matter, as when swimming underwater, putting our ear to the ground, or using devices that use our bones to conduct sound waves.
Hearing is the next highest resolution sense after vision. Through language and music, it is tied to some of our most intellectual and creative pursuits. However, when you break it down, a violin concerto, a line of Shakespeare, or the burbling of a stream is really a bit of air getting smooshed in space between the source and our ears. Sound is perhaps most immediately tied to self-expression. Vocalization is one of our earliest communication capabilities. For obvious reasons, hearing is deeply tied to language ability, as well as a big part of what’s known as paralanguage—nonverbal communication like sighs, facial expressions, gestures, the clearing of the throat, as well as the nuanced tones of words being spoken, known as prosody. Table 3-2 contains the human factors of hearing.
Auditory Interfaces
For designers, focus and cognition are key factors of designing for the experience of sound. Because carrying information in the form of speech is cognition-intensive, the pace at which it is delivered and the usefulness to the immediate setting are important. Google Maps, for instance, is careful to deliver turning directions just in time and to not overload the user with information that would be difficult to remember.
Early uses of sound were to carry simple messages over a distance. People figured out pretty fast that sound was a great way to capture other people’s attention even when they were busy. Sirens on emergency responders’ vehicles, loud alarms to scare intruders or mobilize building inhabitants in a fire, and church bells or calls to prayer basically function as a form of public broadcast. This is probably because hearing is our fastest sense. Milder, more personal uses of sound, such as telephone rings and mobile message alerts, are meant to alert but not alarm people. Many of these sounds are still throwback designs to the metal bells in church towers and telephones or steam horns on trains. Even beeping is a throwback to the simplest electronic speakers, with limited ability to reproduce more natural sounds. The way we are alerted by sound does not need to be so retro. Recent sound design has become more inventive, as with subways in Osaka that play a distinctive song at the arrival at each station. Passengers, who often fall asleep in their commute can easily recognize the unique melody for their home station and wake up when they hear it.
Beyond alerts, functional sound design often plays a supporting role, inviting or—even more commonly—confirming actions. On telephones, the dial tone indicated a functioning system and called for action. As digital interactivity replaces analog, confirmations are becoming more widespread, as with the customizable shutter click sounds for your camera. Because sound is used to supplement other senses—usually sight—it is particularly common in helping blind or visually impaired people, as with sound-augmented crossing signals, or the spoken names of train stops to help not only those commuters without sight, but those whose line of sight might be obscured.
Videogames make strong use of sound, both as an interactive element and a narrative one. An interesting subgenre of video game, “without video,” meant to be played with eyes closed, was represented by the Papa Sangre franchise and its spin-offs, like The Nightjar, narrated by Benedict Cumberbatch (see Figure 3-5). It was based on sound, using binaural audio to deliver a sense of place, momentum, and action, while game play consisted of gestures detected by the phone’s accelerometer, and controller buttons on the otherwise blank screen. These kinds of explorations carved out new, useful (and entertaining!) territory not only for gaming but for audio and haptic interactivity.
Figure 3-5. The game The Nightjar puts players on a disabled spacecraft where key life support systems have failed. A loss of vision heightens the psychological horror of gameplay, which is primarily auditory and haptic.
The rise of voice-based interactions, like the Amazon Echo, Apple Siri, and Google Home and Google Assistant promise a more robust place for voice. Limitations still exist—particularly with hearing—because the technology cannot reproduce the human ability of cocktail party effect in noisy environments. Language adds to the already information-dense realm of sound by adding abstraction of thoughts, concepts, questions, descriptions, as well as paralanguage, to convey additional nuanced meaning. While speech is a motor capability—the ability to create vocal cord vibration and coordinate our mouths, tongues, and lips to create specific phonemes, it’s common to think of voice technology as an auditory interface, because we take our speaking skills for granted.
The quick arrival and acceptance of tools like Amazon’s Alexa and Apple’s Siri show the huge progress that’s been made in the last few years in the ability to process human language and convincingly respond. But just like R2D2 demonstrated by conveying meaning with wordless tone and rhythm, paralanguage and prosody can be a powerful tool for sound designers.
Touch (Somatosensory or Tactile Abilities)
The sense of touch, also known as our somatosensory abilities, includes the ability to feel movement, objects, temperature, and pain. We have nerve endings that are sensitive to all those things and that can sense edges, light, moisture, temperature, and even certain chemicals, like peppermint or chili oils. These nerve endings cover our skin and are also in our muscles, joints, organs, circulatory system, and even across the surface of our bones. These all generally require direct contact with an object or close proximity, where intermediaries like air, water, or nearby objects can relay the stimuli. Even if it can seem like we are detecting temperature over a distance, it’s really because the temperature is coming to us. We also discern many haptic stimuli through movement and friction: we need to run our hands over something to feel its texture or have it moved across our skin, like a loofah.
There are several different kinds of nerve endings that comprise touch. Merkel discs detect fine shapes, details, and edges. Ruffini endings detect finger position and movement, Meissner’s corpuscles detect light touch, and Pacinian corpuscles detect vibration and pressure. As science writer John M. Henshaw says in his book, A Tour of the Senses, “Touch sensitivity can be quantified in terms of how much distance between two stimulation points is necessary to recognize them as separate points.”5 In the places where that sensitivity is greatest, our fingertips, scientists have determined that we are able to feel a bump corresponding to the size of a very large molecule, depending on the type of surface where the bump occurs.6
Sensing the world through our skin and touch is constant, even if on a non-aware level. Our sense of “being in our skin” and “on solid ground” is rooted in our feelings of environmental stability. However subconscious, touch plays a big role in how we regulate our physical comfort within a given situation or environment. Deeply focusing on the sense of touch is not common, but once there, researchers have found that it is harder to shift away from a tactile modality than from an auditory or visual modality.7 It is also strongly tied to guiding our physical and motor skills. Certain abilities, like grip, are dependent on it, as well as detecting slippage, when we are losing our grip. Much of that occurs below the level of awareness, especially if all is going well and working reliably. The sense of touch is one of our most adaptive: we can quickly tune out consistent sensations. But when something changes, we quickly snap out of autopilot. Table 3-3 contains the human factors of touch.
Haptic Interfaces (Tactile, Proprioceptive, and Vestibular)
The word haptic describes the combination of the tactile, proprioceptive, and vestibular systems together. (Proprioceptive and vestibular abilities are described later in this chapter.) Most interfaces—whether for computing or mechanical products—are haptic. It shouldn’t be too surprising, as our hands have the most robust blend of sensory and motor capabilities. They are a part of some of most complex physical interactions with objects and environments. In many cases, our proprioceptive capabilities take dominance, with tactile and vestibular playing supporting roles in the experience. Used in tandem with vision, these capabilities are described as hand-eye coordination, which is far and away the most common type of direct physical interaction. Touchscreens, mice, and keyboards are haptic interfaces. So are wrenches, paint brushes, knives, and other mechanical technologies. The inventor of VR, Jaron Lanier, captures the breadth of what haptics encompasses:
Haptics includes touch and feel, and how the body senses its own shape and motion, and the resistance obstacles. It’s surprisingly hard to define the term precisely because there are still mysteries about how the body senses itself and the world. Haptics is at the very least how you feel that a surface is hot, rough, pliant, sharp, or shaking—and how you sense stubbing a toe or lifting a weight. It’s a kiss, a cat on a lap, smooth sheets, and corduroy desert roads. It is the pleasure of the sex that made us all and the pains of the diseases that end us. It is the business end of violence.8
Game designers have very often been the drivers of innovation in haptic interfaces for computing technologies. This is because game design often creates deeply immersive experiences that completely subsume physical reality, pulling user behaviors into a virtual world. That same desire for complete visceral engagement is driving similar work within virtual reality. Some early controllers were gun-shaped to simulate the experience of shooting. The Nintendo Wii and XBox Kinect technologies were a leap forward in haptic technologies. Using sensors, the Wii moved away from realistic controller objects to realistic control movements, allowing users to swing the controllers in the same way they would swing a tennis racket and a baseball bat or throw a bowling ball. The XBox Kinect explored gestural interfaces that use various cameras to track body position and movements. It ran into the issue that people have much finer proprioceptive and motor control in their fingers and faces than in their whole arm, and game players experienced some difficulty in executing and maintaining precision gestures. The sensor technologies of the XBox and Nintendo Wii are now applied across a wide range of automated and assistive consumer technologies.
Early uses of haptics, sometimes called fly-by-wire, included servomechanisms added to the steering wheels on ships and cars, the control yokes of planes (see Figure 3-6) and space exploration vehicles. Before power steering, the wheel of a ship had direct mechanical control of the rudder, which in turn had direct contact with the water. This created a feedback loop, allowing captains to feel the resistance from water currents and to adjust accordingly. This was similar to the yoke and wing flaps on planes, which had contact with air currents. Power steering augmented steering strength, but it took away that environment feedback and reduced steering accuracy. Simulating that direct increases sensitivity.
Figure 3-6. Wind speed and drag are forces that affect the body of a plane and are not normally experienced through touch. Incorporating them into the steering yoke as haptic feedback can make it easier to pilot.
Smell (Olfactory Ability)
The ability to sense the chemical makeup of our immediate environment emerged long ago in the evolutionary chain. While it’s hard to speak of firsts, even bacteria can detect nearby chemicals. Smell travels more deeply into our brain than the other senses, going directly to the olfactory bulb instead of being translated and relayed by the thalamus.9 Science is still debating the exact number, but our noses have around 350 types of smell sensors, and each of those can detect around 30 different odors.10 Odorants must be present and attach to one of those sensors.
While it has been superseded by other senses, the deep tie to our past is part of what makes smell important. Smell is powerful not only for emotions, but is also closely associated with long-term memories, well-being, and sense of place. It takes a lot for sound and vision to nauseate us, and that is usually through symbolism or a mismatch between senses like vision and proprioception. It alone can do it immediately, through odors like rotten eggs, vomit, or excrement. Smell is deeply tied to hunger, sexual arousal, and comfortable or uncomfortable physical intimacy. Table 3-4 contains the human factors of smell.
Olfactory Interfaces
The filmmaker John Waters experimented with adding scratch-and-sniff cards labeled “Odorama” as an added element to his film Polyester. Somehow that really didn’t give olfactory interfaces much traction with the general public. Smell is sometimes used as a warning, and odorless gases like those in our stoves have an added chemical, mercaptan, to alert users of a leak.
While smell is not really used to create interfaces, it is increasingly popular as an aspect of service design and branding. Many companies like Krispy Kreme and Subway bake their goods on the premises during opening hours. The heat dissipates the smell better, tempting people to come inside and taste their goods. Retail and hospitality are increasingly using branded scents to differentiate themselves and to create positive associations for their customers. In one experiment, the sales of Hershey’s products from vending machines tripled when a chocolate scent strip was attached to their exteriors.10
Taste (Gustatory Ability)
Taste was, until recently, thought of as the four elements of salty, sweet, sour, and bitter. Then recently a fifth, the savory taste of umami was added to the mix. A good reminder that individual senses rarely function alone, these five elements still leave us far from the luscious complexity of a ripe papaya, a fried chicken sandwich, or a bubbly and slightly charred pile of gruyere cheese melted over a cup of onion soup. That’s because our receptors for taste, smell, touch (for texture), and probably several others such as the feeling of hunger work together to create the experience called flavor.
The buds on our tongues, called papillae, respond to chemical stimuli. Contrary to popular belief, the specialized buds for each of these do not appear to be clustered on our tongues by type, but spread evenly. As the two senses that detect chemical stimuli, taste and smell work very closely together, with smell detecting airborne particles and concerned with solids. Our sense of taste is uniquely reserved for interaction with things we put in our mouths. These interactions are primarily focused on the pleasure and delight of eating. Table 3-5 contains the human factors of taste.
| DIMENSIONS | Chemical |
| RANGE | Taste requires direct contact with papillae, therefore the longest physical distance for taste is the distance we can stick out our tongues. (Be careful of frozen poles and triple dog dares.)
|
| RESOLUTION | Taste is low resolution, until coupled with smell and other senses to create flavor. |
| FOCUS | We have the ability to isolate tastes and concentrate on them singly, as well as in combination. Taste is primarily associated with the activity of eating. |
| REFLEXES | Revulsion response describes negative emotional and physical reactions like disgust, nausea, or a general sense of threat to our well-being and health. People may spit out something that causes this response, strongly associated with things that may be harmful to ingest. |
| ACCESSIBILITY | No known accessibility issues, though loss of the sense of taste can diminish appetite and pleasure. |
| ADDITIONAL PROPERTIES | Strong tastes can be overwhelming. |
Gustatory Interfaces
Most of the experiences associated with taste are tied to eating or preparing food. These are intended to make the experience more delightful or, in the case of consumables like medicine or vitamins, less unpleasant. The authors are unaware of any kinds of interfaces that are designed around the sense of taste, though there is a tentative new product category of IoT devices called “ingestibles.” These focus primarily on the ability of devices to withstand the stresses of our digestive system, rather than how the devices taste—so far.
Sixth Senses and More
Beyond the five commonly known senses, we have our sense of time; balance; our sense of movement and our position in space, known as proprioception; our proximity to other objects and people; and other forms of perception that shape our ability to interact with the physical world. We also have a range of senses devoted to internal perception, detecting states like hunger, satiation, muscle soreness, and body temperature (see Figure 3-11). Despite being usually taken for granted, these senses are surprisingly important. Take away the sense of balance, for instance, and people are close to incapacitated.
Time and Rhythm
While we often refer to the “sense” of time, it is an awkward fit among the other senses. There is no main stimulus or equivalent sensor. It’s helpful to look at the ability to measure the passage of time as a sense, and one that, like the others, arises out of a several parts working together.
The sense of time is closely tied to the sense of rhythm, which can be looked at as time perceived as a repeating pattern. A sense of time is a very important functional capability. It allows us to anticipate events, carry out complex tasks that involve multiple parts working together, and set or keep a pace in walking and other activities. This can be momentary personal rhythm that sets the pace for a jog, or longer periods such as 24-hour cycles, known as circadian rhythm. They can also be interpersonal events, as when we get into a conversational back-and-forth, or “turn taking.” As a species, we play endlessly with rhythm and sound, which can feel fundamental to how we experience life. As Igor Stravinsky said, “music is the best way we have of digesting time.”
We have mentioned the cocktail party effect, whereby someone can pick out and listen to one voice in a crowd. Research suggests that rhythm plays a strong role in that effect, driving our ability to anticipate words based on the familiar cadence of speech. Time may play another role in helping us hear where a sound is coming from by recognizing which ear received the sound first.
Proprioception and the Vestibular System
Proprioception is one of our most automatic senses, and one of which we have the least awareness. It is an internal sense that lets us perceive things like body and parts position, joint position, and how much effort we expend in a physical task. It is informed by proprioceptors in muscles, tendons, and joints. It also informs our position in space.
The ear is not just responsible for hearing. It also contains the cochlea, which monitors our orientation, guiding balance and movement. This system plays a strong supporting role in vision since, as a part of the vestibulo-ocular reflex, it syncs the movement of the head with compensating eye movement to retain a clear image. It also coordinates with other systems, even surprising ones like breathing and circulation, that adjust to body position (see Figure 3-7).
Summary
The world is made of diverse matter and energy that our senses have evolved to perceive in unique ways and for particular purposes. While the senses carry limitations—some going back to their origins—they have also evolved to work together for newer purposes. What we think of as perception is usually two or more sensations blended to bring about unique types of understanding, called modalities. To create effective interfaces requires understanding the abilities, limits, characteristics, and expectations encompassed by the senses.
Figure 3-7. Proprioception and time
1 David J. Linden, “How We Sense the Heat of Chili Peppers and the Cool of Menthol,” Scientific American, February 4, 2015, https://www.scientificamerican.com/article/how-we-sense-the-heat-of-chili-peppers-and-the-cool-of-menthol-excerpt/.
2 Marnix Naber, Stefan FrĠssle, and Wolfgang EinhĠuser, “Perceptual rivalry: Reflexes reveal the gradual nature of visual awareness.” PLoS ONE (June 2011). https://doi.org/10.1371/journal.pone.0020910.
3 Denise Grady, “The Vision Thing: Mainly in the Brain,” Discover, June 1, 1993, http://discovermagazine.com/1993/jun/thevisionthingma227.
4 John M. Henshaw, A Tour of the Senses: How Your Brain Interprets the World (Johns Hopkins, 2012), 160.
5 KTH The Royal Institute of Technology. “Feeling small: Fingers can detect nano-scale wrinkles even on a seemingly smooth surface.” ScienceDaily. http://www.sciencedaily.com/releases/2013/09/130916110853.htm (accessed February 3, 2018).
6 Alberto Gallace and Charles Spence, “In Touch with the Future: The Sense of Touch” from Cognitive Neuroscience to Virtual Reality, (Oxford, UK: Oxford University Press, 2014), 148.
7 Jaron Lanier, Dawn of the New Everything: Encounters with Reality and Virtual Reality, (New York: Holt, 2018), 123.
8 Tom Stafford, “Why can smells unlock forgotten memories?” BBC Future, March 2012. http://www.bbc.com/future/story/20120312-why-can-smells-unlock-memories.
9 John M. Henshaw, A Tour of the Senses: How Your Brain Interprets the World (Baltimore: Johns Hopkins, 2012).
10 Karen Ravn, “Sniff . . . and spend,” Los Angeles Times, August 20, 2007, http://articles.latimes.com/2007/aug/20/health/he-smell20.