Device Ecosystems

Device ecosystems allow individual devices to work with each other. Think of your laptop, tablet, and phone, and perhaps a smartwatch. It is really convenient to be able to pick up where you left off reading on your tablet last night when continuing to read on your phone during your morning commute. Ecosystems encompass devices that communicate to each other directly, or across a network, like streaming music to the speaker in whichever room you like. Certain subsets of devices work especially well together, like a phone with a smartwatch or a tablet with a television or home assistant. Core infrastructure shared by multiple devices come into play as well. Amber alerts to phones work by sending a signal to a specifically targeted area of cell towers; all the phones that are receiving their signal from them get the alert (see Figure 11-2). In multimodal experiences, the hardware, software, and data needed for multimodal experiences can be distributed across the many layers of technology we use everyday.

Figure 11-2. Overlapping information and device ecosystems allow Amber Alerts to target and reach a wide, relevant audience (Source: Bob Bobster, and Zanaq, Creative Commons)

Information Ecosystems

Information ecosystems allow sets of data and data processing services to be used together. For example, Siri started out as the integration of the speech recognition service created by Nuance and the natural language query service provided by Wolfram Alpha. New assistant programs like x.ai integrate AI with natural language processing, your contacts, and your calendar. Smart grid technologies cross-reference between general power usage trends, and your own household usage. The rise of information and cloud-based services like AlexaKit, HealthKit, and Watson are especially designed to allow development teams to very quickly integrate enterprise grade functionality into new products very quickly and easily (see Figure 11-3). For multimodal experiences, information ecosystems help parse sensor information, inform decisions in physical experiences, and interpret the intentions of users’ behaviors and interactions. GPS services cross-reference between several satellites as well as referencing GIS, or Geographic Information Services, to not only tell you where you are, but to fill in information like street names, restaurants, and points of interest nearby.

Figure 11-3. Connecting device ecosystems with services lets consumers voice their shopping lists and then refine them (Alexa); integrating calendar and contacts with natural language processing lets AI assistants manage meetings (x.ai)

Physical Ecosystems

Physical ecosystems are becoming a more prominent part of the Internet of Things. Announced in late 2017, Google’s Sidewalk Labs has partnered with the city of Toronto on a project called, “Quayside,” a 12-acre testbed of new urban technologies. Projects like this take into consideration the physical environment, including roads, streets, and buildings in transportation or smart city uses. These tests envision how technology can change cities, literally from the ground up (see Figure 11-4). The home and the systems within the home are important for smart home and smart building technologies. Public and outdoor spaces in particular can introduce some more complex considerations. For example, certain environments are noisier than others. Public spaces can be crowded and busy, whether dazzlingly exciting or simply overwhelming. It can be hard to know where to look and to process what you are looking at when you are simply trying to find your way through a jostling crowd. Because multimodal experiences are increasingly used to augment reality—in the very broadest definition of the term—the reality that is being augmented is a crucial aspect of the experience.

Figure 11-4. As part of its Amsterdam Smart City initiative, the city council can control streetlights based on pedestrian usage; the city builds on the ecosystem by soliciting and acting on suggestions from city residents (Source: Massimo Catarinella, Creative Commons Share Alike)

Social Ecosystems

Social ecosystems offer other important considerations that too often get overlooked. These include whether a device is shared, is used in shared social spaces, and the behavioral and other norms of the usage context. One of the most visible examples of this was that of the Google Glass phenomenon, where people were uncomfortable with the idea that someone could photograph them without their knowledge. They were also out of touch with fashion trends at the moment. Though the technical functionality of Google Glass was highly anticipated, the social acceptability made the product difficult to adopt—not necessarily by the users themselves—but by the people around the user. It felt threatening to their privacy.

Within the home, there may be multiple members of the household sharing a single device, like a home assistant, television, refrigerator. A parent may not want their child to have access to R-rated movies, or to the “Confirm Order” button inside a shopping app. People have different permissions or roles within a social ecosystem but shared access to functionality—for example, children might add toys to a shopping list, though purchase is reserved for parents. In retail, hospitality, or healthcare, there are specialized relationships between those receiving a service and those providing it. These roles shape the way in which a device can be used. And these attributes of social relationships—role definition, social acceptability, connection, and trust between users in a shared social structure—will become more prominent in user experiences, especially as connected products become woven into the fabric of shared resources, environments, and institutions (see Figure 11-5).

Figure 11-5. Privacy expectations in many settings, such as locker rooms and libraries are easily disrupted by technologies that extend the senses (and might allow whatever they sense to be collected or distributed)

Specialized Ecosystems

Specialized ecosystems have considerations outside of typical usage behaviors and technology requirements. For example, certain types of hospital equipment need to be sterilizable. Medications have many different and long names, and the schedule, dosage, and whether or not a patient has received them or is showing dangerous side effects have to be exactly right. There is zero tolerance for error. Driverless cars need to follow local driving laws and speed limits. Workplaces may require specialized forms of equipment like safety glasses or gloves, or long-term exposure regulations that can make some interactions challenging.

Cloud Architectures: Distributing Resources Through Connectivity

It’s common for multimodal products to have several different devices or connected resources working together to deliver a single experience. For example, the round black tower sitting on a tabletop is the just the tip of the iceberg of an Amazon Echo. The device itself is basically a speaker and microphone cleverly optimized for natural language processing and for collecting sound information and distributing it in the open space of your home. The information gathered is sent via your WiFi router and ISP modem to Alexa, the cloud-based voice service sitting in a data center—probably not so close to where you are. Your voice, and the commands you issue, go through several layers of processing. A greatly simplified flow is something like this: first, the sound is gathered and any external noise is removed. Next, the part that is going to be processed is sent to the cloud. In the cloud, the words are processed, and an appropriate response and action are selected. The verbal and visual response may be routed back to the Echo, but the action may go on through several other steps. A song might be queued up, ready to be streamed following the verbal response. A ticket may appear in front of the person who will begin to start to make your pizza. An item may be added to your shopping cart and be updated across all of your Alexa apps, or to your Amazon account. This item then has to make its way into a box with a silly amount of packing material, and then maybe hitch a ride on a few planes and trucks.

Cloud architectures are used to describe the way technology resources are made available across a network. There are a few basic categories of cloud architectures that are described by how information is used and processed within that system. They can greatly shape the way a multimodal product or service delivers an experience. It turns out that it’s also very handy way to think about product architectures for multimodal design as well. The primary architectures are cloud, edge, fog, sensor network, mesh network, and tag/reader networks.

Cloud architectures are products or services that have a very thin layer of client-side access to resources that are centralized within and distributed across the cloud. Siri offers another example of a cloud-based service. Most of Siri lives in the cloud, with basically the home button, microphone, and speaker on your phone, and some screen interface. It would be pretty impractical to have the natural language processing and training happen on each individual mobile device—that would basically kill your battery and require a great deal of memory. Using a cloud architecture also makes it possible to distribute Siri across many devices but maintain it across all of them, sharing the training from all of its users to improve the service.

Edge architectures are products or services that keep most of the resources in context, like on the device itself, with minimal resources distributed through the network. An example of this might be the Oculus Rift. The visual rendering, motion processing, and interaction happen primarily on the computer connected to the Rift itself. Because visual graphics and motion processing are very processor and information intensive, and need to be delivered in real-time to make the experience more realistic, the experience is optimized around speed. This means that processing visuals or movements on a data center or even across WiFi just isn’t practical—the lag it would introduce would create a critical failure of the experience.

Fog architectures are products or services that distribute resources more evenly throughout a network. An example of this might be a Nest thermometer. The thermometer itself controls the heating elements in your home directly. But account access, remote usage, integration with other Nest devices, and some of the data analysis and processing occur through cloud-based resources.

Sensor networks take advantage of how little power sensors require and distribute them across a wide area. They send their data back to a more central node—which might be a local hub or one on the cloud—where it is processed and analyzed. The sensors provide little to no functionality within context—they just collect data. Sensor networks have broad applications for transportation, smart cities, industry, and agriculture. For example, moisture and light sensors across a farm can detect the health of crops during growing season. One slightly more robust example would be the CalTrans camera system that monitors critical points on the California highway system. In the Sierra Nevadas, it helps CalTrans and drivers monitor road conditions remotely to determine if snow chains or additional plowing is required, and perhaps how early you start driving up to Tahoe for your snowboarding weekend (see Figure 11-6). One of the most notable aspects of sensor networks is the scale: dozens, hundreds, and thousands of sensors may be very cheaply connected to a central hub.

Figure 11-6. A state network of traffic cameras and a website lets Californians observe road conditions for themselves before they head anywhere

Mesh networks are similar to the fog architectures of sensors, but provide both sensor data and relay data transmission together. The laptops in the One Laptop Per Child program take advantage of this technology to allow students to share files with each other and use the internet.¹

Tag/reader networks are products or services that use a system of tags and readers. In this case, the tags themselves may not necessarily be connected in the normal sense of the word. RFID tags are powered by their readers and simply send an identification number to the reader, which can then process the data. Optical readers like barcode scanners use a visual code that can be processed by a reader. More complex systems like Apple Pay and iBeacon use the NFC chip on a phone or Apple Watch and can be tied into cloud-based payment services or to retail environments that respond to physical proximity. One of the notable things about tag/reader networks is that a single reader can read thousands of RFID tags simultaneously. Proximity-based reader networks get multiple readings of the same tag in a physical space and then triangulate position, similar to GPS.

Ecosystem and Architecture: Applying Ecosystem Resources to Multimodal Design

Networks of resources provide the informational and physical qualities of our daily experiences. Houses are a network of resources like space, sunlight, and kitchen appliances, but also utilities like gas, water, and electricity. Cities are a network of public infrastructures like transportation networks, public utilities, access to people who play different roles in our lives, and various zones of social activity, like commercial, residential, and recreational areas. Most aspects of our daily lives don’t exist in isolation, but are part of a greater context.

Sensory experiences also don’t exist in a vacuum. Not only do we have many different senses, we can use many different kinds of sensory information to understand the same thing. For example, we can tell that we spilled coffee by hearing the cup fall over, seeing a puddle, or perhaps, unfortunately, feeling the liquid pour into our laps. What is interesting about how our senses work is that we didn’t really think much about how wet coffee was while it was in the cup or whether it stains. Or how dry the table was before we spilled anything onto it. We sense many things and don’t pay attention until we need to notice. Then we switch gears, often very rapidly. We jump into action, quickly sweeping our phone out of the way of the encroaching puddle. Our own senses are a network that we can use very flexibly in different situations, or from one moment to the next. For example, we can use the crosswalk sign, the slowing down of cars approaching an intersection, or the movement of the pedestrians around us to decide whether or not it is safe to cross the street. It depends on whether we are paying attention to our surroundings or doing something else at the same time.

These variations determine the types of resources needed to provide one or several multimodalities within the same experience. An interesting way to understand these ecosystem configurations is to look at a few comparative examples together. Not only do they provide different benefits and constraints, they also tend to prioritize one aspect of the experience over another, depending on the design goals and approach. Comparative case studies can highlight the different types of multimodal experiences possible for specific use cases.

Sensing Experiences: Answering the Door—A Doorbell, Ring, and the August Lock

There is something elegant about an old-fashioned doorbell: a metal bell hung by the front door, with a rope or chain pull attached to a clapper. It can really only do one thing: ring. A visitor can pull the pull, which rings the bell. Perhaps a friend would have their own special way of ringing the bell, giving a hint at who the visitor might be. A visitor on urgent business could ring the bell very loudly or for a long time. But then again, so could a prankster. A peephole through the door could allow visual confirmation of the identity of the visitor, but is limited if they face away or are not directly in front of it. The resident could also ask, “Who is it?” The full activity of answering the door takes many more sensory channels than just the sound of ringing, and many smaller activities are tied to it: opening, closing, and locking the door, lending someone a key, asking who is there, and greeting them as they enter. The simplicity of the doorbell functionality was very much offset by the social etiquette and activity around it.

These different kinds of multimodal interactions around the front door of the home has made it an early focus of smart home devices. The Ring smart doorbell allows for all these interactions but emphasizes remote access and security. The primary functionality is knowing who is at the front door, when they are there. It has an HD video camera, speakers, and microphone to allow residents to see and speak with their visitors, as well as an infrared motion sensor to detect the movement of heat, allowing it to work with or without daylight. When motion is detected, the Ring sends a notification to a users’ phone. When the button is pressed, a video feed is sent via WiFi to the resident’s phone. This allows a user to check on the movement, whether at home or away using the video camera.

The August Lock, a competitor, focuses instead on the experience of locking and unlocking the door, relying on the door that already exists in your home, as well as a really common locking mechanism: the deadbolt. The primary functionality is whether or not the door is locked. The Lock mechanically turns the existing deadbolt of a door from the inside, when prompted by users via Bluetooth and a mobile app. With the addition of peripherals such as the WiFi bridge and video doorbell, it can also support remote answering and unlocking. It also allows the creation of guest keys for those who have been sent an invite and who download the app.

Understanding and Deciding Experiences: Determining Distance—a Pedometer, Apple Watch, and Lyft App

The original pedometers were simple mechanical devices. A small, weighted pendulum would swing as a person took a step, turning a gear that advanced the counter one step. People would have to cross-reference the information with their watch, perhaps a map, and perhaps a written log to keep track of how far they had actually traveled by foot.

The Apple Watch, on the other hand, uses gyroscopes, accelerometers, and GPS. (The first generation tapped into the GPS capabilities of a paired phone, the following generations have it built in.) It has a lot of other sensors, but these are the set used for detecting distance. In running Activity mode, the Apple Watch allows users to focus their activity on calories burned, elapsed time, distance, or open mode with no goal at all. A useful feature is the haptic notification at the halfway point of a specific goal—this allows the user to decide whether or not they want to turn around and return the way they came or to continue their run along a different path. Once the goal is achieved, another haptic notification lets the user decide if they want to keep going or stop. There is no need to keep track of any information about the run at all; the user can simply run and stay focused on their surroundings or their pace. It provides additional information like calories burned, heartrate, and other details about the runner’s physical state as they are running.

While the first two examples can be used to roughly measure distance traveled, ridesharing apps focus on the distance between two points, namely a driver and a new passenger. Using GPS in both users’ and drivers’ smartphones, apps like Lyft and Uber are focused less on the distance and more on the amount of time it will take for the driver to arrive. Like the feature in Google Maps that tries to calculate the time of arrival of a specific route, these apps use information like current driving conditions, distance, and past driving times and traffic conditions of the route to calculate the time of arrival. This information is used to determine which driver should be assigned to a passenger, and by the passenger to decide if they want to keep or cancel the ride.

Acting Experiences: Writing and Drawing—A Pencil, a Tablet, and the Apple Pencil

The urban legend about the space race between the United States and the Soviet Union comes down to crossing your t’s and dotting your i’s—literally. The Americans supposedly spent millions of dollars developing a zero-gravity pen, while the Russians simply switched to using pencils. The truth is that the AG7 Anti-Gravity space pen was developed independently by the Fisher Pen Company, and offered to NASA after it had already been developed in 1965. The Russians, on the other hand, didn’t just use pencils, they used grease pencils to eliminate the shavings and dust from sharpening. However, after also being offered the AG7, the Russians adopted it as well.²

From the very beginning of our education, we are taught to use writing instruments. It takes years for children to puzzle together the skills of the written language, from the big, blocky letters, we are first taught to recognize their shapes, sounds, and meanings, to cursive and perhaps even calligraphy. Whether in the chubby hands of a kindergartener or an astronaut on the Mir Space Station, we are taught a special grip for our writing instruments and use them for writing, math, and drawing. They are a very immediate tool—mightier than the sword, some might say—their trails of ink or graphite appearing instantly as their tips move across paper. They become a part of our legal identity, as our signatures form the acceptance of contracts and other official documents. Writing is one of our most highly developed and complex multimodal activities. It takes years for people to develop fine motor control, blending hand–eye coordination with linguistic cognition. Most develop the skill so highly that it becomes a flow state, where ideas, words, and their graphical representations on the page become fused together. Writers and artists can develop very personal creative styles. Activities like dictation and geometry blend other modalities and cognitive processes into the mix.

A common grip is a pinch between thumb and forefinger, with the shaft of the pencil resting somewhere along the arch between them. We can write without looking, because most of us have memorized the movements—allowing them to recede into implicit memory, but that might not always result in the most legible handwriting. The feedback of using a pencil is self-evident—the trail of graphite left behind, perhaps the sound of the friction between the paper and pencil. Flip the pencil over, and the eraser helps to remove mistakes. Your hand can easily detect the amount of pressure being applied to the tip and can detect the minute vibrations caused by the variations in surface texture. Writing takes advantage of all of our finest resolution senses together: the minute details of pressure and texture haptics, and the proprioception as we move our hands and fingers, using visual acuity to confirm and guide it all.

For decades, graphics tablets have been used by graphic artists and designers for writing and drawing. These tablets use a variety of technologies and come in a variety of form factors, including something shaped like a hockey puck. Depending on the technology, there are sensors in both the drawing surface and the stylus, or in the tablet only. The experience is basically drawing on a pad with a stylus, with the drawing appearing on a computer screen. It is similar to using a pencil, though the visual feedback is not as immediate or connected to the drawing or writing motion, breaking the strong coordination between visual and proprioceptive modalities that people have developed. The most challenging aspect of using these types of tablets is placement: it’s difficult to precisely place a new stroke once the stylus is lifted from the tablet surface.

The Apple Pencil works with the iPad Pros, using a number of sensors and connecting technologies, including a spring-loaded dual tilt-sensor tip and Bluetooth as well as the capacitive touch screen. These sensors scan the position and angle of the tip 240 times a second. The key advantage to this is that both the tip of the Pencil and the surface of the iPad are pressure sensitive, allowing a variety of drawing details. The direct visual feedback of drawing is immediate, allowing users to draw in ways that are highly comparable to drawing with real pencils, pens, and other drawing tools. The additional palm rejection technology allows users to rest their hand against the surface of the iPad without accidentally drawing with it. People often steady their hand for fine details by resting it directly against the drawing surface. These details create a very realistic drawing experience that matches the existing skills that people have developed using regular writing tools. It actually takes a great deal of sensor technology to emulate the precision that humans can create with drawing tools—because our own senses for using them are so acute. But of course, what we really want to know is, can the Apple pencil draw in space, or does grease pencil still win?

Summary

Because multimodal experiences often mean multidimensional sensory inputs and physical outputs, it’s not surprising that multidevice or other types of ecosystems play a strong role in their design. Smart locks tap into the shared aspect of doors by allowing multiple smartphones to access a single device or by tapping into existing attributes of the home like the front door, or the electrical wires for existing doorbells. The Apple Pencil uses multiple sets of sensors in tandem between the iPad and the pencil itself to approach the sensory acuity that people already have for writing and drawing. Ridesharing services cross-reference existing geo-information with GPS information from multiple drivers and riders to determine the most efficient routes. All of these examples intersect across several different ecosystems to provide multimodal experiences. In some cases, like the Apple Watch, they take over some modalities on behalf of the user in order to allow users to more deeply focus on just one part of a multimodal experience: like looking where they are running. Drawing from different types of ecosystem resources allows users to do more or to focus better within their experiences.