Human Spatial Perception, Cognition, and Action

The defining feature of 3D UIs is that users are viewing and acting in a real and/or virtual 3D space. Thus, psychology and human factors knowledge about spatial perception; spatial cognition; and human navigation, movement, and manipulation in 3D space contain critical background information for 3D UI design. We cover much of this background in more detail in Chapter 3, “Human Factors Fundamentals.” Examples of such knowledge and theories include

Visual perception and 3D depth cues (Bruce and Green 1990; Kosslyn 1993; see Chapter 3, “Human Factors Fundamentals,” section 3.3 and Chapter 5, “3D User Interface Output Hardware,” section 5.2), which are important when designing 3D worlds, objects, and visual feedback in 3D space

Spatial sound perception (Durlach 1991; see Chapter 5, section 5.3), which is used to design audio feedback and data sonification

Presence (Slater et al. 1994), which is an important measure of success in some applications, and may be related to the naturalism of the 3D UI

Human spatial abilities and individual differences in abilities (Shepard and Metzler 1971), which inform the design of spatial interaction techniques

Formation of spatial knowledge about 3D environments (Thorndyke and Hayes-Roth 1982; see Chapter 8, “Travel,” section 8.9), which is critical in the design of wayfinding aids

Cognitive planning of actions (Card et al. 1983; see Chapter 3, section 3.4), which is important to consider in 3D UIs generally and system control interfaces particularly

Properties of manual reaching and grabbing (MacKenzie and Iberall 1994; Marras 1997; see Chapter 7, “Selection and Manipulation”), which inform the design of 3D selection and manipulation techniques

Visualization Design

An important application area for 3D UI is visualization—changing abstract data into a perceptual form so that humans can use their well-developed visual sense to find patterns, detect anomalies, and understand complex situations. Research in visualization therefore informs and complements 3D UI design. Example topics include

Principles of visual data representation (Tufte 1990)

Information or data visualization techniques (Ware 2000)

Scientific visualization techniques (McCormick et al. 1987)

Interaction techniques for visualization (Shneiderman 1996)

Basic Principles of HCI and UI/UX Design

A great deal of knowledge, theory, and practical advice has been generated by researchers in HCI. Although some is particularly focused on traditional desktop UIs, much can be generalized to apply to 3D UIs as well. We cover this background in more detail in Chapter 4, “General Principles of Human–Computer Interaction.” Examples include

Generic heuristics or guidelines for UI/UX design, such as visibility, affordances, and constraints (Nielsen and Molich 1992; Norman 1990), all of which are also critical for 3D UIs

Models and theories of HCI, such as activity theory, GOMS (Goals, Operators, Methods, and Selection Rules), and scenario-based design (Bødker 1991; Card et al. 1980; Rosson and Carroll 2001), which can be used to understand, design, and evaluate 3D interaction

UI design and evaluation techniques such as hierarchical task analysis, ethnographic analysis, heuristic evaluation, cognitive walkthrough, and usability studies (as found in HCI textbooks such as Shneiderman 1998; see Chapter 11, “Evaluation of 3D User Interfaces”), which can be applied directly to or adapted for 3D UIs

Methods for managing the entire UX design lifecycle, such as the Wheel methodology (Hartson and Pyla 2012).

Interactive 3D Graphics

Producing realistic but synthetic 3D images on a computer screen has been a focus of computer science research for more than 50 years. One particular line of research has focused on interactive 3D graphics—images that are rendered in real time so that users can interact with them directly (i.e., user input determines what is drawn, as in video games). This technology provides the environment in which 3D UI designers do their work. We do not cover 3D graphics background in this book, but there are many excellent textbooks on the subject (Hughes et al. 2013). Some representative advances in interactive 3D graphics are

Fast line and polygon rendering algorithms (Bresenham 1965)

Texture-mapping procedures (Watt and Watt 1992)

Real-time lighting methods (Bishop and Weimer 1986)

Dedicated graphics processors for fast hardware-based rendering (Olano and Lastra 1998)

Algorithms for drawing stereoscopic images (Davis and Hodges 1995)

Shader algorithms for fast, customizable rendering on graphics processing units (GPUs) (Nguyen 2007)

High-level graphics software toolkits and integrated development environments (Unity3D and the Unreal Engine)

3D Display Devices

All visual displays used with computers today are capable of displaying 3D graphics (i.e., 2D images that are drawn with proper 3D perspective). Often, however, 3D UIs make use of more advanced displays that provide stereo viewing (slightly different images for the left and right eyes, producing an enhanced depth effect) or spatial immersion (being surrounded by a virtual environment). In addition, many 3D UIs make use of nonvisual displays—displays that present information to other senses. Here is a short list of such advanced 3D displays:

Stereoscopic displays for desktop computers (Schmandt 1983)

Walkaround 3D displays (Bimber et al. 2001)

Head worn displays (HWDs; Melzer and Moffitt 2011), including see-through HWDs for augmented reality (Azuma 1997)

Projection-based spatially immersive displays (Cruz-Neira et al. 1993)

3D spatial sound systems (Kapralos et al. 2003)

Force-feedback, tactile, and other haptic displays (Burdea 1996)

We cover 3D displays in detail in Chapter 5.

3D Input Devices

Computer systems have traditionally used text-based input (keyboards), and one- degree of freedom (DOF) (dials, sliders) or 2-DOF (mouse, joystick, trackball) input devices. Three-dimensional input devices provide more DOF for the user to control simultaneously. These 3D spatial devices are the foundational technology for 3D UIs of the type we discuss in this book; they are the means by which the user interacts directly in a physical 3D spatial context. Examples of 3D input device types include

Position- and orientation-tracking sensors (Meyer and Applewhite 1992; Turk and Fragoso 2015; Welch and Foxlin 2002)

Posture- and gesture-sensing devices, often for human hands (Sturman and Zeltzer 1994)

Multiple-DOF joysticks (Zhai et al. 1999) and 3D mice (Simon and Fröhlich 2003)

We discuss 3D input devices at length in Chapter 6, “3D User Interface Input Hardware.”

To this point in the technological background, we have been discussing component technologies used to build 3D UIs. Now we turn our attention to whole technological systems built of these components. These systems provide platforms for 3D UI design.

Simulator Systems

Before the term VR was ever used, simulator systems pioneered the use of large, immersive, interactive displays of 3D computer graphics. Simulators have been used for many applications, including flight simulation, tank and military vehicle simulation, space vehicle simulation, and simulators for entertainment (Pausch et al. 1993).

Telepresence Systems

Telepresence systems enable a user in one real-world location to feel as if he were in a different real-world location. They combine sensors (cameras, microphones, etc.) on the remote side with displays (visual, auditory, haptic) and interactive controls (e.g., for rotating the camera) on the local side. Telepresence technology is similar to VR (see the following section) in many ways (Stassen and Smets 1995), except that real-world data, rather than synthetic data, is being displayed.

Virtual Reality Systems

Immersive VR systems combine interactive 3D graphics, 3D visual display devices, and 3D input devices (especially position trackers) to create the illusion that the user is inside a virtual world. Some important VR systems have included

Sutherland’s original head-mounted display system (Sutherland 1968)

VPL’s HMD and DataGlove (Zimmerman et al. 1987)

The Cave Automatic Virtual Environment (CAVE), originally developed at the University of Illinois-Chicago’s Electronic Visualization Laboratory (Cruz-Neira et al. 1993)

Augmented Reality Systems

AR uses most of the same technologies as VR but displays a combination of real and virtual imagery to give the impression that the user’s real-world view is augmented, enhanced, or modified. 3D UI design for AR is especially challenging because the user must interact with both real-world and virtual objects, perhaps using different techniques. Examples of AR systems include

HWD-based AR, such as Columbia University’s MARS system (Höllerer et al. 1999)

Spatial AR, where the virtual objects are projected onto the real world (Bimber and Raskar 2005)

Handheld mobile AR, popular on smartphones and tablets (Schmalstieg and Wagner 2007)

Interaction Techniques for Universal Tasks

Selection, manipulation, navigation, and system control are common, low-level user tasks in 3D interfaces. For each of these tasks, there is a large number of possible interaction techniques (combinations of input device and UI software). Interaction techniques may be based on real-world actions, or they may involve “magical” interactions that enhance capability. Chapters 7–9 discuss the design space of these techniques in detail, forming the core of this book.

Interaction Techniques for Composite and Application-Specific Tasks

More complex tasks in 3D UIs are often composed of the universal tasks described above. For example, the task of changing an object’s color might involve choosing a color picker item from a menu (system control), pointing out an object (selection), and positioning a marker in a 3D color space (manipulation). The low-level interaction techniques for these subtasks can be composed to form a high-level interaction technique for the composite task. Other tasks are specific to a particular application. For example, the task of cloning objects in space could be seen as a composite task involving selection, system control, and manipulation, but there are benefits to considering this task independently and designing specific interaction techniques for it (Chen and Bowman 2009).

3D UI Widgets and Tools

Not all 3D interaction operates directly on the objects in the world. For many complex tasks, we need specialized objects that are not part of the environment but that help the user to interact with the environment. For example, a virtual knife might help a designer to slice through an automobile model to see a particular cross-section, or a small icon representing a piece of paper could be attached to a building to indicate the presence of textual information about it. Such tools are discussed in Chapter 9, “System Control,” section 9.4.

3D Interaction Techniques Using 2D Devices

As we discussed in Chapter 1, “Introduction to 3D User Interfaces,” 3D interaction takes place in a physical 3D spatial context (i.e., uses 3D spatial input devices), a virtual 3D spatial context (i.e., a 3D virtual world), or both. In this book, we focus on the design of 3D UIs using 3D spatial input devices. 3D interaction techniques that operate in a 2D input context (i.e., using devices such as a mouse or touchscreen) are another subarea of 3D UIs. For example, in 3D modeling programs running on desktop computers, we need a way to map 2D mouse input to the six degrees of freedom (DOF) of a 3D object. We do not cover this form of 3D interaction in this book. For the reader interested in this area, we recommend checking out research in desktop 3D interaction (Conner et al. 1992; Zeleznik and Forsberg 1999) and touch-based 3D interaction (Wigdor and Wixon 2011).

Hybrid Interaction Techniques

One way to improve on the usability of individual interaction techniques is to combine the best parts of existing techniques. For example, the HOMER manipulation technique (see Chapter 7, section 7.4.5) is a hybrid of two other types of techniques: ray-casting and arm-extension. Another hybrid interaction approach is to combine 2D and 3D UIs together, taking the strengths of each one to create a more robust interface. This type of interaction is often done in system control (see Chapter 9).

Two-Handed Interaction

3D UIs can take advantage of a much richer set of inputs than can 2D interfaces. One powerful approach is to develop interactions that enable the user to use both hands in a complementary way. Taking this approach even farther, 3D UIs can be designed around whole-body interaction. See Chapter 10, “Strategies in Developing and Designing 3D User Interfaces,” section 10.2 for a discussion of this and similar strategies.

Multimodal Interaction

Another design strategy that makes sense for 3D UIs is to use more than one input modality at the same time—so-called multimodal interaction. For example, combining hand-based gestures with speech input provides a powerful and concise way to specify complex actions. Multimodal interfaces are covered in Chapter 9, “System Control,” section 9.9.

General 3D UI Design Strategies

Overall strategies for designing 3D UIs, discussed in Chapter 10 and in the case studies throughout the book, include

Using real-world metaphors that help guide the user to the correct actions

Using physical props or physics-based constraints to lessen precision requirements

Applying principles of aesthetics and visual design

Basing UI design on formal taxonomies of devices or interaction techniques

Basing UI design on guidelines developed by researchers

Using “magic” to allow the user to go beyond the perceptual, cognitive, or physical limitations of the real world

Intentionally violating assumptions about the real world in the virtual world

Development Tools for 3D Applications

A wide variety of software libraries, toolkits, application programming interfaces (APIs), and integrated development environments (IDEs) exist that enable programmers to develop 3D graphical applications. Typically, these applications are written in a standard programming language such as C++ and make use of special APIs for 3D graphics (e.g., OpenGL), 3D device drivers, and so on.

Specialized Development Tools for 3D Interfaces

Fewer tools are designed specifically to aid the implementation of 3D UIs. Some 3D toolkits include default interaction techniques or interface widgets. Also, some work has been done on 3D UI description languages. Standards such as Virtual Reality Modeling Language (VRML) and Extensible 3D (X3D) include some interaction functionality, although the implementation of this functionality is left up to the browser or viewer developers.

3D Modeling Tools

All 3D UIs include 3D geometric objects and/or scenes. We are not aware of any 3D modeling tools aimed specifically at 3D UI visual design, but modeling tools used in other domains, such as animation, architecture, and engineering, can also be used to develop the objects and elements in a 3D UI.

Evaluation of Devices

In 3D UIs, evaluation must start at the lowest level because the UI components are almost always novel and unfamiliar. Thus, comparisons of the usability of various input and output devices are necessary. For example, one might compare the performance of different device types for a simple 3D rotation task (Hinckley, Tullio et al. 1997).

Evaluation of Interaction Techniques

As new 3D interaction techniques are developed for universal or application-specific tasks, simple usability studies can help to guide the design of these techniques. When there are many possible techniques for a particular task, a comparative evaluation can reveal the tradeoffs in usability and performance among the techniques (Bowman and Hodges 1997).

Evaluation of Complete 3D UIs or Applications

At a higher level, UX evaluation can be used within the design process (formative evaluation) or at its end (summative evaluation) to examine the quality of the user experience provided by a fully integrated UI or complete 3D application.

Evaluation Methodologies

Some researchers have investigated generic methodologies for evaluating the usability of 3D interfaces. For example, in test-bed evaluation (Chapter 11, section 11.6), researchers compare interaction techniques by having subjects use them in a wide variety of different tasks and situations so that a complete picture of the technique’s quality can be obtained.

Studies of Phenomena Particular to 3D UIs

Most usability evaluations measure things like time to complete a task, perceived ease of use, satisfaction, or error rates. There are some metrics, however, that are unique to 3D UIs. One of these is presence—the feeling of “being there” that you get when immersed in a virtual 3D world (Slater et al. 1994). Because presence is not a concept that applies to most UIs, researchers have only recently begun to define it precisely and devise methods for measuring it. Another unique phenomenon is cybersickness—feelings of physical discomfort brought on by the use of immersive systems (Kennedy et al. 2000). Again, precise definitions and metrics for cybersickness are just beginning to emerge. Finally, since many 3D UIs attempt to replicate real-world interaction or are inspired by actions common in the physical world, studies of interaction fidelity—the level of realism of the UI—have become important (McMahan et al. 2012).

Design and Prototyping

A 3D UI can be used to allow designers of real-world artifacts to work directly in a realistic 3D context (Weidlich et al. 2007). For example, an architect can navigate through a proposed new building and make changes to its design directly rather than working in the traditional 2D medium of drawings and plans (Bowman, Wineman, et al. 1998). The scenario in the preface illustrates the types of design problems and tasks a 3D UI might help to address.

Heritage and Tourism

Visiting historical sites can often be disappointing. Buildings have crumbled, cities have grown up around the site, and information is difficult to obtain. AR technology can address some of these issues by allowing a visitor to see directly what the site might have looked like in earlier times. The combination of real-world images and synthetic images seen from a first-person point of view can be quite compelling (Wither et al. 2010). 3D UIs can be used, for example, to set the time period the user wants to view or to navigate through text, audio, or image information related to the site (Gleue and Dahne 2001).

Gaming and Entertainment

An added component in video games is the ability to interact spatially in 3D (Norton et al. 2010). This type of interaction provides not only natural interaction metaphors, such as hitting a virtual tennis ball or steering a virtual car with a real steering wheel, but also more magical interfaces such as using 3D gestures to cast spells or to fly through a VE. Other examples include drawing in 3D to create 3D sculptures and using virtual walking techniques to move through first-person VR games. We discuss the design of a gaming 3D UI in one of the running case studies.

Simulation and Training

Three-dimensional environments based on virtual or augmented reality can be used for simulations of military operations, robotic agent actions, or the spread of a disease within the body, just to name a few. Training in a 3D environment for tasks such as surgery, spacewalks, or piloting an aircraft can also be very effective. In most cases, simulation and training applications need interactive capabilities and thus need 3D UI design.

Education

Students can learn topics from Newton’s laws to historical inquiry in 3D virtual worlds and augmented real-world environments. If the worlds are highly interactive, students can experiment with a range of situations to help them construct their own mental models of how something works or to explore and analyze artifacts and information.

Art

Three-dimensional worlds provide artists a new canvas for new types of expression. Although some of today’s 3D art is passive, most of it is interactive, responding to viewers’ positions, gestures, touch, speech, and so on.

Visual Data Analysis

Scientists, engineers, business analysts, and others all work with large, complex 3D (or higher-dimensional) datasets. This data can be visualized using 3D graphics, providing understanding and insight that could not be obtained from looking at numeric results. With 3D UI components, the user can interactively navigate through the data, query various points in the visualization, or even steer the simulation computation (Bryson 1996). The mobile AR case study we present in later chapters of the book fits into this category.

Architecture and Construction

Architectural design and construction projects are organized around large 3D physical environments. With 3D interfaces, architects can visualize and modify their designs directly, contractors can address the coordination of construction equipment on a worksite, or interior designers can try hundreds of combinations of wall colors, furniture, and lighting and see the results immediately.

Medicine and Psychiatry

Three-dimensional applications are being used in the medical domain for telemedicine (remote diagnosis and treatment), 3D visualization of medical images such as MRIs, and psychotherapy, just to name a few examples. Virtual medicine and psychiatry can be less expensive, less embarrassing, and less dangerous. VR can also be used for pain control as part of physical therapy (Hoffman et al. 2008). A 3D UI can be used to allow the patient to interact with the environment. For example, someone with a fear of snakes might be able to pick up and handle a virtual snake with a combination of 3D input devices and a realistic toy snake.

Robotics

As robotic technologies become more prolific in society, designers need to provide methods for guiding and controlling them easily and intuitively (Pfeil et al, 2013). These robots might be humanoid, unmanned aerial vehicles (UAVs), or mobile manipulators. 3D UIs can be used to control these robots in a number of different ways. For example, 3D hand gestures can be used to control a UAV, or a user’s arm and hand can act as a 3D proxy for a robot’s gripper in mobile manipulation tasks. 3D UIs can be used for direct teleoperation, or to guide a robot to move in a general direction, or to point at physical objects that the robot should interact with.

Collaboration

More and more of our work is done in groups or teams, and often these groups are geographically scattered rather than located in a single office. This situation has led to the rise of a whole new software industry focused on collaborative applications, including videoconferencing, online presentations and classes, collaborative document editing, and design reviews (Cao et al. 2006). There are many ways 3D UIs can be used for collaborative work in a number of the application domains above (Prince et al. 2002). For example, a virtual meeting can be held in a 3D environment, providing more of the spatial and visual richness of a face-to-face meeting, or collaborators can enter a 3D environment to work together on the design of a new car.

For Interactive 3D Graphics

The World Wide Web Consortium (W3C) defines international standards for many aspects of the Internet, including interactive 3D graphics. This work has led to the VRML specification and its successor, X3D. These standards provide a well-defined method for describing interactive 3D environments and indicate the features and functionality that 3D Web browsers need to implement. Although they focus on geometry, appearance, and organization, these standards do include some interactive components, and more are being added all the time.

For UI Description

The HCI community has worked to develop methods for the abstract, platform-independent description of UIs, and several have been produced for 2D GUIs (Hartson and Gray 1992). Although these could not yet be called standards, that is their intent. The 3D UI community has also seen the need for such description languages (Figueroa et al. 2001), and we expect that this will be a focus area in the future.

On Graphics

To be usable, 3D UIs often require complex visuals. For example, the principle of feedback indicates that the visual display should show users information about their actions both during and after the users’ input. This means that during 3D object manipulation, we should provide users with sufficient depth and position cues to understand where the object is in relation to the target location. These cues might include subtle lighting effects, realistic shadows, or various levels of transparency, all of which require complex real-time graphics algorithms. In this and many other ways, the requirements of 3D UIs can drive graphics research.

On HCI

The study of 3D UIs has revealed many areas not addressed by traditional HCI. For example, what metrics should be used to study the user experience of a system? In typical UIs, metrics such as speed, accuracy, satisfaction, and perceived ease of use may be sufficient; in 3D UIs, we also need to assess things like physical comfort and presence. The development of heuristics or guidelines for good UI design is another area that has been studied thoroughly in traditional HCI but that requires further thought and expansion for 3D UIs.

On Psychology

As we noted above, the design of 3D UIs is heavily dependent on knowledge from perceptual and cognitive psychology. In an interesting way, even these areas have benefited from 3D UI research. One issue in perceptual psychology, for example, is the design of valid and generalizable experiments studying visual perception, because it’s very hard to tightly control what a person sees in a real-world setting and because some visual stimuli are hard to produce in the real world. In a synthetic 3D VE, however, we can remove all the real-world visual stimuli and replace them with anything we like, producing an extremely powerful environment for studying topics like human navigation (Riecke et al. 2002). Interactive virtual worlds also provide an ideal platform for studying human behavior in a variety of situations (Blascovich et al. 2002)