Chapter 12. The Future of 3D User Interfaces

In this final chapter, we present a list of open questions that need to be addressed by researchers, students, or developers. Although we could have listed many more important questions, we’ve limited this list to those that we consider to be “grand challenge” questions for 3D UIs, in hopes that this will serve as a research agenda for the field.

Although the first edition of this book was published more than ten years ago, several of the questions we posed then are still open and important today. This speaks to the difficulty of some of the major challenges to be solved in this domain. At the same time, the field has changed significantly in the last decade, especially with respect to high-quality consumer-level technologies and real-world applications. With all this change, new challenges have emerged, and so we’ve significantly updated the list of open questions as well.

Of course, any such list of grand challenges will to some degree be subjective, speculative, and biased by the authors’ presuppositions. Despite this, we hope that this list will help you think about the next steps for 3D UIs and inspire you to tackle some of these important issues yourself.

Another issue with projected displays (not HWDs) is the occlusion problem. This occurs when a physical object (like the user’s hand) passes behind a virtual object. Because the virtual object is actually being displayed on a screen behind the physical object, the virtual object is occluded, even though according to the “real” positions of the objects, the physical object should be occluded. The occlusion problem also occurs in AR displays when the system doesn’t have sufficient depth information about real-world objects.

Both of these problems could be solved by “true 3D” displays—devices that display the graphics for a virtual object at the actual depth of that object, so that light appears to come from the proper points in 3D space and all the depth cues agree. Prototypes of such displays (volumetric, holographic, multifocal, and light-field displays) have been developed (see Chapter 5, section 5.2.2), but they are not yet ready for real-world use, and technical challenges remain.

From a 3D UI point of view, we know very little about user experience with such displays or how to design 3D UIs appropriately for such displays. The assumption is that more perfect depth cues will help users improve their spatial understanding, increase the sense of presence, reduce discomfort, and improve perception of distance and size. But these displays may produce other unforeseen effects, because the visual experience still won’t be indistinguishable from the real world in every way (e.g., FOV, spatial resolution, color reproduction, dynamic range, etc.). How will these limitations affect the 3D UI user experience?

There are systems that have attempted to integrate more than one haptic sensation, like the one shown in Figure 5.30, which provides ground-referenced force feedback that keeps the whole hand from going through a virtual object as it moves in space, and grasp feedback that displays forces to the fingers as they grip a virtual object. But these require significant infrastructure and are not at all easy to put on, use, or take off. So a second challenge is the design of usable haptic systems that provide a pleasant user experience. A great deal of interdisciplinary research is needed here.

The real challenge, however, is to integrate all of these single-sense display types into a seamless multi-sensory display system. Similar to the Holodeck on Star Trek, in such a system, objects and environments would look, sound, feel, smell, and taste completely realistic. One specific challenge is integrating haptics with immersive visuals. The most realistic haptic systems are ground-referenced, while the best immersive visual systems allow free movement by the user. Most haptic devices, because they must sit on a desk or because they require so many wires, mechanical arms, and other hardware, are too complex to integrate with a surround-screen display or HWD. Assuming that we can solve these technical integration issues, the actual interplay of various multisensory cues at a perceptual level is still a subject of research. Assuming that we won’t have a perfect multisensory display anytime soon, how will users react to moderate levels of multisensory fidelity?

A common way to achieve this is to use a tangible interface, where virtual objects have an associated physical object. To manipulate the virtual object, therefore, the user simply picks up the associated real object. As we’ve seen, however, magic interaction techniques can be very powerful, so AR researchers should also consider whether it is possible to interact with real objects using magic interaction techniques. As a simple example, note that it certainly makes sense to select both virtual and real objects using a magic technique like ray-casting, as long as the system has knowledge about the objects in the real world.

A fully seamless real-virtual UI may not always be possible. So we also need to consider how users switch between acting in the real world and the virtual world. How much does it disturb the user’s flow of action to switch between an AR display and viewing the real world, or between virtual- and real-world interaction techniques?

Take a simple example: suppose you choose the pointing technique for navigation and the ray-casting technique for selection and manipulation and that you are using an input device with only a single button. When the user points in a particular direction and presses the button, what does she mean? She could intend to travel in that direction, or she could intend to select an object in that direction. The integrated 3D UI must disambiguate this action in some way. For example, we could decide that if the user clicks the button quickly, we’ll treat it as a selection action, but if the user holds the button down, we’ll treat it as a travel action.

One solution to this integration problem is to use integrated or cross-task techniques, which use the same metaphor or action for multiple tasks, but more work is still needed to determine the best ways to integrate techniques in general. A deeper understanding of what techniques work well together as a set is also needed.

This has given rise to many locomotion devices that keep users in one place as they walk, but so far none of these devices provide a fully natural walking experience and do not allow for the whole range of human walking movements. The other approach is to use natural walking as the input, but to modify the output in subtle ways to give the user the illusion of walking through an unrestricted space, as in redirected walking techniques. So far, though, redirected walking still requires a large physical tracking space, and there’s no general-purpose method that will guarantee that users (a) will never reach the boundary of the physical space and (b) will never notice the redirection.

Both of these approaches still have promise, but there is still a long way to go. A general-purpose, fully natural locomotion device or redirected walking technique is in some sense the “holy grail” of VR travel research.

Recently there’s been a resurgence of interest in gesture-based interaction because of technological developments that allow much more precise tracking of users’ bare hands through optical sensing. Perhaps, it is thought, the reason 3D gestures never took off is that users had to wear cumbersome gloves to use them. Today’s markerless tracking systems are significantly different from yesterday’s gesture-sensing technologies. There are still technological challenges, however, including the accurate prediction of hand poses in the presence of occlusion, and the precise detection of touch events, such as a pinch between thumb and forefinger.

At the same time, the barriers to gesture-based 3D interaction are not simply technological. Despite its seeming naturalness, it is nontrivial to design gesture-based interactions for many tasks. We don’t use gestures for all real-world interactions, so it’s not a question of simply replicating the real world. There are tasks in 3D UIs that don’t have real-world analogues, including many system control tasks. And there are significant ergonomic barriers—humans cannot hold up their arms/hands in space for long without experiencing fatigue. Although there have been some interesting research prototypes, it remains to be seen whether gestural, bare-hand 3D UIs will become commonplace or whether they will remain a novelty.

For example, if the user says “put that there” accompanied by some directional gestures, a traditional interface would require precise specification of the object (“that”) and the target location (“there”) via accurate pointing gestures at the right time. An intelligent 3D UI, on the other hand, could make predictions about the identity of the object and the target location based on many factors, such as which objects are most likely to be moved, which locations are reasonable places to move the selected object, which objects the user has glanced at over the last few seconds, and how this particular user tends to point. Existing techniques for scene understanding and deep learning can be leveraged to enable such insights.

ML techniques can be used in many other ways for predictions and suggestions that could aid the user and make 3D interactions feel magical, as if the system knows what the user is thinking before the user himself does. And ML perhaps has even more potential to enhance 3D UIs than it does other types of interfaces, because of the complexity of 3D interaction.

But this is a very different style of interaction than today’s 3D UIs. Such smart 3D UIs would be based on a combination of explicit input (the users directly indicating what they want to do) and implicit input (the system watching the user to build a model of the user’s intent based on learning and probability). Inherently, this means that the user is not fully in control, and the system’s actions may not be fully comprehensible. Thus, although we are excited about smart 3D interaction based on ML, it is also a grand challenge for 3D UI designers to ensure that smart 3D UIs provide a great user experience.

This sort of combined platform could be very powerful, especially if it’s in a form factor that can be worn all the time, like glasses, as this would allow VR and AR experiences to take place anytime, anywhere, without the need to go to a special room and put on special equipment. But what does this mean for 3D UIs? If the display transition is seamless, we would also want the interface transition to be seamless. Users who are familiar with the interface for one mode should be able to reuse that knowledge in the other mode. And a dual-mode display is not as useful if users have to pick up different input devices to use in each mode.

At the same time, as we have argued earlier, VR interactions have the potential to be much more expressive and magical than AR interactions, because VR does not have any of the limitations of the real world. 3D UI designers have a balancing act to manage here: interfaces for AR and VR that are consistent and seamless, while still taking advantage of the unique opportunities afforded by each of them.

But of course, in the most extreme situations no body-based input whatsoever (or only a very limited amount) is possible. This is the domain of brain–computer interfaces (BCIs; see Chapter 6, section 6.4.2). Current BCIs are very limited, providing only a very coarse-grained signal that can be used to produce simple triggers or choices between a limited number of alternatives. A great deal of research is needed to be able to use BCIs (perhaps in combination with other inputs such as eye tracking) for general-purpose 3D interactions such as travel and object manipulation. It is likely that machine learning will play a major role in this effort, as computers learn to decipher user intent from complex, noisy brain signals.

Currently, there is nothing even close to a standard for 3D UIs. The interface is designed and implemented separately for each new application. Some techniques and metaphors have become fairly common, but even when the same technique is used in two different applications, the implementation may be entirely different.

In order to define a standard, we would need much more empirical evidence on the usability of various techniques and metaphors and on the combination of these techniques and metaphors with various devices. In order for a standard to be practical, we would need a set of standard, generic implementations that could be reused in any application.

But perhaps the bigger issue is whether we want a standard. One might argue that 3D UIs are much more complex than 2D UIs, that the input and output devices for 3D UIs are too diverse to match up to a single standard, and that 3D UI applications need specialized, domain-specific interface designs for optimum usability.

However, with the advent of consumer-oriented VR and AR systems, there are forces that may be driving us closer to a 3D UI standard. First, the number of “real people” (not just researchers and developers) who are using these systems has grown exponentially, and these users are using many different applications. Second, the systems are trying to provide between-application experiences—basically home worlds where users can launch apps and adjust settings—and these also require their own 3D UI. Making all these experiences comprehensible and usable for consumers will push designers to define at least some ad hoc standards for basic things like selection and menus. Initially, these are likely to be lowest common denominator solutions that will work in any situation and with any technology. It will be challenging to continue to innovate in the huge design space of 3D UIs while still providing a consistent, understandable UI.

There are many potential directions here. We might need a platform-independent description language that allows developers to specify 3D UIs without specifying the details of their implementation. Not only would this be a useful abstraction, it would also provide a way for researchers to share the details of their UI designs even though they are using different platforms or engines. It would also be useful to have a plug-and-play 3D UI toolkit that has generic implementations of many common interaction techniques and allows developers to simply choose techniques (and devices) a la carte, resulting in a complete working 3D UI. Generic environments designed for prototyping techniques for particular tasks (e.g., manipulation, navigation) would keep designers from having to custom build a test environment for each prototype. Finally, programming-by-example approaches might be useful to develop working prototypes of gesture-based interfaces without implementing complex gesture recognizers; research in this direction has been promising.

An alternative would be to give evaluators separate viewpoints into the same augmented scene, using additional HWDs or tracked cameras. This would provide a natural view of the user’s actions but still not reproduce what the user is experiencing directly. A combination of the two approaches might be used, but this will require multiple evaluators and/or post-session review to make sense of the data.

Outdoor mobile AR results in additional evaluation challenges. For example, it’s not possible to control the environment (weather, lighting, presence of other people in the scene). One approach to address this issue is to run studies in an AR simulation (i.e., simulating AR with VR). AR evaluation methods are an area ripe for further research.

But longitudinal studies, which are tricky at best with traditional user interfaces, are especially problematic with 3D UIs. The effects of long-term use of VR and AR are still unknown, and evaluators run the risk of adverse effects in participants both during and after the study. Furthermore, many current 3D UIs, especially more experimental ones, do not have sufficient robustness to ensure that they will continue to work flawlessly over the course of a long study. Still, sooner or later we will need to perform such evaluations, especially since 3D UI are rapidly becoming a part of daily life.

So research on cybersickness now has much more urgency. But it also may need to shift in focus from understanding why people get sick to finding the best methods to prevent people from getting sick. We can’t simply tell users not to use the systems or to limit their use. Users want to play intense games in VR, for example. There are some promising methods for preventing cybersickness, including manipulating the environment, changing the travel technique, using anti-sickness medication, providing minimal vestibular cues to minimize cue conflict, or even stimulating the vestibular system directly. The researchers who come up with the most effective methods will have a huge impact on comfort in AR and VR.

How do long exposures to VR affect real-world perception and action, and how long do these aftereffects last? What about task performance—does it improve over the course of a long session due to learning, or does it start to break down after a certain time period? Does the level of presence increase the longer the user is immersed in VR? What are the effects on presence of notifications coming from the outside world? Are there psychological effects of being immersed in realistic virtual worlds that portray violence or scary situations? And how does long-term repeated VR exposure affect social interactions and relationships in the real world? These important questions and many others are now available to be studied for the first time.

Here, we need to study issues like divided attention—do AR displays distract users from potentially dangerous or otherwise important things in the real world? On the one hand, AR displays should be better than smartphones in this regard, because the real world is always in view, but on the other hand, the augmentations add clutter to the real world and may still be perceived as a separate layer of information, forcing users to choose to attend either to the augmentations or to the real world. Always-on AR also has important social implications, as users pay attention to and interact with their AR device while simultaneously engaging socially with other people. Finally, how do we design microinteractions with AR that enhance people’s real-world activities without distracting too much from them?

These benefits are difficult to prove, but even worse, it is difficult to even design a valid experiment to attempt to measure them. Suppose you run an experiment comparing task completion time on a typical HWD-based VR system and a typical desktop computer system and that you find significantly faster times with the HWD. What would your conclusion be? What caused the difference? Was it physical immersion, head tracking, presence, the use of a 3D input device, the use of a 3D interaction technique, or some combination of these things?

A first step toward quantifying the benefits of 3D UI technologies, then, is to use a methodology that separates all these variables so that we know from where the benefit comes. Again, we point readers to the component evaluation approach in Chapter 11, section 11.6.3. In the example above, a component evaluation might show us that the 3D input device was actually the crucial difference. Then perhaps we could achieve the same benefit by using the 3D input device with a standard desktop display monitor, avoiding the cost, complexity, and potential discomfort of the HWD-based system.

Researchers over the years have built prototypes of VR and AR applications for work and tasks in areas as diverse architectural design, construction visualization, virtual tourism, medical training, physical rehabilitation, classroom education, on-site data analysis, and intelligence analysis, just to name a few. But many of these applications were designed only as proofs-of-concept, because the time was not right to make them viable for actual work, whether in the office, in the classroom, or in the field. The challenge for 3D UI designers now is to do the difficult and detailed work of designing real-world user experiences, not just toy demonstrations.

Like many of the applications described in the previous question, there have been a large number of research prototypes, especially VR design tools meant for modeling VR scenes. There has been less research into developing interactive AR and VR applications from within AR or VR, but the concept has been explored to some extent. What hasn’t yet been proven is whether these approaches can be used in production to build real 3D applications, and whether it’s actually better, in terms of efficiency, understanding, or user satisfaction, to build 3D UIs with 3D UIs.

Some applications are purely social; for example, VR can be used to bring people from all over the world together in the same virtual space. But this raises many questions about how people should be represented (i.e., avatar design), how we give the appropriate social cues beyond speech and coarse-grained gestures, and what the appropriate social protocols are when everyone is virtual (e.g., how do we maintain “personal space” when two virtual characters can inhabit the same virtual space, or how do we have a private conversation when another user can teleport right next to us instantaneously?).

For many 3D UI application areas, collaborative, multiuser work is the norm. Consider automobile manufacturers: when a new car model is being designed, the work is not all done by a single person. A large team of specialists must work together over a long period of time to create the final design. But a collaborative 3D UI for such a team is not trivial—it’s not simply the sum of several single-user UIs. Collaboration brings with it a multitude of interface challenges, including the social challenges described above, but also issues such as awareness (Who is here? Where are they? What are they doing?) and floor control (Who is working on that object? How do I get the object?). Increasingly, we are seeing research on social and collaborative VR and AR, but there is still much to be done.