15
“Transporting” a Billion of Us with VR and AR
Building the last screen
As I was walking through the Facebook offices on March 25, 2014, a friend stopped me in surprise to read me the announcement of Facebook’s $2 billion acquisition of little-known virtual reality (VR) headset maker Oculus VR.
As a confessed tech-headline-of-the-day hunter, I was aware of Oculus’s crowd-funded indie project to put screens and head tracking sensors into what looked like heavy—and heavily duct-taped—ski goggles with an elaborate umbilical cord back to a kind of souped-up, blinking mini fridge computer that serious gamers kept under their desk the way street racers keep their perfectly waxed Japanese-made tuners in their garage.
Nevertheless, coming just a month after the $19 billion WhatsApp blockbuster had marked Facebook’s 40th acquisition, I wondered for a moment whether my friend had read me a headline from satirical news website The Onion mocking Facebook’s acquisitive streak.
How did Oculus VR fit with Facebook?
Shortly after the acquisition news had hit the public relations wire, Zuckerberg took to Facebook to address that question by asking—and answering—a question of his own:
What platforms will come next to enable even more useful, entertaining and personal experiences?
Oculus’ mission is to enable you to experience the impossible. Their technology opens up the possibility of completely new kinds of experiences.
Imagine enjoying a court side seat at a game, studying in a classroom of students and teachers all over the world or consulting with a doctor face-to-face—just by putting on goggles in your home.
One day, we believe this kind of immersive, augmented reality will become a part of daily life for billions of people.
It slowly became clear that this was just crazy enough to work. But, to believe that, you have to zoom out. A lot.
The roughly 70-year history of computing makes it undeniably clear that no platform or interface lasts in a specific state forever. Room-sized mainframe computers of the 1950s and 1960s were overtaken by cabinet-sized minicomputers in the 1970s, which were overtaken by desktop and laptop PCs in the 1980s and 1990s, which themselves were overtaken—in the largest and fastest disruption ever—by smartphones in the 2010s.
As well documented as this long pattern of disruption is, it can still catch leaders of one era by surprise. You can see in Figure 15-1 how hard it is to be paranoid during a particular platform’s salad days: the PC industry’s projections in 2004 of PCs (the upper dashed line) and smartphones (the lower dashed line) sales over the coming decade were completely upset just a half dozen years later.
This lesson is not lost on Zuckerberg. No matter how crucial the emergence of mobile—and Zuckerberg’s famous internal I’ll-end-the-meeting-if-you-don’t-show-me-a-mobile-screenshot-of-your-product-first push—was to Facebook’s unrivaled worldwide leadership in user engagement, he can’t assume that mobile in its current form will be the dominant medium in five to 10 years.
The time for the watchful CEO to look ahead was five minutes ago because perhaps the only thing better than being mobile is being “transported” by VR and its more-likely-to-succeed-at-global-scale cousin, Augmented Reality (AR).
Figure 15-1. Global sales of PCs and smartphones (millions)
How Can VR and AR Possibly Matter to You and Facebook?
We’ll get to the motley crew at Oculus working to make all this possible in a minute. They include the 23-year-old undergrad dropout you might expect, the 59-year-old PhD dropout you wouldn’t, a revered gaming wizard and the new Pixar made up of the old Pixar. Some of them have been waiting more than 25 years for technology to catch up to their vision and that of Star Trek’s holodeck, Neal Stephenson’s Snowcrash, Morpheus and Neo in The Matrix and Ernest Cline’s Ready Player One.
But first, why do virtual reality and augmented reality matter?
If you have technology that can track head movement, recompute 60 to 90 times a second what you should see corresponding to that new position from two slightly horizontally separated viewpoints simulating your eyes and display the two results on high-resolution screens mere inches from your eyes enclosed in a head-mounted display (HMD) such that they are all you see, and if it can do all of that in 20 to 30 milliseconds, you can fool a person’s vision and brain into believing that they “are” somewhere “else,” possibly “with” others. A feeling known as “presence.”
Reality, therefore, is merely what your brain believes it is.
Once you expand what someone can be made to believe through VR, you can build games that give you fantastical powers in impossible places. Watch your friends’ Italian vacation videos in their full 360-degree 3D splendor. Watch an NBA game as if you were courtside. Sit in the front row of a college lecture by a world-class expert even though you live in a small town halfway around the world. Be in a movie rather than watch it. Develop life-changing empathy to the conditions in a refugee camp without having actually visited it, perhaps making virtual reality as a medium to human suffering and inequality what video was to war whether as newsreel (World War II), broadcast (Vietnam) or gun camera (Desert Storm). Talk and work with people thousands of miles away in ways as immediate as if they were in the same room. Pull that junior high school classroom into medieval times to depths for which you would have previously needed a Michael Crichton best seller. Train athletes, doctors, pilots and soldiers in ways that are both more visceral and safer. Even use exposure therapy to help the majority of us that suffer from fears to overcome them.
A technology slightly less heady than VR but perhaps even more useful in the long term is augmented reality (AR), which shares many of the technical requirements but seeks to overlay visual information on your existing world rather than to replace that world entirely. It comes in the general form of glasses with clear lenses onto which the visuals are rendered. The visuals can be something as simple as the text of a message displayed in a corner of your glasses or as sophisticated as information related to something or someone in your field of view where the visuals are constantly re-rendered to match the location of that target in your actual view.
Once you can combine generated visuals with your actual world, you can do navigation for driving and walking the way it always should have been: right onto the path in front of you. Virtually annotate the people and things at live sporting events, concerts, theater productions and museum exhibits. Translate what you are seeing and hearing in real time. Play a game of Settlers of Catan—or Star Wars Dejarik holochess—on the empty table in front of you with your friend from Minneapolis. Extend what you are seeing on television to your family room. Work on six virtual displays on your desk instead of the 13-inch screen on your laptop. Or just watch Netflix on a virtual 120-inch TV instead of the actual 45-inch one hanging on your wall.
VR and AR are not just the next screens. They are the first screens with no edges, and, as screens capable of simulating all other screens, they are our last screens. As unlikely or even as fantastical as this may sound if you have not experienced it, VR and AR are just another step forward in the multimillennial progress of human media: from language to typography to pictures to video to . . . experiences.
And that is why they matter to Facebook. As a means to drive forward the mission to make the world more open and connected. To connect more viscerally, empathize more deeply and be transported more completely.
That, however, is not the whole truth. As much as VR and AR create possibilities for user experiences far beyond merely showing you Facebook’s News Feed on your glasses, there is one other big reason that they matter to Zuckerberg: strategic protection against being disrupted in a media transition occurring over the next five to 10 years, which in the complicated world of technology could come from below, beside or above Facebook.
Most important in the category of competitors below Facebook are the operating systems on which the service runs—especially Apple’s iOS and Google’s Android. Even today, these operating systems—which require developers of apps to exist in a controlled environment accessible only by going through Apple’s and Google’s approval process to be made available in their respective app stores—exert a tremendous amount of control that is Zuckerberg’s greatest strategic concern. Luckily, the current situation is one of mutually assured destruction as neither operating system would use its strategic choke point to block Facebook, their most heavily used app. However, if a transition to a new or evolved medium like VR and AR would allow either Apple or Google to prevent or disadvantage Facebook from evolving, things could erode quickly.
Competition beside Facebook would be another “lens” (recall Chapter 4) emerging to play a bigger role in people’s ability to connect with the world. Google is Facebook’s biggest competitor here, and if they could advance an interesting technology—or perspective on the world—using VR and AR faster than Facebook, they would threaten the stranglehold Zuckerberg has on people’s attention as their lens on the world.
And competition from above would come in the form of interesting future content and experiences—especially around VR and AR—that people want to connect with and that Facebook did not have access to or that could not be shared via Facebook. People’s attention, like water, will find a way around the obstacles of the no-longer-cool to the most interesting things.
Having the hardware, software and content efforts of Oculus—and VR and AR generally—running early in this perhaps decade-long window of strategic uncertainty gives Facebook the best possible experience, tools and bargaining chips to adjust to the realities of a significant transition if and when it occurs.
Who Are These People?
Oculus VR—which along with artificial intelligence is one of Zuckerberg’s most important bets for the foundation of Facebook’s future—started in 2011 with Palmer Luckey, a then 18-year-old homeschooled tinkerer from Long Beach, California. Luckey’s hobby of repairing smartphones threw off enough money and the kind of miniaturized electrical components necessary that he was able to turn his interest in VR headsets—he had collected more than 50 examples of the species—into building an HMD of his own.
VR HMDs have a long history of being bedeviled by technical issues that prevent a true feeling of immersion: screen resolution, width of the field of view (FOV), smearing visuals of displays with persistence delays between successive images and especially the time it takes between head movement and corresponding screen refresh, known more technically as motion-to-photon latency, that can cause discomfort when the brain experiences dissonance between movement and vision. Luckey, however—benefiting from the advances in screens and sensors that former Wired editor Chris Anderson has called “the peace dividend of the smartphone wars”—was starting to make real progress.
As is often the case in the infinite bazaar of ideas that is the Internet, Palmer’s progress attracted attention.
One of the people taking note in the summer of 2012 was pixel wizard John Carmack. Widely considered the most important and broadly influential graphics and gaming coder of the prior two decades, Carmack—42 at the time—was credited on over 40 games, including the seminal Castle Wolfenstein, Doom and Quake franchises and more than half a dozen groundbreaking graphics engines used by countless other games. His career at id software, the gaming company he founded, was the thing of legends. As the high priest of how to turn zeroes and ones into images that immerse you, Carmack could enthrall the masses at id’s conventions for three-and-a-half-hour keynotes. Even Steve Jobs’ keynote superpowers topped out at around two hours.
According to Carmack’s own recollections, the psychological evaluation he was ordered to undertake after being arrested for computer theft at the age of 14 and prior to serving a year in a juvenile home diagnosed him as a “brain on legs.”
It should come as no surprise, then, that after more than two decades of programming and 2 million lines of graphics code, the untiring Carmack began to tinker with VR headsets in June and July of 2012 on his own. However, upon asking Luckey for a prototype of the Oculus Rift—and promptly receiving from him the only working one—Carmack became a convert to the headset just ahead of the E3 gaming show in Los Angeles in August.
Another convert at that time was gaming industry veteran Brendan Iribe. The then 32-year-old was coming off stints as a user interface programmer on the strategy gaming franchise Civilization, as cofounder of a user interface technology provider that had been sold to Autodesk, and as product team lead at a streaming gaming provider that had just been acquired in July 2012 by Sony for its PlayStation franchise.
Casting about, Iribe’s keen eye for the visual future of computing caught Luckey’s ever growing Oculus Rift campfire, and Iribe would come aboard as Oculus’s CEO just ahead of the crowd-funding campaign that would change the course of VR.
Following the wildly successful Kickstarter campaign in August of 2012 that had aimed to secure $250,000 from backers but wound up receiving $2.4 million in the first 24 hours alone, and delivering the first Oculus Rift Developer Kit headset (DK1) in March of 2013, John Carmack would formally end his 22-year career at id software to become CTO of Oculus in August of 2013. Saying that the feeling of “presence is religion on contact,” Carmack was ready to start the next chapter of his prolific career.
In March 2014, in a scene mirroring conversations with Kevin Systrom over Instagram and Jan Koum over WhatsApp, Iribe struck the $2 billion deal to be acquired by Facebook one-on-one at Zuckerberg’s house.
Only a few days later, Oculus would announce the arrival of the final puzzle piece in their technical team as computer graphics and VR veteran Michael Abrash joined as chief scientist. The then-57-year-old Abrash was a respected programmer with stints at Microsoft’s operating system, natural language processing (NLP) research and XBOX divisions, id (where he had worked with Carmack), gaming tools developer RAD, Intel and game developer Valve. More importantly, he was also an expert on the human perceptual system and a VR philosopher. It is Abrash who most often makes poetic pronouncements like, “If the world wide web was alpha, then VR is omega.”
Between Luckey, Carmack and Abrash, Oculus VR has an outsized share of the best pixel pushers in the world, but they haven’t stopped there. To make sure that Oculus is also a leader in the actual pixels that are being pushed, they formed Oculus Story Studio in August 2014, a combination of lab and production studio meant to discover and openly share with the industry how to move various media forward into the rich—but poorly explored—opportunities of VR and AR.
Oculus Story Studio is a Pixar for VR. Fittingly, it is led and comprised of many former Pixar employees.
Oculus Story Studio Creative Director Saschka Unseld was at Pixar for six years, including as director of the short film Blue Umbrella; Technical Director Max Planck was at Pixar for a decade and worked on half a dozen of their feature films; and Director Ramiro Lopez Dau was at Pixar for five years and worked on three films. All in all, as of spring 2016, a third of Oculus Story Studio’s 30 employees had stints at Pixar in their backgrounds. Their first three VR “films,” Lost, Henry and Dear Angelica, featured a missing robot hand finding its owner, a lonely hedgehog who likes hugs and balloons, and a daughter going through her mother’s memories in the form of 3D impressionistic illustrations.
All these efforts would culminate—nearly five year’s after Palmer Luckey’s first prototype—in the launch of Oculus VR’s first consumer version of the Rift headset (CV1) in March 2016. A successor is already planned for 2018 or 2019.
How We Go from Ski Goggles to a Billion Users
Virtual and augmented reality are powerful magic, so the current push is not the first time the industry has tried their hand at it. First efforts on VR HMDs came from graphics pioneer Ivan Sutherland in the late 1960s and were revived in the mid-1980s by Jaron Lanier and his company VPL Research, whose HMD was called—no kidding—Eye Phone. Cultural figures no less important than Timothy Leary would partake in the technology, but the demands of making your brain truly believe in an alternate reality exceeded what was technically possible in the past.
But, as of 2016, we’re finally at a point to take VR and AR to consumer scale, and Facebook is certainly not the only one working on it.
With all six of the largest consumer technology companies in the world deeply invested and feverishly in development, VR and AR have become too big to fail.
Oculus Rift, Samsung GearVR (powered by Oculus), Google Cardboard and Google’s Magic Leap investment (and perhaps even the second coming of Google Glass), Sony PlayStation VR and Microsoft HoloLens are already public, and the eventual entry of Apple is presumed, given hiring headlines and CEO Tim Cook’s pronouncement that VR and AR are not a niche application.
However, as much as 2016 saw the launches of the best VR to date with Oculus Rift, HTC Vive and PlayStation VR, this generation of hardware—not portable and tethered to a PC or gaming console—is merely intended for millions of early adopter enthusiasts consisting mostly of gamers. HMDs of 2016 are to the future of VR and AR what Gordon Gecko’s brick-like Motorola cellular phone is to your rose-gold iPhone 7.
The devices that will turn VR and AR into an interface for hundreds of millions will blend into our lives much more easily. To do that, they will embrace the one device that rules them all—the smartphone—and in doing so pull it out of its stagnation.
They will look and feel much like ordinary large-framed glasses and feature clear lenses with built-in waveguides transmitting the output of tiny projectors on each arm across the entire surface of each lens, covering a total 120-degree field of view with 90-frame/second, 1,080p video (960 × 1080 resolution for each eye) that will be delivered wirelessly by your phone. In the out years it’s even possible the projection-plus-waveguide combination will be replaced with extremely high-resolution transparent edge-to-edge LCDs in each lens.
Instead of distinguishing between AR and VR, these glasses will have the ability to go from being completely clear (off) to displaying images that augment what you see (AR) to displaying images that cover the lenses edge to edge (VR), leveraging next-generation phones for all the computing necessary. While they will sacrifice some of the image quality and perfect sense of virtual reality “presence” that large, fully enclosed HMDs tethered to PCs can deliver, they will suffice for the vast majority of popular VR and AR applications and be dramatically easier to wear and carry, driving much greater consumer adoption than more complex HMDs, which will stick around to serve the high-end gaming community.
While the performance and miniaturization demanded by this lightweight alternative are not available at consumer scale in 2016, prototypes of the necessary technologies exist, making the first easy-to-wear, consumer-grade VR/AR glasses plausible between 2018 and 2020.
With this approach, the glasses benefit from the popularity, portability, connectivity, computing, touch and voice controls that the phone can deliver, and the phone benefits from the new display options and applications the glasses make possible.
Instead of an entirely new and different platform, these VR/AR glasses will be an evolution and extension of mobile.
Looking at the major players for mainstream VR/AR glasses, I would project another competition between Apple (starting in the form of Apple Glasses 1 together with a new iPhone between 2018 and 2020) and a federation of hardware suppliers—including Samsung and Facebook’s Oculus—on a Google platform connected to Android. On the services and content front, expect Facebook to evolve all its offerings—Facebook, Instagram, Messenger, WhatsApp and potential new additions—to take advantage of the capabilities that AR and VR offer and to make them available for both Apple and Google hardware platforms as they do today. Expect Facebook also to have to wrestle with uncomfortable questions about the topic of pornography in VR and AR. Facebook and Instagram have always had policies preventing such content, and, similar to Apple’s App Store and Google Play, the Oculus VR and Samsung GearVR stores will not allow adult entertainment either, but unlike the Apple iPhone and Google Android phone, Oculus VR is open to all developers, and pornography is rapidly finding its way onto the platform.
How big can the extension of the mobile medium to VR and AR get? Looking at the market for tablets that began in earnest in 2010 with the introduction of the Apple iPad may be instructive as an upper bound for projections since prices—anywhere from $300 to $800—and use cases are similar:
Since its introduction in 2010, nearly 300 million Apple iPads have been sold.
Total annual tablet sales are predicted to be 300 million by 2018.
An installed base of about 1 billion is expected by 2018.
Keeping in mind that VR/AR glasses will be a newer medium in 2018 than tablets were in 2010 and that glasses may not drop to low-end price points as rapidly as the more familiar tablet technologies, it may be difficult for glasses to get all the way to an installed base of 1 billion by 2026—eight years after their mainstreaming in 2018. But an installed base above 500 million by 2026 is plausible, and as our “last screen,” the glasses are likely to continue to build momentum to an installed base of 1 billion, especially as the smartphones powering the glasses are expected to go to a total installed base in excess of 5 billion in that time frame.
Transporting a billion of us is no less likely than connecting a billion of us to begin with.