6
AR and MR
AUGMENTED REALITY’S (AR) breakout moment in learning and development has arrived. The hype machine is in high gear. The barriers to entry are lower than ever before. All this is not so subtly telling us that AR is here, and it will change everything!
OK, maybe not everything. In all honesty, we could swap out AR for any other “new” technology from the recent past and I’m sure similar things were said. Web 2.0, e-learning, mobile learning, and Flash all had their own everything! hype. Some lived up to it. Others did not. Some still are going strong while others have seen their time come to an end. Despite it all however, each had a moment of new opportunities and expectation to change L&D as we know it.
Used in consumer applications like games, shopping, and other everyday apps, AR is seeing a lot of buzz. That buzz is carrying over into the L&D industry. But while I believe it has the power to transform learning, it may not change everything—just some of the things. AR can enhance our content and engage our learners in an effective and meaningful way. By mixing real and digital information it can supplement learning on demand. In fact, as a learning development tool, AR enhances our ability to provide meaningful experiences. But we all know buzz only goes so far and then questions arise. Where do you get started? How should you use it? And, possibly most important, what should you be using it for?
These are not easy questions to answer, and the answers may be different depending on your situation. So, let’s make this our journey. Let’s explore these questions and discuss what the answers might look like, and along the way interact with some AR experiences.
To interact with this chapter, follow these steps:
• Download the Zappar App from the app store on your smartphone and open it.
• Locate an image in the chapter (Figure 6-1, for example) and aim your camera at it.
• Make sure the image is in full view of your camera to activate, then listen, watch, and interact!
Figure 6-1. Augmented Reality: Open, Aim, Interact!
What Is Augmented Reality?
Before we can understand what AR is, we have to understand immersive technology, which is the bigger trend it’s part of. Immersive technology encompasses virtual reality (VR), 360 virtual video, AR, and mixed reality (MR).
Reality Options
We will be focusing on AR and MR in this chapter, but it’s worth understanding all facets of immersive technology for context:
• Virtual reality is a simulated environment that completely engages the user’s senses, cutting off sight and sound from the outside world. VR experiences are computer-designed renderings of a space, which are interactive, 3-D models. The user has a headset, usually a controller or two, and are often tethered by a data cable to a PC (except for a few hardware offerings). These experiences offer a complete immersion into a created space, which simulates a real or fictitious environment for play, exploration, or learning.
• 360 images and video work like VR environments, only they represent captures from the real world. These 360 videos can be viewed within a fully immersive headset or a handheld digital device. In either case, the user can then look in all directions to explore what is happening.
• Augmented reality is the digital overlay of information into a real-world space (Figure 6-2). In an AR experience, the hardware recognizes real-world objects or locations and adds digital content to enhance them. Videos, documents, and hyperlinks can all be used as overlays. These elements can be passive or active for users to explore. The most common way to consume AR is through a handheld digital device like a smartphone or tablet. There are also wearable hardware offerings that provide a hands-free experience.
• Mixed reality is AR with the addition of real-world interaction (Figure 6-3). Where AR overlays information onto an object or space, MR recognizes and interacts with that object or space. For example, MR scans the table or floor in front of you. It then projects the experience onto those areas to add a greater immersion to enhance the experience. Tables, floors, walls, doors, and more can be used to create more realistic engagements and interactions. In an MR experience, you can place a person, place, or thing right in your room to interact with or explore.
Figure 6-2. Augmented Reality Digital Overlay
Figure 6-3. Mixed Reality Portal: Scan, Place on Flat Surface, Interact
For the remainder of this chapter I will discuss and show examples of both AR and MR. Although they share some attributes, I will point out any differences or considerations that need to be made.
What You Need to Know About AR
Augmented reality lends itself well to training. Within the L&D space, a running theme has been that hands-on learning is a better choice than other modalities when possible and appropriate. AR enables us to put this to the test. By engaging in content with a more direct and active approach, we can add as much or as little to the experience as we want.
Let’s talk about the why, how, where, and what’s next for AR. We’ll look at why it’s a good place to start, discuss the tools to create and consume AR, peer a little closer at investment, and share some thoughts on what the future of AR looks like.
Why Choose AR?
So, let’s talk about why augmented reality is the best immersive choice. Accessibility. This is why AR has legs, so to speak, compared with the other immersive options. Almost everyone you know (or don’t know for that matter) has an AR-ready tool in their pocket: a smartphone. Roughly 2.5 to 3 billion people have smartphones globally (Pew Research Center 2019). This means learners are immediately able to take part in your AR experiences. Plus, many smartphone users won’t think twice when told to download an app, take a selfie, or maneuver around to scan a code or symbol. Each of these actions are also ways to engage with AR. It’s like we have been training for this our entire smartphone-using lives.
This also means that AR doesn’t require new equipment or new training on how to use that equipment, for most learners. VR, on the other hand, often requires expensive headsets and unfamiliar interfaces. With smartphone-enabled AR, users can start learning almost immediately, with little time spent explaining the tool itself.
Users can experience AR a few different ways, such as through mobile device apps, browser-based web AR, and programmed wearable devices. Each one has pros and cons, as well as varying levels of complexity to get them ready for your users.
How to Create AR
As creators, the tools we use must accommodate our skills, schedules, and expected outcomes. They should be flexible and able to meet our waterfall and Agile workflows. Augmented reality creations can vary based on need—they can involve simple interactions lasting a minute, all the way to complex, multistep branching experiences. Our technical ability and the ability of our teams will limit the type of experiences that we can create, AR or otherwise.
The wide variety of AR software available today allows for teams with little experience to develop engaging programs. However, it’s important to remember that software is only a vehicle. L&D programs must be built based on needs analysis, objectives, and audience identification.
Two considerations when choosing AR development software include your current skill set and how complex an AR experience you want to build. AR software can be organized into three generic skill categories: simple, moderate, and advanced (I told you they were generic!). Breaking down each can help show what is needed from a skills and resources perspective. They have their own design advantages and learning curves as well as hurdles.
• Simple: Looking for a no-code, push-button solution? Try a cloud-based editor with a WYSIWYG (what you see is what you get) interface:
Possible apps: ZapWorks Designer, Spark AR, or Wikitude 3.0
Coding needed: None
Learning curve: Low; appropriate for all skill levels. Intuitive, easy to use, add content, and share
Delivery: Mobile, handheld digital device through an app
• Moderate: Looking to develop more interactive and immersive AR experiences? Know enough code to be dangerous? Use a cloud-based editor or standalone program:
Possible apps: Zappar Studio, Roar, Wiarframe, or Entiti 3.0
Coding needed: Little to none; at this level there is still a good mix of WYSIWYG with the option to enhance through basic coding (JavaScript or typescript are common code languages at this level)
Learning curve: Medium; a basic understanding of how code works and how to use assets common to e-learning development will allow you to get started here
Delivery: Mobile, handheld digital devices through an app or a web browser using web AR, or a wearable device programmed with the experience
• Advanced: Have an advanced background in coding? Looking to develop standalone apps with multiple types of media and advanced interactions?
Possible apps: Unity 3D, Unreal Engine, Amazon Sumerian, or Vuforia
Coding needed: Little to significant depending on delivery and experience
Learning curve: Difficult; an understanding of intermediate to advanced coding concepts within the software being used (JavaScript, typescript, and C# are common coding languages at this level); a more advanced understanding of multimedia and graphic design is helpful
Delivery: Mobile, handheld digital devices through an app or a web browser using web AR, or wearable device programmed with the experience
The options I list out here are not exhaustive, and there are quite a few development tools available at each level. I encourage you to find one that works for your skill level, project needs, and IT department.
Keep in mind your or your team’s skill set as well. Coding isn’t exactly a common skill, and many instructional designers don’t have it. But regardless of your level, you will never be called upon to sit down with notepad and write out 100 lines of code from scratch. You will need to learn how to code within the confines of the development tool you choose. Most will have an interface that allows you to set up the basic structure of your experience by importing and moving content around like any other development tool. Coding in this instance is done as an add-on that allows for more advanced actions and interactions to be created.
The good news is that there will never be a lack of software options to choose in this space. From time to time it’s also worth looking at what’s new and reevaluating your current development tool. Future software will also continue to become more user friendly, requiring less and less coding knowledge to create advanced and engaging experiences. With that evolution, more developers, teams, and organizations will be able to expand into AR development with less cost and training needed.
One last quality to research in a potential software or tool choice is longevity. You want to make sure the choice you make is going to be supported for a while. If you’re going to commit to using software to support your business or clients, the company that created it can’t fold within a year. AR has seen its share of companies come and go already—in some cases, leaving early adopters and their organizations in a bind. Imagine building a global campaign to roll out a new training or initiative only to have the vendor close its doors three months after your roll out. It’s happened. And it will happen again. Do your homework and get to know the vendor before committing so the chances of it happening to you are low. Doing this research could save your company time and money, and it can also save you some face. You don’t want to be the one who suggested a new direction only to have the rug pulled out from under you. It could save you from future pushback or rejection the next time you suggest new technology.
Experiencing AR
AR devices are expensive. Mobile phones, the smart kind, are expensive. Wearable devices are also very expensive. There’s no getting around it—for this kind of hardware you’re going to have to pay a premium. Today the two main ways of experiencing AR are through mobile and wearable devices.
AR Through Your Mobile Device
When you implement AR on mobile, the cost for the device has been absorbed by your learners. In the best-case scenario, your learners will own the latest smartphone model and their workplace will have a bring-your-own-device (BYOD) policy in place. This means you’re ready to go—the only cost you will incur from an L&D perspective is development.
Now I know this is not the case for every potential user. However, for the sake of this discussion I’m playing the odds that it’s true for most potential users. This also goes with the understanding that there will be exceptions who need hardware and extra coaching. Some organizations will also have a less flexible or nonexistent BYOD policy. These are all barriers you will need to address through vendor and IT discussions. In fact, the success of your plan for AR may rest on your ability to navigate these types of barriers, and they won’t be the only ones. While I don’t have all the answers for the unique experiences and situations in your organization that you’ll have to navigate, I leave you with some thoughts on addressing common barriers you might face at the end of this chapter.
Our ability to use mobile AR, as mentioned, is already built into our day-to-day usage of mobile devices. We download, upload, scan, and snap all the time. This means your potential users have an existing skill set you can design and plan for. This allows us to get started in AR with little to no learning curve. For this chapter I have already asked you to download an app and point it at some pictures. Was it difficult or second nature? Keep in mind how you felt during those moments. This is also how your learners will feel when asked to do the same or similar steps. They will be pulling from their own experience using a smartphone or tablet.
SETTING THE LEARNER UP FOR AR SUCCESS
So, what do you do if your learner simply doesn’t want to download an app, or struggles with it? There’s a good chance that at least a small number of your potential users will have some issues. In these cases, it’s important to keep a few things in mind. Keep it simple and make the experiences valuable. Let’s take this chapter as an example.
At the beginning, you were asked to download the app and scan any images along the way. Each image provided a direct example of the content being discussed and gave you a chance to experience the technology—they added value and context. With the exception of downloading and opening the app, the barrier was relatively low.
By doing these two things, I tried to set you (the learner) up for success. This is especially important in this delivery method because it doesn’t allow for additional support by the moderator, developer, or instructor. So, if you had any issues you would have to seek out your own solution.
In your own environments, you can make additional tools and support available. You can make AR content unique from other training pieces, tie it to a completion event, or incorporate it into a group activity to add a peer element. The peer element is a good way to get people involved if you’re concerned about your learners’ overall technical ability—a certain bond forms when trying, failing, and learning together.
But remember, you won’t convince everyone. Some learners may outright refuse to learn the technology, but more than likely you’ll see them just get by. They will find the AR equivalent of just hitting the next button as quick as they can. Not everyone wants to participate. All we can do is the best we can.
Experiencing AR Without a Phone
Are mobile devices an ideal way to experience AR? No, but they are an effective way to get started and have your learners experience AR. However, it’s also important to think about and recognize that there are some limitations—participants will have a device in their hands throughout the experience. Keep that in mind as you develop AR experiences; until other options become available, and affordable, this is our reality.
In an ideal world we would have our hands free to interact and move about during the AR experience. Think of the heads-up display in Iron Man or the floating screen in The Minority Report—the users are simply swiping, pinching, and zooming through with ease. This is where we want to be, and to some extent this is where the current prototypes of hands-free AR are trying to take us.
A hands-free option would be much more fluid to experience and to design for. Imagine an experience that takes users on a hike, climb, or crawl. Actually, you don’t have to imagine all that much because options like this are available today; they’re just limited by quality and cost. These offer a glimpse into what the future might look like. The goal is for participants to experience AR in a seamless way.
Wearable Devices, Where Are You?
There are currently several wearable options for experiencing augmented reality. The big players like Microsoft, Facebook, Amazon, and Google are quite active in this space and other companies are getting involved as well. A few issues with most wearables are that they look silly, don’t deliver the quality we hope for, and are priced in a way that makes adoption at scale very tough. These limitations will continue to be an issue in the short term.
We also need to consider the development aspects for wearable devices. The current software works well for the development of mobile AR, but software and skill sets will likely have to evolve to accommodate wearable capabilities.
The good news is that all things wearable are trending in the right direction. Quality and accessibility are improving and the price point for entry has gone down a notch. I think now we’re just waiting for that one wearable product that takes it to the next level—the wearable that sets the standard for how we will be using the technology for the next several years.
If you’re in the market to get started with wearables today, make sure to do your research and strongly consider partnering with a learning development company or vendor. Partnering is crucial, especially if you are considering scaling up to an enterprise. Keep the limitations I mentioned in mind, demo as much as you can, and make sure you have the software and skill set necessary to develop the kind of content you want.
Where Does AR Go From Here?
With everything said and done, the biggest takeaway about AR is that its use is growing and if it hasn’t arrived already it will be on your doorstep soon. According to the ARVR Association there has been steady growth and the trend continues upward (Figure 6-4).
Figure 6-4. AR Future Trends
Consumers and industry are seeing an interest in immersive technology, and this upward trend is going to carry it into the L&D world. Where is the best fit? The honest and simple answer is whenever it makes the most sense, wherever can it be the most useful, and wherever can it have the most impact. This is going to involve trial and error, pilots, and institutional support. This answer is obviously more complex and unique to each use case than the process of selecting from earlier forms of training delivery, but then again, so is AR technology itself.
For all the hype we’ve seen so far, AR for L&D is still in its infancy. By the time this book publishes that breakthrough product may have been released, the perfect use case found, or the most affordable and useful wearable developed. Whatever is coming our way, here are a few ways to prepare:
• Dip your toe. This is a no brainer. If you haven’t already looked at it, played with it, or prototyped something to show your team or organization, then it’s time to do so. Get started!
• Start small and work your way up. AR has a lot of potential—so much so that it can be overwhelming. Start with a small project. Curate some content into a one-stop scan that pulls everything into a device. Add instructions or a video to a step in a process. Record a daily update or quota video that operators or learners can scan in the morning as part of a daily stand-up. Enhance your business cards. Build something small but cool.
• Pilot, pilot, pilot. Build your experiences and share. Get feedback from your audience: your learners, co-workers, managers, or anyone else willing to be a part of it. Get to know how your learners will interact, react, and engage with a new approach like AR.
• Keep learning. I imagine with this audience continuous learning is a given. There are plenty of resources available, including vendor support, conferences, webinars, and local ATD meet-ups. Start a group and learn together—it’s always better than going it alone. You don’t need to be a coder, but understanding how code works can help you design better. Improve your skill set. Keep up with the new hardware and software options that become available. Stay current with the latest ways AR is being used.
• Share. Blog, give presentations, record videos, do town halls, or share what you know however you are most comfortable. Share what works and what doesn’t. This will be the key to AR becoming as useful as it can be for us in the L&D industry.
Conclusion
I wanted to leave you with some sage advice to make your use of AR in future L&D programs a success. But no single thing came to mind. In fact, I am honestly just like you: I’m still learning how, when, and where is best to use AR. I’m trying to design the most engaging, interesting, and effective experiences for my learners every time. I struggle with software, hardware, and delivery of the content. Everything you have experienced, or have yet to experience, I have too. It’s important to remember that we are in this together, and collectively sharing and learning from one another is the only way we get better. I didn’t learn early enough that I didn’t have to be an island in learning this. There are websites with information, conferences for sharing demos and case studies, and peers in the wild experiencing the same things we are. Reach out, join groups, share, learn, then rinse and repeat.
Throughout this chapter I have tried to share the information I wish I had known when I was starting down this path, especially understanding the different reality options, including the difference between AR and MR. We covered some of the software and hardware options used to build and deliver an AR experience. It was also my intent to give some context into the how, when, and why behind using AR for L&D. Now it’s up to you to take and use this information to make it work for your L&D world.
So, the advice I have for you is pretty straightforward. Enjoy it; this is a good time to be in L&D. Embrace the technology—it’s your friend and it can help in so many ways. Do something new and unique; don’t take something new and try to fit it in an old box. Take the time to design something that makes a difference to your learner’s experience with AR. Then share it with the rest of us.