Why did Boeing start an AR program? And why is it particularly relevant now?

“In my world [it] is a technical answer. The desire to do AR has been around for a long time. But why now...it’s because the two hardest parts of doing augmented reality are tracking—which is understanding things in 3D space, where things are...where a viewer is...or where you’re holding a tablet or cell phone precisely—and how you do the visualization: How do you combine digital data on top of the real world? Is it on a headset, is it on a tablet? There’s many ways to do it,” says Davies.

Both of those two things he mentioned—tracking and visualization—are now becoming possible, whereas in the past they required workarounds and stopgap solutions. AR has been difficult in a realistic production setting in a factory.

“We’re just on the cusp of being able to do it,” he says. “I think that’s why now.”

As to the question of why do AR at all, Davis gives a simpler answer: most companies build and design products in the virtual space using software, yet they build those products in the physical space. Any technology that connects those two worlds makes it easier for humans—nothing gets lost in translation.

In Davies’s words: any thing we can do to bring those two spaces closer together is only going to help people understand design [and] interpret information easier or faster.

Davies says the most common way of tracking things has historically been to use markers—essentially 2D images, which look a lot like barcodes or QR codes, that let a camera identify the physical object, matching it up to the digital version that appears within the AR software. But now motion capture systems for tracking objects in 3D are very fast, and more 3D mapping, tracking, and localization technologies are available. This is to say: markers are one way to go, but they may soon be a thing of the past, particularly in large-scale factory settings. “My view has always been, nobody wants to put markers everywhere. It’s time-consuming. You have to add them, they have to be positioned very precisely, and all these things,” says Davies. “I know other companies and other people will tell you that they’re using markers and they work great. There’s different folks, different strokes. But now we’re moving away from markers, and we’re starting to see more things that people have used—including ourselves. We’re seeing more 3D mapping and tracking and localization machines. They don’t even need markers.”

What is Boeing doing in industrial AR right now?

Boeing is also conducting pilots around remote support and for visualizing “stay out” zones—showing people doing aircraft assembly which areas to avoid when adding wiring bundles. This is necessary because sometimes the major components are assembled in what is essentially “reverse” order, and wiring is more difficult to move later.

“So as somebody is connecting up a harness,” Davies explains, “they’re visualizing the stay-out zones and they have to keep the harness out of that zone to prevent any problems later when they come to install other parts of the assembly.”

Davies also pointed to one of the biggest challenges in industrial enterprise AR right now: connecting everything in with legacy systems. When building an AR program and strategy it is important to consider four things:

• How you get the data from a manufacturing execution system in the first place?

• How you transmit that data over the network securely into a device?

• How you translate it from traditional instructions into an appropriate format for AR?

• And finally, how do you feed information from an AR headset or tablet on the shop floor back into a manufacturing system?

Right now, those linkages—largely—do not exist. When it comes to designing AR systems, there are a lot of people looking at tracking, visualization, and headsets. Far fewer people are looking at the bigger and very real question: How do I link this to my systems, which probably were not originally built with AR in mind?

ThingWorx Studio is a platform that is addressing this issue, and one possible AR solution for creating links between industrial systems. In Chapter 3’s tutorial, we will look at how it works and test it as a tool for building an AR experience.

How can AR fix failures that cause million-dollar delays?

McCombe says, “Let’s just say a very, very basic fault occurs on an oil rig and it requires a change-out of a seal, a very basic part. An O-ring seal needs to be changed. It’s taken the whole rig down, so you’re on a huge delay. And for whatever reason you don’t have a spare, but you have something similar. So you now need to get sign-off from higher-ups to use that. But those questions the authorities ask is: What is it? Where is it? What’s the risk if it goes wrong? Is it compatible?”

McCombe continued by saying that when all of these questions come up, “you’re able to dial [others in the company] in and share that information with them...they [can] give you the sign-off there and then.”

That is to say: AR basically gives you eyes and ears to see and assess the problem, as if you were the one actually on the rig.

McComb—who operates from both the US and Sydney—pointed out an interesting book written by another Australian, Andrew Hopkins, called Disastrous Decisions: The Human and Organisational Causes of the Gulf of Mexico Blowout. Hopkins is an expert on process safety who works in corporate psychology. His book describes what’s known in the oil and gas industry as the “Swiss cheese bottom,” the moment when all the holes, issues, and problems line up to create a serious, life-threatening, and costly problem. The reason McComb mentioned the book is this: he thinks AR can prevent the Swiss cheese bottom from happening, and that it is a great way for an entire industry to come together to create tools that can be shared collaboratively. Once one company or consortium builds an AR system for a particular use case, many companies could benefit.

“I find it fascinating because the amount of holes that lined up in the Swiss cheese [for the Gulf],” says McComb. “It’s very rare, but when they do, you have an incident. And I believe that AR will be the key to making sure that we don’t ever have that scenario, because there’s too many eyes and ears watching over that thing for it to occur.”

“Industries as a whole are going to need to invest money collaboratively to solve these things,” he continues. “Safety is everyone’s [issue].”

AR can be used as a tool that benefits society, as well as individual companies. That also points to one of the primary use cases for enterprise AR right now: connecting subject matter experts, remotely, to any other worker in the world.

Why does this matter so much right now? Because of our changing workforce.

How can AR affect our changing workforce?

As a global economy we are facing some significant challenges in and around the future of work—as the previous chapter discussed, we have an aging workforce and a shortage of labor for industries like manufacturing. A World Economic Forum study quoted in the Wall Street Journal said that even in China there will be a shortage of more than 200 million workers by 2050. One solution: remotely connecting more skilled workers to less skilled ones, enabling knowledge transfer—and potentially, training from one to many.

In a panel at O’Reilly’s 2016 “NextEconomy” summit, Mary Gray from Microsoft Research noted that when looking at the future of work, we should be thinking more globally. Talent comes from everywhere. Age is not a barrier. With translation services, neither is language. AR can equip companies to broaden their talent pool while hiring local labor with different types of expertise.

Consider also that the next generation of workers coming into the industrial world are used to digital interfaces. They grew up on video games. Being able to give them a toolset they’re comfortable with is important. And tools like Dotty’s 3D exchange help to fulfill that need.

As McComb concludes: “Managing business risk at the moment is huge. To not invest in [AR] now is to potentially not have a solution. With large companies, the chance of this working out of the box is fairly unlikely. So you need to start customizing an AR solution that works for your industry and for your sector.”

How is AR used to reduce complexity?

When I asked Kenly and Markle to share some examples of how AR is being embedded into products, they gave two examples. The first is from a global fork truck manufacturer. A fork truck is a more complex industrial product than you might imagine. It has more than 1,000 parts. And inside it, mechanical, electronic, hydraulic, and computing systems are all assembled in very tight spaces. There’s not a gap of space in these things anywhere, and they’re really difficult to service. In a lot of cases you have to take components out to get to other components. “You might have two or three layers of systems to get to something,” Markle says.

Because of the complexity of the product, it is really hard to find qualified service technicians. This particular company has educated their workforce through classroom training for many years, and they’ve been watching the washout rate rise dramatically year over year. The feedback they’re getting from technicians who wash out is that they don’t like classroom-style training and instead prefer YouTube videos. Thus, the company is trying to find different mediums to connect with this generation. That, Markle says, is the reason they are turning to augmented reality. “They’re looking at augmented reality as a way to provide the information in real time, perhaps with less skilled and trained technicians,” says Markle. “[They want] ways to better engage them in the process, and to be able to push engineering revisions down to the technicians more quickly.”

Kenly and Markle also spoke of another case study that points out how AR can be especially useful in emerging markets. In this case, it is being rolled out to ensure accurate service of complex medical equipment—a sophisticated electromechanical faucet used in hospitals in infectious disease wards.

How can AR ensure the fix fits the problem?

“Imagine somebody who was dealing with a patient with a disease that in an unconfined state would kill people—bad, bad bugs,” explains Markle. “The goal is being able to wash your hands in that situation without touching anything.”

The way this faucet is designed is highly sophisticated. It is loaded with sensors so that it understands the presence of a nurse or doctor in their protective clothing. It gets the temperature right, and allows them to wash their hands or whatever else they need to wash. And then it runs a cycle where it sterilizes itself with exceptionally hot water. All of this needs to be measured; in the UK and other places, it’s required that you log that the faucet completed its sterilization procedure properly.

Filling that need for more information is where industrial AR comes in. “The problem,” Markle says, “is the traditional plumber is not a computer technician and doesn’t deal with servos and motors and sensors and the things that are there, so they’ve got a real issue with making sure that it’s serviced appropriately. They [can] use AR as the overlay, so they see the schematic and the diagrams.”

The other reason this company is especially excited about AR: there is high demand for the product in emerging economies, whereas Markle says, “the health system is a little bit frailer, and the disease issues are much more significant.” In those markets, in addition to augmenting the technical skills needed for product repair, the company also sees AR as a way of overcoming language barriers. Reading a manual or verbally receiving information about how to complete a repair gets tricky when you’re working in multiple languages. Because augmented reality is primarily visual, they think of it as a tool to help them to drive greater penetration of the product in markets that are non-English-speaking—which is to say, a lot of the world.

Just like the other interviewees, Kenly and Markle have specific preferences and recommendations about how to build enterprise AR experiences. In their case, rather than building from scratch, they prefer to use PTC’s ThingWorx technology. I asked them why.

“What’s powerful about PTC’s platform and technology is that they have a legacy history going back to the 80s of providing CAD models, where all the products are designed and set up before they’re manufactured,” says Markle. “You can take the geometry and the parametrics of products that are already designed. For example, every single part of this fork truck is sitting in PTC technology, and it can be translated into augmented reality very quickly using ThingWorx Studio.”

“Now you’re talking about not having to rebuild any geometry. Now you’re talking about animating...and deciding how you’re going to make the visual experience overlay the actual physical experience.”

How does Caterpillar use AR and VR to change the way work is done?

Lewis says their AR and VR work includes views of various pieces of equipment, allowing people to physically compare products and understand dimensions. It includes apps that make maintenance easier—which is common across all industries. It also includes virtual, visual operating instructions that replace paper manuals. As Lewis says:

Every product that goes out the door has got an owning and operations maintenance manual...you can’t normally find it. Now with AR, you can just take your cell phone and app and look around. And it shows you [everything you need to know]. Our vision too is taking the smart, connected product and putting it also on the desktop of a Caterpillar engineer, and taking a lot of data about our product and putting that in a visual context so they can see. As an example, for an engine, we’re taking pressure and temperature so you can see actually how they’re playing out on the engine.

Because the technology that supports enterprise AR has advanced so significantly in the last two years, Caterpillar is pretty far along in the development process. Lewis says that they’ve already done proof of concepts and have gotten “all of those applications working.” Interestingly, they have also created a corporation-wide, cross-brand team to compare notes and best practices on all of their AR and VR work. Lewis has been leading this charge. As she says, “Late last year, we found out that there was a lot of innovation going on in the company around this space. We had a workshop. We brought everybody together, and we found that there were seven major workstreams going on in the AR/VR space—and some had been going on for over a year.”

“[That was about] leveraging the innovation that’s going on because people are trying to solve business problems and collaborating together to start putting that into the technology roadmap [and] to start getting that into production in the most cost-effective way across a large, very complex organization,” she says.

Unifying those efforts across a massive corporation in and of itself is a huge accomplishment. And, as Lewis also admits, while it is one thing to do proof of concept and get things technically working, it’s another thing to actually put it into production. There is still work to be done.

“Everybody gets really excited about it, but the dirty, rotten secret is data is messy. That’s the hard part of getting it into production. And because people didn’t know about it—we’ve got products that have been designed for years, you’ve got to go back and get some of that product structure and the data structure into digital media,” she concludes.

Lewis points to one reason so many divisions at Caterpillar have been interested in the technology: Caterpillar’s equipment operates remotely and in hazardous environments. It can be hard to find trained technicians. With AR you can have a smart domain expert remotely troubleshoot and not have to fly them out there: “Dispatching your experts all over the world—you don’t have to do that anymore. They can sit at their desks, and they can take a less skilled technician and walk them through a process and visually be there. Less travel, and then having fewer experts and increasing productivity in leveraging them, is a big value.”

Another big motivation is reducing the cost, Lewis explains.

One of the things we looked at was—we’re still having a bit of a discussion—operations and maintenance manuals. If we send a gen set out, legally we send them out with an operations and maintenance manual, but you can’t usually find them and that doesn’t help anybody. It might make the lawyers happy, but how do we really help our customers use our equipment? How do we also reduce the cost of operations and maintenance manual? If you get consumer products anymore, the operations and maintenance manuals are really simple. It’s pretty basic. If you want any more information, you go download some huge PDF off of a website. In lieu of that, we have a basic manual but then you’ve got an app. Digital media it’s always going to make it easier to keep it more updated and relevant to the product. We’re a global company, so you also have to put the manual in languages and print media.

How can incremental savings of AR scale across an enterprise?

Manuals are a seemingly small example, but this points to the scale at work across a huge multinational company, as well as the real benefit that can come from implementing a new technology. By eliminating user manuals, you eliminate printing, translation, and creation costs. Across a company, even eliminating the incremental savings of making a maintenance manual could be huge. More than the money, though, the benefit comes from making things easier for the people in the field using the equipment.

“It could be a cost reduction,” Lewis says. “But I really think the driving factor for what we’re doing is trying to make it easy for our customers to use our equipment.”

That is the ultimate goal of all of this: to help people. But to get there, you first need to understand the tools for making this happen—how to build AR systems and the technologies that are most useful now. These tools are the focus of the next chapter.