4
Design for Data: xAPI
YOU AND A COLLEAGUE make up a two-person team creating e-learning courses for a software company. These courses include tutorials that walk learners through a process in the software. Each tutorial consists of a recorded demonstration, a simulation to click through, and a PDF job aid explaining the process. Unfortunately, you and your teammate are working through a large backlog of courses. Streamlining the development process would be beneficial and result in significant time savings.
You plan to deploy these courses using basic Google Analytics to measure basic usage. You also want to see which pages within the tutorials are being accessed. But you cannot see exactly what is taking place on each screen as the learner navigates through the courses. What are they watching or using to learn the process? Are they using the activities or resources you developed? What if you could gather more data on how learners are interacting with your content? Could you use that to make data-driven decisions about the development of the content?
You decide to use the Experience API (xAPI) to measure each button that launches elements of the course. Now you can get data for each button to visualize and review. After collecting data for a period of time, you realize that the simulation is almost never used. Is it a fluke? After collecting more data, you confirm the simulation is not used because individuals can perform the task live instead. As a result, you decide to eliminate the simulation in future courses because creating it is a tedious and time-consuming task. This saves you and your colleague 800 development hours per year. Implementing xAPI enabled you to see exactly what was taking place in your course and how it was being used.
You can change the design of future courses by making data-driven decisions.
The scenario above is an actual use case from the early days of xAPI. Collecting more granular data improved a team’s development process and provided a proof of concept that led to more in-depth xAPI projects.
Are you ready to learn more about how xAPI works? Let’s dive in with the basics.
xAPI: A Primer
Let’s start by answering two frequently asked questions: What does xAPI do that other systems can’t, and why is it important?
One thing that makes xAPI stand out is that it can be used to gather more granular data than we can with previous e-learning standards like SCORM. SCORM limited us to reporting on data points such as course completions, scores, and time spent in the course; xAPI enables us to gather data on what happens within a course alongside content accessed outside a course. It also provides us with the potential to measure real-world, on-the-job data from other business systems in a common format. We measure these activities using a mechanism called a statement.
A statement is a way to break down an activity into a simple, human- and machine-readable format. They consist of three parts: actor, verb, and object. You can also include other elements, but actor, verb, and object are the requirements of a basic statement. The following statement describes an activity:
“Sean Putman attempted the xAPI Statements course.”
Sean Putman is the actor, attempted is the verb, and xAPI Statements course is the object.
Statements pass data using the JavaScript Object Notation (JSON) data format (Figure 4-1). You’ll also need a learning record store (LRS)—a platform used to collect and store statements generated by xAPI-enabled content—to receive and store this data. Storing data in the JSON format allows for interoperability of collected data across many platforms.
Figure 4-1. Activity Statement in JSON Data Format
You may be wondering why this is important, so let’s consider a typical interoperability challenge in our field. Have you ever been part of a project to switch from one learning management system (LMS) to another? If so, you know how challenging it is when the content and records in the old system cannot be moved to the new one. Many organizations are forced to manually transfer the data from one platform to the other. This is what makes xAPI special: xAPI data collected in a conformant LRS can move to any other conformant LRS with no issue.
What Can You Do With xAPI?
Collecting data from activities as a series of statements allows you to analyze the behavior of learners in a more specific and strategic way. Instead of just seeing that someone attempted and completed a course, you can see what they actually did while taking the course. As learners interact with tracked activities over time, you can gain insight into the content, and use that insight to improve it if necessary. The data can inform decisions affecting design, outcomes, and, ultimately, the business that are made around a course. Let’s dig deeper into these three categories.
Design Decisions
How many times have you finished an e-learning project and wondered if the design was effective for your target audience? We often spend a lot of time designing the interface and interactions. But how can we be sure they are effective or even being used at all? Getting more detailed data from xAPI can help us understand usage patterns and start conversations that help inform design decisions.
However, it’s important to remember that xAPI will not solve all the problems. What it will do is inform you of where to start asking questions. Data is not always the final outcome. Sometimes, it is the beginning.
One of the great things about xAPI is that it can be implemented into early prototypes so you can start collecting data right away. You choose how to set up your statements and to what specificity for data collection. This can be intimidating, because you can collect almost anything—the depth of data collection depends on your specific project needs.
Identifying this depth of data collection must be a design decision you make early in the process. You’ll need to figure out how much information is “good enough” to inform you, but not so much that you have to wade through stacks of data to see anything meaningful.
The process we use to determine which elements to track is called data strategy. In the data strategy process, you should ask questions such as:
• What are the outcomes we are trying to get from this content?
• What would we need to track to get information on the outcomes?
• How will tracking these items give us actionable information?
• What action are we going to take when we have this information?
A myriad of other questions will surface during your data strategy session. Their answers are critical to inform your design so it aligns with the desired outcomes, which we’ll discuss in the next section.
Now that you have an informed design, you can build xAPI into your prototype or interactive storyboard to pilot with a test group of learners. These early tests provide usage patterns and indicators of overall design quality. You’ll be able to see if proposed interactions are used by the test group, and start asking more pointed questions about navigation and interaction. Remember, xAPI may not necessarily provide all the answers at these early stages, but it will help inform decisions. Even if you never implement xAPI into the final project, using it during the design phase often results in a better end product.
Performance Outcomes
Once the design phase is complete and the content is deployed, we need to answer the age-old question, “Did the training do what we expected it to do?”
Will xAPI solve the problem of correlating training completion to better job performance? There are many projects in which xAPI, on its own, shows a strong correlation to training effectiveness and producing desirable outcomes. However, it is not as simple as adding xAPI to a course, collecting data, and then using the data to show that performance improved.
For example, if we have a compliance course, we can use xAPI to track how people navigated the course, what items they interacted with, and how they performed on each assessment question. By gathering and evaluating this data alone, are we showing improved performance? I think we can definitively say it will not. We often need other business data to prove a correlation. Performance outcomes are just half the equation.
Business Decisions
The desired outcomes the content produces should be based on the business outcomes desired. For example, if you are producing material for sales representatives to help them with a new product or market, the desired outcome is likely improved sales. Data can help us understand how and where the content is being used. Are the salespeople accessing the content on a mobile device when getting ready to meet a client? If yes, device data is something we could not get from SCORM. We can also see what is being accessed at a granular level so we can ask relevant questions that make future content better and more suited for the audience. If the content is aligned with desired outcomes, hopefully the improved business results will be tied to the training.
Another aspect of the business that can be improved from the use of this data methodology is the learning ecosystem. In the past we’ve often been bound by a monolithic LMS, which handles our content storage, people management, delivery, and data collection and visualization, among other things. By using xAPI, we are no longer tied to one large system. We can use specific systems that meet our organization’s requirements for each. Does your IT department have a better solution for people management? Use it. Do you need to maximize content delivery to mobile devices, but your LMS does not provide that function? Find a better system for that. Are you unable to get a visualization that you need from the data? Pull the data from the LRS and use a better visualization tool to build your reports. xAPI gives you the power to build the best system, not just a system that’s “good enough.” Further, if one piece no longer meets your criteria, you can replace it with something that does. You are not breaking the whole system, just updating one part of it.
xAPI Case Examples
Let’s take a look at some examples of xAPI in action. We have curated these to show a variety of content types, using xAPI to track activity in everything from videos to beacons. As you review, consider how you might use xAPI in your own projects.
Training and Self-Assessment Tool
This training and self-assessment tool was developed for public health agencies to determine current levels of capability (Figure 4-2). Individuals using the tool have access to support for building capacity in low-ranking areas. This project was developed in Adobe Captivate using JavaScript with advanced actions. Then, the published files were modified to use cmi5 to send xAPI statements to the LRS (ADL 2016).
Figure 4-2. Critical Success Factors for Informatics Projects
Using xAPI, the Public Health Informatics Institute is able to determine which public health departments need support in each of the critical areas for informatics capacity. This allows for improvements to the tool, resources provided, and data needed to support requests for continued funding (Gilbert 2017).
Software Help Videos
In this example, xAPI is being used to track whether end users are using the videos in an organization’s help system. xAPI helps identify what search terms are being used and provides a detailed view of what a user does during a session.
Interactive E-Book
This project is an xAPI-enabled interactive e-book (Figure 4-3). Using the Sigil EPUB Editor, Rustici JavaScript library, and JavaScript, the program generates xAPI statements as an individual interacts with the e-book’s elements. Activities tracked with xAPI include each time the book is launched, specific page views, what links are clicked, and video plays (Learning Ninjas).
Figure 4-3. xAPI Records for Interactive E-Books
Supervised Simulation
MedStar uses a blended approach to teach clinicians how to respond to a code blue. Supervisors observe the learners as they go through a mock code blue simulation with a mannequin, and take notes in an xAPI-enabled application. The application data can then be matched with the learning data to help establish a more complete picture of demonstrating proficiency.
Internet of Things (IoT)–Enabled Devices
IoT-enabled devices are another source of xAPI data. There are many examples of prototypes and real solutions that show the possibilities for xAPI data coming from IoT devices. One example from TorranceLearning leverages beacon technology.
TorranceLearning worked with a children’s science discovery museum to implement a beacon system that tracks how visitors experience their interactive exhibits (Figure 4-4). For example, when students visit the museum during field trips, the beacons capture data and then send real-time xAPI statements to the LRS. Both teachers and the museum can then access that data about visits, interactions, curriculum standards, and data entered by the students.
Figure 4-4. The Beacon Device Used to Send xAPI Statements to the LRS
Sales Comparison Initiative
In this example, HT2 Labs, a Learning Pool Company, worked with Villeroy & Boch to evaluate the effectiveness of a training program using A/B testing (Figure 4-5). They were able to demonstrate that group B (those who experienced a blended training approach) outperformed group A (those who received traditional e-learning only). xAPI allowed the organization to evaluate learning data with performance data to compare the sales outcomes, and the results were staggering. The organization saw a $2.5 million improvement in sales for group B, validating that the blended approach helped them realize an increase in sales.
Figure 4-5. xAPI in A/B Testing
What’s Next for xAPI?
Real-world examples like the ones in the previous section are helping the broader industry see the value of xAPI and what it can offer. As more projects kick off and more companies start evaluating and prototyping xAPI-enabled solutions, the shortcomings of the existing tool sets become more and more apparent. We have seen some tools add functionality to make a more robust xAPI offering, while others continue to lag.
As more projects are developed, we believe it is only a matter of time before rapid development tool vendors improve their products to better support xAPI. With the advent of learning experience platforms (LXPs), which seem to provide the means to take the place of the LMS, tool vendors will have to keep up with the changing ecosystem.
Typically in the LXP, xAPI is used to track the many facets of learning that are taking place, including social, mobile, traditional e-learning, and blended learning. Tracking all these different modalities is impossible with SCORM. So, if the tools only produce SCORM and SCORM-type xAPI (such as attempts, slide views, completions, and test scores), users won’t be able to see the real value of an LXP. We need a more robust output from the tools we use to generate content. There are some on the market now that have improved implementations or can be customized with JavaScript to generate more granular statements. However, customization is not a viable long-term option for most users. It is too time-consuming and full of places to make mistakes. As a community, we need to push our vendors to develop robust xAPI implementations.
Conclusion
We are finally getting past the “trough of disillusionment” and moving into the “slope of enlightenment” in the xAPI hype cycle, mostly due to more widespread demonstration of real-world project results. Many LRS vendors are releasing case studies showing how large corporations are effectively using xAPI in complex integrations with other systems. These implementations are showing the effectiveness of the learning ecosystem, as well as how that is translating to real business results. The numerous case studies that have been released in the last 12–18 months have helped answer one of our industry’s biggest concerns: “xAPI is great in theory, but how are people using it?”
What is your next step? There are a few popular xAPI learning opportunities available—including xAPI cohort, xAPI MOOC, Learn xAPI, and xAPI.com—which are excellent for learning how to work with xAPI. However, do your research because these projects can have a narrow focus or may still be in the prototype phase, and may not necessarily provide business value right away.
With that said, these are the pieces of a puzzle that is now helping the L&D community more readily access and understand xAPI. For example, work in the xAPI Cohort resulted in a tool that would help create better statements from a rapid development tool. There have also been projects that enable business systems like Slack to write xAPI statements, and the code to generate these statements is publicly available.
Thanks to this support and by learning from published case studies, more practitioners are connecting the dots, allowing them to understand the value of xAPI and how it could be useful in their projects. Many practitioners have transitioned from asking, “What is xAPI and how does it work?” to “I understand what xAPI is, I have some ideas, now how do I start implementing it?”