Technology-Driven Changes

A series of technological capabilities are already used on a small scale within organizations, and we expect they will only expand in the near future. These extrapolations of existing practice include:

• Pervasive data. Arguably the biggest change in analytics over the past decade—and probably the next as well—is the availability of massive quantities of data. The internet and social media applications are already streaming massive quantities of it—more than 25 terabytes of data streamed by fans at the 2017 Super Bowl, for example. Internet of Things sensors (one estimate suggests 8.4 billion of them will be in use in 2017) in cars, factories, hospitals, and many other settings will provide much more data. At an individual level, smartphones, activity trackers, and other personal devices both generate and receive massive amounts of data.

  We will of course need analytics to make sense of all this data, and at the moment we are only scratching the surface of how pervasive data and analytics can change our work and our lives. Pervasive data is changing the technologies we use to analyze it and the locations for the analysis; more analytics are being performed at the edge. Pervasive data also implies a strong need for better tools—including the machine learning tools we described in chapter 8—for “curating” (cleaning, integrating, matching, and so forth) data. And data is also playing a more important role in creating and improving models (see the next trend); that’s really what machine learning is all about.

• More autonomous analytics and decision making, as opposed to relying on humans to look at data and make decisions. The resource that’s already most in short supply with analytics is the human attention to look at them, interpret them, and make decisions on the basis of them. Cognitive technologies, AI, machine learning, deep learning—all of these will increase the ability of smart machines to do automated analysis, make automated decisions, and take automated actions. Machine learning already helps many organizations to dramatically increase the productivity of human analysts by creating thousands of models in the time previously required for one. Quantitative analysts and data scientists’ jobs aren’t threatened yet, but they do need to learn how to work with these new tools. At the moment, machine-created models can be difficult to interpret, but in the future we may see machines that can not only find the best-fitting model for data, but also make sense of it for humans who want an explanation.
• The democratization of analytics software. The ability to analyze and report on data is already very common from a software standpoint. Vendors such as Microsoft embed analytical capabilities in business versions (particularly Office 365, the cloud version) of Microsoft Office, even including web analytics and personal productivity analytics. Many traditional applications systems, such as Salesforce.com for sales, marketing, service, and e-commerce applications, include various forms of analytics and even artificial intelligence capabilities. Smaller companies that can’t afford expensive analytics software packages have available free or inexpensive open-source tools such as R and RapidMiner. And plenty of big, rich companies are using these free tools as well. At one point, the most advanced analytical capabilities were expensive, but the pace of development for open-source software is such that they are now more likely to be free. Of course, lower costs for software are sometimes canceled out by higher costs of people capable of using it—the data scientists who are expert at open-source tools may be more expensive than traditional quantitative analysts with proprietary software skills.
• Increasing use of in-memory processing for analytics that can dramatically speed the response and calculation time for typical analyses. Instead of storing the data and algorithms on disk, they’re loaded into the computer’s memory. These are available from vendors like SAP (Hana), SAS, Tableau, Qlik, and several more. In the future, we may see even greater speed from in-chip analytics. We’re already also seeing edge analytics in which some analytics and decision making are performed by small, smart devices at the edge of a network.

• Increasing use of real-time (or at least “right-time”) analytics. It has historically taken some time—from days to weeks—for firms to extract data from transaction systems, load it into analytical applications, and make sense of it through analysis. Increasingly, however, managers need to make more rapid decisions, and firms are attempting to implement analytics in real time for at least some decisions. Of course, some real-time systems make use of autonomous decision making in order to take humans out of the loop altogether. The granddaddy of real-time applications with human users is UPS’s ORION, the project we’ve mentioned throughout this book, to provide routing to UPS drivers. Before they started to use this application, UPS drivers drove the same route every day. Today, they get a new route every morning that optimizes their deliveries and pickups based on the packages and requests that came in last night. Tomorrow (or at least within the next few years), their routes will change in real time based on such factors as traffic, weather, and new requests from customers for package pickups.

  Most organizations should adopt a right-time approach, in which the decision time frame for a class of decisions is determined within an organization, and the necessary data and analytical processes are put in place to deliver it by that time. In an InformationWeek survey, 59 percent of the IT executive respondents said they were trying to support real-time business information.¹ But research that Tom did with SAP suggests that many managers care about real-time information much more in some areas than others.² Companies shouldn’t waste their efforts on delivering real-time information when there isn’t a need.

• Going beyond alerts to preserve management attention. Alerts have been a useful strategy for organizations seeking to preserve management attention. They say, “Look at this number—you said you wanted to hear about it if it went this high!” More organizations are beginning to make use of automated alerts to notify managers when key indicators are at undesirable levels. Intel, for example, uses alerts to let supply chain commodity managers know when they should act on purchasing and pricing data.³ The concern with alerts, however, is that too many of them will lead to “alert fatigue” on the part of the alerted individuals. Of course, if a system can take an automated action, that prevents humans from needing to be in the loop at all.
• More explanatory analytics, as opposed to numbers and programming languages. This trend has been taking place for a while, simply because many managers prefer to see and digest analytics in visual formats. Of course, different people have different learning styles. Those who prefer verbal narratives can increasingly have a visual analytic converted into a story that summarizes the result. The new trend will be for software to deliver the best format for you as an individual, your data, and your decision or question. Let’s hope that all these developments mean the end of the pie chart, which visual analytics experts have noted for years is rarely a useful format.
• More prediction and prescription (and less reporting). It’s obviously more useful to predict what’s going to happen than to explain what’s already happened. Prediction, however, generally requires more sophisticated analysis and data than reporting or explanation. Prescriptive analytics require enough context about the task and the situation to make an informed recommendation. Despite these challenges, predictive and prescriptive analytics are extending into more and more business domains—from predicting the behavior of customers to telling them what to buy; from predicting disease to recommending treatment strategies. In a discussion with Salesforce.com users, for example, Tom heard many of them say that they wanted to move beyond descriptive analytics. One commented, “We don’t have time for people to look at bar charts and figure out what to do.” They prefer the idea (which Salesforce and other companies have begun to implement) of “smart data discovery” in which smart tools identify trends and anomalies in data—with no need for a human hypothesis—and point out their implications to users. Before long, managers may simply be able to converse with their automated assistants, who will be able to help interpret their financial reports, point out the weaknesses in their demand planning forecast, and predict that inventory is likely to run out two quarters from now. In addition to rapid data discovery, another benefit of this approach is that it’s less subject to biased human interpretation than traditional descriptive and predictive analytics. If a machine is finding patterns in data, it’s somewhat more difficult to “torture the data until it confesses.”
• More mining of text, speech, images, and other less structured forms of data. The mining or detailed analysis of structured data is by now quite advanced, but the mining of text, speech, images, and even video are clearly in their early stages and are likely to expand considerably over the next few years. Technological capabilities to categorize and discern meaning in the lab are already better than humans in many cases, but they have yet to penetrate many business applications. Consumer applications, such as Apple’s Siri and Amazon’s Echo/Alexa, are further along, and businesses are beginning to employ them within products and applications. Deep learning algorithms based on neural networks are able to learn how to categorize and make decisions on unstructured data—at least when given enough data to learn from. The availability of labeled training data—for example, the 14 million ImageNet images, or the 8 million labeled YouTube videos, will dramatically improve the performance of these algorithms over the next several years.

• Model management finally comes of age. The other major advance lies at the opposite end of the analytical process and involves the capture of models, learning, and insights from analytical and experimental results. In cultures with a broad-scale analytical or experimental orientation, there are likely to be many models created by many people, each with its own assumptions, variables, and results. Experiments have designs, test and control groups, and results. How do you keep track of such models and experiments without a repository? The answer, of course, is that you can’t. Capital One, one of the earliest firms to embrace experimental design for business, had a repository of findings from its many experiments, but it was extremely unwieldy for users to search through and learn from. What the company decided to do, then, was to take a just-in-time approach to providing experimental knowledge to its analysts. The company built a system to guide the analyst through the process of designing a new credit card offering for a specified class of customers. It uses knowledge from the company’s experiments to make suggestions at each stage of the design process about what options might work best. It might suggest everything from the optimal interest rate for balance transfers to the best color for the envelope to be used in the direct mail offer to the customer.

  Such systems for maintaining information about models are called model management systems, and they are currently widely used only in financial institutions (Capital One was an early adopter of these too). They are used in that industry primarily because regulators insist on them. However, as analytics become an enterprise resource—the source of competitive advantage and considerable value—we expect to see more model management tools employed, even when they aren’t imposed by regulators. They will provide not only backup when a quantitative analyst leaves the firm, but also can prevent the need for new analyses when a similar one has already been performed elsewhere in an organization.

Human-Driven Changes

While humans don’t change as rapidly as information technologies, there are changes in analytical competition that will be driven by the capabilities and configurations of analytical people within organizations. We expect, first of all, that the growth of analytical competition will lead to a need for substantially greater numbers of analytically oriented people—some number of analytical professionals and data scientists, and a much larger group of analytical amateurs, as we have called them. If many more decisions are to be made based on detailed analysis, many more people will have to have some understanding of how those analyses were performed and when they should be overridden. In short, analytics, and the use of them in decisions and actions, will be increasingly extended to the frontline analytical amateurs within organizations.

From where will these professional and frontline analysts come? Some of these, we believe, will come from business schools and other parts of universities, which have always offered some courses in statistics and data analysis. Over the last five years, literally hundreds of universities have added degree programs, certificates, and courses in analytics and data science. We expect that perceptive schools and their students will focus even more on analytical training in the future. The most advanced data scientists, who come from more diverse backgrounds like computer science and physics PhD programs, will continue to be sourced from relatively nonconventional academic sources. As yet, there are very few PhD programs in data science.

Corporations may also need to offer internal programs to educate their people on various forms of analytics and data science. Cisco Systems, for example, created a distance education program on data science for interested and qualified employees, partnering with two universities. The program lasts nine months and concludes with a certificate in data science from the university. More than two hundred data scientists have been trained and certified, and are now based in a variety of different functions and business units at Cisco. Cisco also created a two-day executive program led by business school professors on what analytics and data science are and how they are typically applied to business problems. The program also covers how to manage a workforce that includes data scientists, and how to know whether their products are any good.

Of course, not all workers will need to be involved in all analytical activities. It is likely that analytical professionals will have to become expert not only in quantitative analysis but also in job and process design for frontline analysts. They may also have to design information environments for frontline workers that provide just enough analytics and information for them to perform their jobs effectively. As one analyst (who refers to the concept as “pervasive business intelligence”) put it, “It’s the ability to take relevant information that is usually reported up to management and push it down to users. At the various organizational levels, the data is presented so that people see only what is most relevant to their day-to-day tasks . . . with expectations and performance clearly identified.”⁴

We also expect substantially increased use of outsourced and offshore analytical resources. Some of that has already emerged in the form of “math factories” like Mu Sigma in India. We noted in chapter 7 that it’s difficult to establish the close, trusting relationships between analysts and executives that are necessary for widespread analytical decision making. However, there are certainly analytical tasks that can be accomplished without close interactions with executives. Back-office development and refinement of algorithms, cleaning and integration of data, and the design of small-scale experiments can often be done remotely. Substantial numbers of analytically trained workers in India, Russia, and China will undoubtedly be doing more analytical work in the future. Offshore companies in the outsourcing business are beginning to specialize in such services. However, if an analytical task can be outsourced, there is a growing likelihood that it can be automated with tools like machine learning. This may reduce the growth of analytics outsourcing over time.

We also anticipate that increasing numbers of firms will develop strong analytical capabilities within their IT organizations. We’ve already described Gartner surveys from 2006 to 2016 finding that business intelligence or analytics was the number-one technology priority of corporations.⁵ As we have also pointed out in previous chapters, “better management decision making” is the number-one objective of large companies that have installed enterprise resource planning systems. With these priorities and objectives, it’s only natural that CIOs and other IT executives will want to increase the IT function’s ability to support analytics. This means that they’ll be hiring quantitative analysts, specialists in analytics software, and IT professionals with expertise in data warehouses and marts. Some, like Procter & Gamble, will make this capability a primary focus of the in-house IT function, while outsourcing less critical capabilities to external suppliers. The business functions that IT supports—logistics and supply chain management, marketing, and even HR—will also be hiring analytical experts with a strong IT orientation. It will become increasingly difficult to distinguish analytical people in IT from those in the rest of the business. If you’re trying to decide on a technical specialization, analytics are a good bet.

With the rise of analytical people across the entire organization, we can expect a greater need for structure and guidance in the management of their activities—either provided by humans or by the technology itself. As we’ve noted earlier, there will be no shortage of analytical tools, whether they’re spreadsheets, visual analytical systems, machine learning, or some other form of software. However, if corporate strategies depend on the results of analytics, they have to be done with accuracy and professionalism.

How will companies provide greater structure and human capability for strategically important analyses? There will be no single method but rather a variety of tools and approaches. One way is to have the software guide the analysis process—letting a human analyst know what assumptions about the data are being made, what statistical analysis approach to employ, or what visual display to best summarize the data. Analytical software applications can guide an analyst through a decision process, either making the decision itself (perhaps with a human override option) or ensuring that a human decision maker has all needed information. Another answer would be substantial education. We believe that most organizations would benefit from education to create more capable analysts and improve the skills of existing ones. A third would be a group of “coaches” to help amateur analysts and certify their work as well done. For analytical work that substantially affects financial performance, internal and external auditors may need to get involved. There is no doubt that audits are becoming increasingly analytical.⁶ Whatever the means, companies will need to both build analytical capability in their employees and ensure that they’re doing a good job. As we’ve already described, some leading companies are beginning to employ these approaches for building human analytical capability through some sort of centralized analytics hub.

This attention to human capabilities won’t stop at the analysis stage. Many organizations will begin to automate decisions and actions, which will have a considerable impact on the humans that previously performed those tasks. Tom and coauthor Julia Kirby have already written a book on how humans can add value to smart machines, so we won’t go into detail on that topic here.⁷ But determining the relationships between humans and machines, and how human work and business processes need to be modified to take advantage of machine intelligence, is clearly going to be an important topic in the near and distant future.

Strategy-Driven Changes

We anticipate that a number of changes in the analytical environment will be driven by business strategies. As more firms become aware of the possibilities for analytical competition, they will push the boundaries of analytics in their products, services, and business models. Virtually every provider of data and information services, for example, will probably offer analytics to its customers as a value-added service. Data itself has become something of a commodity, and customers for data often can’t find the time or people to analyze it themselves. Software, once primarily focused on business transactions, increasingly includes analytics.

We also expect to see more analytics embedded in or augmenting products and services—describing, for example, the optimum way to make use of those offerings within the customer’s business processes. The golf club sensor we described in chapter 3 that tells you how well you are swinging your club is a good example. We are already seeing automobiles (or at least insurance companies) that tell you how safely you are driving, health care and fitness trackers that analyze how healthily you are eating and living, and industrial machinery that tells you how well you are using it. Even industrial companies like GE and Monsanto are now selling products or services that tell their customers how to use their offerings more effectively. Of course, we may tire of all this advice, but it is potentially very useful.

This trend will be only a part of a broader one involving supplying analytics to customers and suppliers. We’ve already mentioned some firms, such as Walmart, that furnish analytical information to their customers or channel partners. There are others we haven’t mentioned that are beginning to do this to some degree. Marriott shares analytical information with both channel partners—online and traditional travel agencies, for example—and major corporate customers. Channel partners get analyses involving pricing, joint promotions, and inventory; customers receive data and analysis that helps them with their own travel management. We expect that most firms will begin to view the internal audience for analytics as only one of several potential recipients, and that relationships with suppliers and customers will increasingly include the provision of analytics.

Another strategic trend involves the content of analytics. Thus far, most quantitative analyses are about internal business entities: inventory units, dollars, customers, and so forth. Most organizations realize, however, that internal information, no matter how well it’s analyzed, gives a limited window on the world. Peter Drucker commented in 1998 that management has a tendency to “focus inward on costs and efforts, rather than outward on opportunities, changes, and threats.”⁸ Drucker said that outside information didn’t exist then, but it does now. Data and analytics are increasingly available on what customers and non-customers are saying about our company, on trends and concerns in our industries, and on economic and sociopolitical movements that could affect our futures. Any firm that wishes to control—or at least react quickly to—outside forces must be applying analytics to this external information.

Thus far external information, when accessed at all, has not been put into a structure for easy, ongoing access. And it has been the subject of descriptive analytics at best—very little predictive or prescriptive analytics, at least outside of financial services. But such structured information systems—they might be called situational awareness systems, as they are in the military and intelligence communities—are beginning to be found in organizations. Several cities (e.g., Chicago’s WindyGrid) and police forces (the NYPD’s Domain Awareness System [DAS] is the best one we’ve seen) are using them. Deloitte has created one (actually, multiple tailored versions) for its senior executives, and it builds them for clients too. Recorded Future, a company Tom advises, scans and analyzes internet text to better understand what people are saying and doing around the world, particularly with regard to intelligence and cybersecurity. Many companies are using similar approaches to understand customer perceptions about products and brands.

Finally, we expect that strategic concerns will also drive firms to pay substantial attention to new metrics and their interrelationships in analyses and scorecards. We heard from a number of analytical competitors that they start with metrics in thinking about applying analysis to a distinctive capability. They either invent a new metric from their own proprietary data or refine an existing one. As metrics become commonplace (e.g., as we have discussed, the FICO score in consumer credit or the batting average in baseball), companies and organizations go beyond them to new measurement frontiers. We anticipate particularly high levels of activity in the domain of human resources and talent management, since these have been relatively unmeasured in the past. Once developed, of course, metrics must be incorporated into established scorecards and measurement processes, and the relationships between different measures must be explored and understood. Most importantly, these metrics must be incorporated into business and management decision-making processes. Just developing a measure, and just using it in some analyses, is never enough.

The Future of Analytical Competition

We’ll end this book by discussing broadly what will happen to analytical competitors in the future. This will serve as both a summary of the key attributes of analytical competitors and a prediction of the future, because analytical competitors will continue to do more of what made them successful in the first place.

Analytical competitors will continue to examine their strategies and their business capabilities to understand where they can get an analytical edge. That’s more and more important as more companies jump into analytics, at least at a surface level. But the best companies will focus on what makes their organizations distinctive and how analytics can support or drive a distinctive capability. After they address the most distinctive areas, analytics will eventually be applied to most other parts of their businesses—their motto will be, “If it’s worth doing, it’s worth doing analytically.” These companies will identify measures of the distinctive capability that other organizations don’t yet employ. After they identify a measure, they’ll collect data on it and embed decisions based on the measures into their daily work processes.

Take Google, for example. The company is perhaps the most analytical firm on the planet today. But the presence of other analytical companies in its markets hasn’t made it retreat at all. Instead, it’s doubled down on such capabilities as artificial intelligence software, proprietary mapping data, analyzing the data from its autonomous vehicles, analyzing YouTube videos, and so forth. It started with its PageRank algorithm and then advertising algorithms, but has since moved on to being the leader in analytics for human resources, attribution of digital ads, venture capital, and many others. And not surprisingly, the company continues to perform extremely well.

In order to continue refining their analytical capabilities, companies will focus on both their human and technological dimensions. On the human side, they’ll try to further embed an analytical orientation into the culture and to test as many hypotheses as they can. A 2017 survey of executives in fifty large companies by NewVantage Partners on big data suggests that while companies have found big data efforts successful and financially rewarding, the creation of a data-driven culture has been problematic.⁹ Of those who responded, 86 percent said their companies have tried to create a data-driven culture, but only 37 percent said they’ve been successful at it.

The best analytical competitors will keep trying to achieve that type of culture, however. Their executives will argue for analytical strategies and decisions with passion and personal example. Their managers will constantly press subordinates for data or analytics before they take major actions. Employees at every level will use data and analytics to make decisions and take actions. And data and analytics will be employed to seek out truth, not to advance some executive’s private objectives.

The managers of analytical competitors of the future will not be narrow “quant jocks.” They’ll always be thinking broadly about whether their analytical models and data are still relevant to their businesses. They’ll constantly be reexamining the assumptions behind their analytical models. If a particular type of analysis becomes commoditized throughout their industries, they’ll find some new basis for analytical competition. They’ll use intuition sparingly but strategically when it isn’t possible to test an assertion or gather data for an analysis. They’ll be able to be more experimental and innovative. They’ll advocate for new methods and new technologies like machine learning and cognitive technologies. They’ll be looking for how they can employ these intelligent machines for new business strategies and models, and how to extract more productivity from every activity.

As a result of their efforts, they’ll undoubtedly be hotly pursued by other firms that also want to be analytical competitors. If their employers are smart, these analytical heat-seeking missiles will find their jobs stimulating and satisfying and will stay put as long as they’re recognized and promoted.

There will continue to be people in these organizations whose job primarily involves developing and refining analytics—analytical professionals and data scientists. They will either work in a central group or be highly networked, and they’ll share approaches and ideas. They will also work to educate and partner with the analytical amateurs of the organizations, who need to understand how analytical models and tools support them in their jobs. The analytical competitor of the future will also supplement internal analytical resources with outsourced or offshore expertise. And these firms won’t be shy about thinking of ways that machines themselves can do the difficult, time-consuming work of analytics. They’ll focus particularly on how to automate the really labor-intensive part of analytics: preparing the data for analysis.

Analytical competitors will continue to have lots of data that is generated from enterprise systems, point-of-sale systems, and web transactions, as well as external data of various types from customers and suppliers. They’ll organize it and put it aside for analysis in warehouses and Hadoop-based data lakes. They will ensure that data is integrated and common in the areas of their business where it really matters. They’ll have integrated analytics suites or platforms that support reporting and analytics—both proprietary and open source. In domains where decisions must be made very rapidly or very often, they’ll embed analysis into automated decision systems, allowing human overrides only under specified conditions.

Perhaps most importantly, analytical competitors will continue to find ways to outperform their competitors. They’ll get the best customers and charge them exactly the price that the customer is willing to pay for their product and service. They’ll have the most efficient and effective marketing campaigns and promotions. Their customer service will excel, and their customers will be loyal in return. Their supply chains will be ultra-efficient, and they’ll have neither excess inventory nor stock-outs. They will embed data and analytics into new innovative products. They’ll have the best people in the industry, and the employees will be evaluated and compensated based on their specific contributions. They’ll understand both their internal and their external business environments, and they’ll be able to predict, identify, and diagnose problems before they become too problematic. They will make a lot of money, win a lot of games, or help solve the world’s most pressing problems. They will continue to lead us into the future.