Chapter 2. Business Motivations and Drivers for Big Data Adoption
Information and Communications Technology
In many organizations it is now acceptable for a business to be architected in much the same way as its technology. This shift in perspective is reflected in the expanding domain of enterprise architecture, which used to be closely aligned with technology architecture but now includes business architecture as well. Although businesses still view themselves from a mechanistic system’s point of view, with command and control being passed from executives to managers to front-line employees, feedback loops based upon linked and aligned measurements are providing greater insight into the effectiveness of management decision-making.
This cycle from decision to action to measurement and assessment of results creates opportunities for businesses to optimize their operations continuously. In fact, the mechanistic management view is being supplanted by one that is more organic and that drives the business based upon its ability to convert data into knowledge and insight. One problem with this perspective is that, traditionally, businesses were driven almost exclusively by internal data held in their information systems. However, companies are learning that this is not sufficient in order to execute their business models in a marketplace that more resembles an ecological system. As such, organizations need to consume data from the outside to sense directly the factors that influence their profitability. The use of such external data most often results in “Big Data” datasets.
This chapter explores the business motivations and drivers behind the adoption of Big Data solutions and technologies. The adoption of Big Data represents the confluence of several forces to include: marketplace dynamics, an appreciation and formalism of Business Architecture (BA), the realization that a business’ ability to deliver value is directly tied to Business Process Management (BPM), innovation in Information and Communications Technology (ICT) and finally the Internet of Everything (IoE). Each of these topics will be explored in turn.
Marketplace Dynamics
There has been a fundamental shift in the way businesses view themselves and the marketplace. In the past 15 years, two large stock market corrections have taken place—the first was the dot-com bubble burst in 2000, and the second was the global recession that began in 2008. In each case, businesses entrenched and worked to improve their efficiency and effectiveness to stabilize their profitability by reducing costs. This of course is normal. When customers are scarce, cost-cutting often ensues to maintain the corporate bottom line. In this environment, companies conduct transformation projects to improve their corporate processes to achieve savings.
As the global economies began to emerge from recession, companies began to focus outward, looking to find new customers and keep existing customers from defecting to marketplace competitors. This was accomplished by offering new products and services and delivering increased value propositions to customers. It is a very different market cycle to the one that focuses on cost-cutting, for it is not about transformation but instead innovation. Innovation brings hope to a company that it will find new ways to achieve a competitive advantage in the marketplace and a consequent increase in top line revenue.
The global economy can experience periods of uncertainty due to various factors. We generally accept that the economies of the major developed countries in the world are now inextricably intertwined; in other words, they form a system of systems. Likewise, the world’s businesses are shifting their perspective about their identity and independence as they recognize that they are also intertwined in intricate product and service networks.
For this reason, companies need to expand their Business Intelligence activities beyond retrospective reflection on internal information extracted from their corporate information systems. They need to open themselves to external data sources as a means of sensing the marketplace and their position within it. Recognizing that external data brings additional context to their internal data allows a corporation to move up the analytic value chain from hindsight to insight with greater ease. With appropriate tooling, which often supports sophisticated simulation capabilities, a company can develop analytic results that provide foresight. In this case, the tooling assists in bridging the gap between knowledge and wisdom as well as provides advisory analytic results. This is the power of Big Data—enriching corporate perspective beyond introspection, from which a business can only infer information about marketplace sentiment, to sensing the marketplace itself.
The transition from hindsight to foresight can be understood through the lens of the DIKW pyramid depicted in Figure 2.1. Note that in this figure, at the top of the triangle, wisdom is shown as an outline to indicate that it exists but is not typically generated via ICT systems. Instead, knowledge workers provide the insight and experience to frame the available knowledge so that it can be integrated to form wisdom. Wisdom generation by technological means quickly devolves into a philosophical discussion that is not within the scope of this book. Within business environments, technology is used to support knowledge management, and personnel are responsible for applying their competency and wisdom to act accordingly.
Figure 2.1 The DIKW pyramid shows how data can be enriched with context to create information, information can be supplied with meaning to create knowledge and knowledge can be integrated to form wisdom.
Business Architecture
Within the past decade, there has been a realization that too often a corporation’s enterprise architecture is simply a myopic view of its technology architecture. In an effort to wrest power from the stronghold of IT, business architecture has emerged as a complementary discipline. In the future, the goal is that enterprise architecture will present a balanced view between business and technology architectures. Business architecture provides a means of blueprinting or concretely expressing the design of the business. A business architecture helps an organization align its strategic vision with its underlying execution, whether they be technical resources or human capital. Thus, a business architecture includes linkages from abstract concepts like business mission, vision, strategy and goals to more concrete ones like business services, organizational structure, key performance indicators and application services.
These linkages are important because they provide guidance as to how to align the business and its information technology. It is an accepted view that a business operates as a layered system—the top layer is the strategic layer occupied by C-level executives and advisory groups; the middle layer is the tactical or managerial layer that seeks to steer the organization in alignment with the strategy; and the bottom layer is the operations layer where a business executes its core processes and delivers value to its customers. These three layers often exhibit a degree of independence from one another, but each layer’s goals and objectives are influenced by and often defined by the layer above, in other words top-down. From a monitoring perspective, communication flows upstream, or bottom-up via the collection of metrics. Business activity monitoring at the operations layer generates Performance Indicators (PIs) and metrics, for both services and processes. They are aggregated to create Key Performance Indicators (KPIs) used at the tactical layer. These KPIs can be aligned with Critical Success Factors (CSFs) at the strategic layer, which in turn help measure progress being made toward the achievement of strategic goals and objectives.
Big Data has ties to business architecture at each of the organizational layers, as depicted in Figure 2.2. Big Data enhances value as it provides additional context through the integration of external perspectives to help convert data into information and provide meaning to generate knowledge from information. For instance, at the operational level, metrics are generated that simply report on what is happening in the business. In essence, we are converting data through business concepts and context to generate information. At the managerial level, this information can be examined through the lens of corporate performance to answer questions regarding how the business is performing. In other words, give meaning to the information. This information may be further enriched to answer questions regarding why the business is performing at the level it is. When armed with this knowledge, the strategic layer can provide further insight to help answer questions of which strategy needs to change or be adopted in order to correct or enhance the performance.
Figure 2.2 The DIKW pyramid illustrates alignment with Strategic, Tactical and Operational corporate levels.
As with any layered system, the layers do not all change at the same speed. In the case of a business enterprise, the strategic layer is the slowest moving layer, and the operational layer is the fastest moving layer. The slower moving layers provide stability and direction to the faster moving layers. In traditional organizational hierarchies, the management layer is responsible for directing the operational layer in alignment with the strategy created by the executive team. Because of this variation in regard to speed of change, it is possible to envision the three layers as being responsible for strategy execution, business execution and process execution respectively. Each of these layers relies upon different metrics and measures, presented through different visualization and reporting functions. For example, the strategy layer may rely upon balanced scorecards, the management layer upon an interactive visualization of KPIs and corporate performance and the operational layer on visualizations of executing business processes and their statuses.
Figure 2.3, a variant of a diagram produced by Joe Gollner in his blog post “The Anatomy of Knowledge,” shows how an organization can relate and align its organizational layers by creating a virtuous cycle via a feedback loop. On the right side of the figure, the strategic layer drives response via the application of judgment by making decisions regarding corporate strategy, policy, goals and objectives that are communicated as constraints to the tactical layer. The tactical layer in turn leverages this knowledge to generate priorities and actions that conform to corporate direction. These actions adjust the execution of business at the operational layer. This in turn should generate measureable change in the experience of internal stakeholders and external customers as they deliver and consume business services. This change, or result, should surface and be visible in the data in the form of changed PIs that are then aggregated into KPIs. Recall that KPIs are metrics that can be associated with critical success factors that inform the executive team as to whether or not their strategies are working. Over time, the strategic and management layers injection of judgment and action into the loop will serve to refine the delivery of business services.
Figure 2.3 The creation of a virtuous cycle to align an organization across layers via a feedback loop.
Business Process Management
Businesses deliver value to customers and other stakeholders via the execution of their business processes. A business process is a description of how work is performed in an organization. It describes all work-related activities and their relationships, aligned with the organizational actors and resources responsible for conducting them. The relationships between activities may be temporal; for example, activity A is executed before activity B. The relationships can also describe whether the execution of activities is conditional, based upon the outputs or conditions generated by other activities or by sensing events generated outside of the business process itself.
Business process management applies process excellence techniques to improve corporate execution. Business Process Management Systems (BPMS) provide software developers a model driven platform that is becoming the Business Application Development Environment (BADE) of choice. A business application needs to: mediate between humans and other technology-hosted resources, execute in alignment with corporate policies and ensure the fair distribution of work to employees. As a BADE, models of a business process are joined with: models of organizational roles and structure, business entities and their relationships, business rules and the user-interface. The development environment integrates these models together to create a business application that manages screenflow and workflow and provides workload management. This is accomplished in an execution environment that enforces corporate policy and security and provides state management for long-running business processes. The state of an individual process, or all processes, can be interrogated via Business Activity Monitoring (BAM) and visualized.
When BPM is combined with BPMSs that are intelligent, processes can be executed in a goal-driven manner. Goals are connected to process fragments that are dynamically chosen and assembled at run-time in alignment with the evaluation of the goals. When the combination of Big Data analytic results and goal-driven behavior are used together, process execution can become adaptive to the marketplace and responsive to environmental conditions. As a simple example, a customer contact process has process fragments that enable communication with customers via a voice call, email, text message and traditional postal mail. In the beginning, the choice of these contact methods is unweighted, and they are chosen at random. However, behind-the-scenes analysis is being done to measure the effectiveness of the contact method via statistical analysis of customer responsiveness.
The results of this analysis are tied to a goal responsible for selecting the contact method, and when a clear preference is determined, the weighting is changed to favor the contact method that achieves the best response. A more detailed analysis could leverage customer clustering, which would assign individual customers to groups where one of the cluster dimensions is the contact method. In this case, customers can be contacted with even greater refinement, which provides a pathway to one-to-one targeted marketing.
Information and Communications Technology
This section examines the following ICT developments that have accelerated the pace of Big Data adoption in businesses:
• data analytics and data science
• affordable technology and commodity hardware
• hyper-connected communities and devices
Data Analytics and Data Science
Enterprises are collecting, procuring, storing, curating and processing increasing quantities of data. This is occurring in an effort to find new insights that can drive more efficient and effective operations, provide management the ability to steer the business proactively and allow the C-suite to better formulate and assess their strategic initiatives. Ultimately, enterprises are looking for new ways to gain a competitive edge. Thus the need for techniques and technologies that can extract meaningful information and insights has increased. Computational approaches, statistical techniques and data warehousing have advanced to the point where they have merged, each bringing their specific techniques and tools that allow the performance of Big Data analysis. The maturity of these fields of practice inspired and enabled much of the core functionality expected from contemporary Big Data solutions, environments and platforms.
Digitization
For many businesses, digital mediums have replaced physical mediums as the de facto communications and delivery mechanism. The use of digital artifacts saves both time and cost as distribution is supported by the vast pre-existing infrastructure of the Internet. As consumers connect to a business through their interaction with these digital substitutes, it leads to an opportunity to collect further “secondary” data; for example, requesting a customer to provide feedback, complete a survey, or simply providing a hook to display a relevant advertisement and tracking its click-through rate. Collecting secondary data can be important for businesses because mining this data can allow for customized marketing, automated recommendations and the development of optimized product features. Figure 2.4 provides a visual representation of examples of digitization.
Figure 2.4 Examples of digitization include online banking, on-demand television and streaming video.
Affordable Technology and Commodity Hardware
Technology capable of storing and processing large quantities of diverse data has become increasingly affordable. Additionally, Big Data solutions often leverage open-source software that executes on commodity hardware, further reducing costs. The combination of commodity hardware and open source software has virtually eliminated the advantage that large enterprises used to hold by being able to outspend their smaller competitors due to the larger size of their IT budgets. Technology no longer delivers competitive advantage. Instead, it simply becomes the platform upon which the business executes. From a business standpoint, utilization of affordable technology and commodity hardware to generate analytic results that can further optimize the execution of its business processes is the path to competitive advantage.
The use of commodity hardware makes the adoption of Big Data solutions accessible to businesses without large capital investments. Figure 2.5 provides an example of the price decline associated with data storage prices over the past 20 years.
Figure 2.5 Data storage prices have dropped dramatically from more than $10,000 to less than $0.10 per GB over the decades.
Social Media
The emergence of social media has empowered customers to provide feedback in near-realtime via open and public mediums. This shift has forced businesses to consider customer feedback on their service and product offerings in their strategic planning. As a result, businesses are storing increasing amounts of data on customer interactions within their customer relationship management systems (CRM) and from harvesting customer reviews, complaints and praise from social media sites. This information feeds Big Data analysis algorithms that surface the voice of the customer in an attempt to provide better levels of service, increase sales, enable targeted marketing and even create new products and services. Businesses have realized that branding activity is no longer completely managed by internal marketing activities. Instead, product brands and corporate reputation are co-created by the company and its customers. For this reason, businesses are increasingly interested in incorporating publicly available datasets from social media and other external data sources.
Hyper-Connected Communities and Devices
The broadening coverage of the Internet and the proliferation of cellular and Wi-Fi networks has enabled more people and their devices to be continuously active in virtual communities. Coupled with the proliferation of Internet connected sensors, the underpinnings of the Internet of Things (IoT), a vast collection of smart Internet-connected devices, is being formed. As shown in Figure 2.6, this in turn has resulted in a massive increase in the number of available data streams. While some streams are public, other streams are channeled directly to corporations for analysis. As an example, the performance-based management contracts associated with heavy equipment used in the mining industry incentivize the optimal performance of preventive and predictive maintenance in an effort to reduce the need and avoid the downtime associated with unplanned corrective maintenance. This requires detailed analysis of sensor readings emitted by the equipment for the early detection of issues that can be resolved via the proactive scheduling of maintenance activities.
Figure 2.6 Hyper-connected communities and devices include television, mobile computing, RFIDs, refrigerators, GPS devices, mobile devices and smart meters.
Cloud Computing
Cloud computing advancements have led to the creation of environments that are capable of providing highly scalable, on-demand IT resources that can be leased via pay-as-you-go models. Businesses have the opportunity to leverage the infrastructure, storage and processing capabilities provided by these environments in order to build-out scalable Big Data solutions that can carry out large-scale processing tasks. Although traditionally thought of as off-premise environments typically depicted with a cloud symbol, businesses are also leveraging cloud management software to create on premise clouds to more effectively utilize their existing infrastructure via virtualization. In either case, the ability of a cloud to dynamically scale based upon load allows for the creation of resilient analytic environments that maximize efficient utilization of ICT resources.
Figure 2.7 displays an example of how a cloud environment can be leveraged for its scaling capabilities to perform Big Data processing tasks. The fact that off-premise cloud-based IT resources can be leased dramatically reduces the required up-front investment of Big Data projects.
Figure 2.7 The cloud can be used to complete on-demand data analysis at the end of each month or enable the scaling out of systems with an increase in load.
It makes sense for enterprises already using cloud computing to reuse the cloud for their Big Data initiatives because:
• personnel already possesses the required cloud computing skills
• the input data already exists in the cloud
Migrating to the cloud is logical for enterprises planning to run analytics on datasets that are available via data markets, as many data markets make their datasets available in a cloud environment, such as Amazon S3.
In short, cloud computing can provide three essential ingredients required for a Big Data solution: external datasets, scalable processing capabilities and vast amounts of storage.
Internet of Everything (IoE)
The convergence of advancements in information and communications technology, marketplace dynamics, business architecture and business process management all contribute to the opportunity of what is now known as the Internet of Everything or IoE. The IoE combines the services provided by smart connected devices of the Internet of Things into meaningful business processes that possess the ability to provide unique and differentiating value propositions. It is a platform for innovation enabling the creation of new products and services and new sources of revenue for businesses. Big Data is the heart of the IoE. Hyper-connected communities and devices running on affordable technology and commodity hardware stream digitized data that is subject to analytic processes hosted in elastic cloud computing environments. The results of the analysis can provide insight as to how much value is generated by the current process and whether or not the process should proactively seek opportunities to further optimize itself.
IoE-specific companies can leverage Big Data to establish and optimize workflows and offer them to third parties as outsourced business processes. As established in the Business Process Manifesto edited by Roger Burlton (2011), an organization’s business processes are the source for generating outcomes of value for customers and other stakeholders. In combination with the analysis of streaming data and customer context, being able to adapt the execution of these processes to align with the customer’s goals will be a key corporate differentiator in the future.
One example of an area that has benefited from the IoE is precision agriculture, with traditional farming equipment manufacturers leading the way. When joined together as a system of systems, GPS-controlled tractors, in-field moisture and fertilization sensors, on-demand watering, fertilization, pesticide application systems and variable rate seeding equipment can maximize field productivity while minimizing cost. Precision agriculture enables alternative farming approaches that challenge industrial monoculture farms. With the aid of the IoE, smaller farms are able to compete by leveraging crop diversity and environmentally sensitive practices. Besides having smart connected farming equipment, the Big Data analysis of equipment and in-field sensor data can drive a decision support system that can guide farmers and their machines to optimum yields.