Selecting the right off-the-shelf applications or designing and building in-house applications to meet the needs of a growing organization is challenging. Identifying the requirements of the organization, designing a solution, and rolling the solution out to users are difficult tasks. But there is another long-range challenge that continues after application selection or design and construction: integration. Integration challenges the IT department differently because it involves connecting and adapting existing components of the IT environment, which are tasks that require insight into working applications rather than building new ones. For most organizations, integration is as important as selecting and developing new components. Nevertheless, integration is often an afterthought that occurs after an application is installed and in production. Usually, attempting to add integration to a system that has not been designed to be integrated from the outset is expensive, and the result is often only marginally satisfactory. Perhaps worse, the cost of integration is easily under-estimated because the effort to connect applications looks small compared to the effort required to design and build the application’s functionality. However, if suitable interfaces are not present, the cost of change can approach the initial cost of construction as components are rebuilt to accommodate proper interfaces.

Applications that work in isolation are rare; most serve the organization better when they are woven into the fabric of the system that make up the information technology of the enterprise. This fabric has expanded to include many aspects of an organization that were formerly not considered part of IT. Not too long ago, the heating, ventilation, and air-conditioning (HVAC) system was in the hands of the facilities department. The system was controlled remotely, first by pneumatic tubing, then by electricity. As controllers became more sophisticated, Ethernet interfaces were added. Ethernet is a bidirectional protocol. Consequently, the controllers could transmit information, such as temperature, as well as receive commands. To save costs, instead of an independent HVAC Ethernet network, the corporate network could be used. At that point, the IT department took on at least a portion of responsibility for the HVAC system.

Around 2000, the concept of the Internet of Things (IoT) began to appear. In its early stages, the IoT was associated with radio frequency identification (RFID) used for inventory control and the notion that computers could be supplied with content from things rather than humans.¹ The developments in HVAC fit into IoT well. At about the same time, the internal enterprise telecommunications network began to integrate into the corporate network. These developments extended the purview of the IT department beyond its earlier limitation to computing.

Enterprise Integration

Enterprise integration focuses on exchanging data and control between a wide range of elements involved in an enterprise. The most prominent among these elements are applications that run on computers that support the enterprise goals. Exchange with human systems and processes, mechanical processes, and various forms of instrumentation are also included. A critical aspect of integration is to identify and meet business drivers and requirements for integration.

Where does cloud computing fit into the enterprise integration challenge? The previous chapter discussed the architectural challenges involved in building an SKMS that ties together the service knowledge of an entire organization and how the features of an SKMS architecture can applied to other service management applications. The SKMS example reveals many aspects of the integration challenge. This chapter will go on to discuss enterprise integration more generally.

Challenges to Growing Businesses

As businesses grow, their IT needs follow a general pattern. Small systems start simple. When a business is small, its IT department may be only one person in operations who knows a little more about computers and has become the go-to person for computing problems. That person is probably not even called IT. There is no IT department and no formal IT system.

Most small businesses have an accounting system that is either a collection of spreadsheets or a desktop program. The finance department is a single accountant, with the entire accounting system at their desk. They make all entries themselves and maintain all the records. A business may also have specialized programs tailored to their business. For example, a law office may have a system for managing legal forms in their legal specialty. A construction firm may have a construction cost estimation programs tailored to the firm’s type of construction. Other small firms might have specialized inventory programs.

Billing information from specialty systems can be entered manually into the accounting system, but as the volume of business increases, manual entry becomes an issue. The accountant’s expensive time is taken up with data entry, the process is error prone, and the entries may not be timely. Either another person is delegated to data entry, or the systems can be integrated, which is often a better answer.

An integrated system quickly and accurately transfers data from specialty applications to the accounting system. After the initial cost, the ongoing costs are probably less than an additional employee. In addition, integration tends to be more scalable; the cost of running an integrated system increases only marginally as the volume of business increases. Depending on the architecture of the specialty applications and the accounting system, integration can be easy or can be difficult to the point that one or both of the systems must be replaced. In retail businesses, for example, the accounting system and the point-of-sale system are often integrated out of the box, but eventually, as a business grows, there are almost always new applications that are not integrated and need to be.

The previous example shows integration aiding in coping with increased business volume. Volume is not the only way business grows. Business IT grows along three axes: the volume of data and transactions processed by the system increases, the business spreads over a larger geographic area increasing the need for a geographically distributed system, and the system becomes more complex in its types of data and processing and interconnections between subsystems. In this chapter, these axes are called volume, spread, and complexity.

Not all businesses grow at the same rate along each axis. A business may concentrate on providing more locations that are convenient to their customers and charging premium prices for the improved service without increasing volume. Other businesses can increase volume without spreading geographically. Most growing enterprises find increasing complexity difficult to avoid.

Growth on each of these axes affects enterprise integration. Cloud deployments both complicate and simplify the development of integrated systems.

Volume

As an enterprise becomes more successful, it gets busier. The organization attracts more customers and clients, increases the amount of inventory to track, onboards² more employees to administer, and develops new lines of business to manage. All this results in more frequent and larger transactions of various types. Some are financial; others involve HR data on an expanding employee population, customer relations data on an expanding customer base, or larger and more varied inventories. The rise in transaction size, variety, and frequency generate one type of IT challenge that can be characterized as an increase in volume.

Applications

Volume forces applications to change. Employees can manage records in simple spreadsheets when the enterprise is small, and the spreadsheet lines of data and number of changes per day are measured in hundreds. As the enterprise grows, the volume of records becomes hard to manage in a spreadsheet. Changes and additions become too frequent to enter by hand, and the errors from manual entry become a drag on the organization. The performance of simple homegrown databases begins to bog down, and the data design requires a professional’s expertise to streamline data management.

Single-user spreadsheets, simple databases, and other applications become inadequate when the volume reaches the point that one person cannot manage the data. Enterprises typically replace these semi-manual processes with applications that support many simultaneous users. These applications use forms tailored to the tasks they perform. The forms discourage entry errors and have error-checking routines that increase accuracy. Databases replace simple files to manage increasing volumes of data and maintain data consistency and integrity. At some point, servers move from desktops and closets to a climate-controlled datacenter with conditioned power and backup generators where they are tended by trained IT staff.

Simple shared files are no longer adequate for managing data when data volume and number of users increase. Single-user applications are not concerned with colliding transactions when two or more users attempt to modify the same data at the same time. Supplying prompt and reliable data access to large numbers of users is a job that is usually delegated to commercial database management systems designed by specialists and running on dedicated hardware. The data and transaction models and structures, indexes, and physical data layout are designed and tuned by skilled database administrators.

Network

The IT network also expands as the enterprise volume expands. Instead of an externally connected router, a few simple switches, and some wiring to connect each desk, a large corporate network requires wiring plans and closets, firewalls, load balancers, perhaps several routers, many switches, and other facilities designed to support and optimize network high-volume usage within the enterprise.

Online customer sites are also challenged by volume increases. As the number of customers using the site increases, the enterprise has several new demands to face. Reliability becomes more important as the financial consequences of an interruption in service increase. The site must be prepared to support increasing volumes of transactions, which may be bursty, such that lower volumes are mixed with sudden bursts in activity that may be orders of magnitude greater than baseline activity.

Often, the bursts occur at exactly the time when performance is most critical—the performance of the web sites of advertisers during the Super Bowl, for example.

With increased volume, downtime increases in expense. In a small business, a failing system will affect a few employees and cause some delayed transactions. It’s not a desirable situation, but it’s not a catastrophe. However, when the business volume goes up, a down system affects more employees and customers. Failures become more expensive, and IT must begin to consider disaster recovery and design systems that keep working. Robust code that handles errors without ill effects, backup systems, and resilient systems that keep processing when a portion of the system fails all become more important as business volume increases.

Big Data

As business volume increases, the data generated by business transactions grows and becomes an asset in its own right. The data must be analyzed to gain value from it, but analysis of the volume of data businesses now collect presents new challenges. Data-intensive Internet enterprises like Google or Facebook generate daily data measured in terabytes and petabytes.³ Just storing this data has spurred the development of new mass storage systems. The old tools of data analysis, mostly relational and object-oriented databases with associated reporting tools, are not adequate. New tools have been developed to handle the new volume of data.

Support

In addition to capacity to deal with increasing volumes of activity, supporting IT users becomes an issue as volume increases. With thousands of customers a day, every permutation of user error, incorrect entry, ambiguous prompt, and user confusion will occur regularly. The site must be prepared to support these customers. This raises the requirements for user interfaces. When a site has 500 users a day, it may be acceptable if 1 customer in 50 requires intervention from support. When volumes reach tens of thousands of customers per day, reducing the number of issues becomes critical. Careful design of user interfaces becomes crucial.

Increased volume and subsequent criticality of IT systems engenders a need for a sophisticated service desk. Not only must the service desk support the volume of user issues and support representatives necessary to respond to the issues, the service desk must be proactive, detecting and resolving issues before they are noticed by users. To resolve issues quickly, the desk must also have diagnostic tools and manage assignments and workloads to optimize both human and computing resources directed at a problem.

As lines of business increase, the number and types of applications also increase. Each service offered by the organization has its own set of automation requirements. A new product line may require an automatic credit check on purchases over a certain amount. Automated manufacturing equipment may be added that requires extensive external data storage. This not only increases the volume and diversity of services supplied by the IT department, but it also expands IT along other axes.

Security

IT security is also often a challenge as enterprise IT volume grows. With increases in volume, security usually becomes more granular, which increases the volume of processing. For a small organization, often three classes of users are sufficient: nonemployee, employee, and administrator. Nonemployees, such as customers and employees of other organizations with which the enterprise does business, typically have limited access. Employees usually have access to everything but critical systems, which are reserved for administrators. Simple security schemes like this work well for small organizations. However, as the enterprise grows, security becomes more segmented. Access is assigned by “need to know,” and few need to know everything. One group of employees may have access to human resources data, others may have access to accounting information, others might have access to server configurations, and so on. Fine-grained segmentation like this requires processing that identifies users and their access rights, adding overhead to almost everything IT does. It is no longer sufficient to know who is using the system; the security system must also know what that user is authorized to do.

Enterprises with many customers and assets become targets for cybercriminals, further stressing the security system. The number of transactions that must pass increasingly complex security checks soars, but user patience does not increase proportionately, creating a demand for more efficient and tighter security.

Volume in Large and Small Enterprises

All of these challenges apply to any growing enterprise. As the enterprise grows, scale becomes a greater challenge. Management can no longer stroll around the floor asking questions and shaking hands to get a feel for the state of the business. Increasingly they must rely on reports and dashboards for an understanding of conditions. Instead of an accounting department of half a dozen trusted employees, large organizations may have hundreds, even thousands, of employees, each with different privileges, needing access to an integrated accounting system. Instead of a few applications that the IT architect must integrate with accounting, there may be hundreds of applications spread across hundreds of sites located around the globe. In addition, accounting may not be the only application needing integration. Supply chain management, inventory control, human resource management, and online business sites all have to be tied together.

A small organization that recognizes that these challenges grow as the enterprise grows and makes decision that will support rather than hinder growth will have a significant competitive advantage. A growing large enterprise is already caught up in these issues whether it realizes it or not.

Geographic Spread

Most businesses, as they grow, also expand to additional locations. They may have branch stores, regional sales offices, or remote manufacturing facilities. A law office may open a branch in another city but want to keep a single electronic document management system in the main office. A retail business may open branch stores with a single accounting system in the main store. Large enterprises expanding into new—often global—markets require new remote facilities, ranging from small sales offices to entire factory complexes. Geographic spread adds other factors to the integration challenge. Integrating with a remote system introduces unreliable networks, transmission delays, and new security vulnerabilities.

All networks are subject to accidents and events that may interrupt or interfere with transmission, some much more than others. An internal network within a single geographic site is usually more reliable than a wider area network. Global networks often depend on satellites and undersea cables. A private wide area network can be made more reliable than a public network. The Internet, with a wide range of known and unknown cooperating providers, can be the most perilous of all.

There are hosts of challenges when dealing with unreliable communications. Data transmission protocols are designed to minimize the effects of data delivery slowdowns, gaps, and cessations, but they do not eliminate these problems. When a data link goes down, transactions have to be rolled back. If the rollback is not done properly, data may become inconsistent or corrupt in unpredictable ways. Processes dependent on data from a remote site may halt, slow down, or become unstable. The remote site may suffer from the same conditions.

These situations can be mitigated. Data connections can be made more reliable, and redundant connections can be established. Applications can be designed to cope with vagaries in data transmission and avoid inconsistency and other deleterious effects. However, even with the most elaborate and careful preparation, sites must balance the needs for continuous availability and instantaneous data consistency. Said differently, if you want perfectly correct information from a remote site, you must be prepared to sometimes wait. If you must have data instantly and never have to wait, you must be prepared for data that occasionally will not be correct until the network catches up.⁴

Complexity

As businesses grow, their systems are likely to increase in complexity. A law firm may expand to different specialties and require systems for handling different forms and processes. A construction firm may also take on different specialties requiring additional systems such as computer-aided design and automated logistics planning. As the number of systems increases, integration is likely also to become more complex as more systems are wired together in different ways.

Complexity is a concept used in a number of disciplines. In some contexts, complexity has a precise meaning. In other contexts, it is descriptive term without precise meaning. Here, the complexity of enterprise IT systems is descriptive. The IT system of an enterprise can be seen as a single entity that provides services to the enterprise and its customers. This entity must be maintained and modified to meet changing business requirements. As the demands placed on the IT system increase, the interactions between each system component increase in number and variety. Connections between components increase. As these interactions and connections grow, the implications of any input or change to the system become more difficult to predict, and the system becomes more difficult to manage. The increased number and type of connections and interactions is what is meant by complexity here.

Some of the factors that contribute to enterprise IT complexity are the number of subsystems and applications, the number and types of relationships between the system, and the system dependencies and other state relationships between subsystems. Other contributors to complexity are the potential for influence on the activity of the system from its users and changes in system dependencies based on the activity of the system.

Integration

Complexity and integration are linked. Integration is a cause of complexity. An enterprise IT system consisting of a single application with few inputs and outputs running in isolation may be internally complex, but the enterprise system is not. Two such applications running in isolation are also not complex. But when two applications are integrated, they no longer run in isolation. The two applications can and often do influence each other, and the complexity of the IT system increases.

Anyone who has administered systems is familiar with an increase in complexity. Two applications launched on the same server (even simple applications with few inputs and outputs, minor internal processing, and no interactions between the applications) eventually contend for system resources and make management more difficult. For example, the administrator may have to be careful not to start the two applications at the same time to avoid contention for startup resources. Even when there is no overt interaction, each additional application contributes to complexity. This simple example illustrates the increase in complexity as a system grows.

Coupling

Complexity is not the same as coupling, as most practitioners understand coupling. Coupling refers to the extent that components rely on the internal details of other components. In a well-designed system, internal changes will not impair other components as long as designated external interfaces are not affected. In a loosely coupled system, all interaction between components is through defined interfaces. Closely coupled components go beyond defined external interfaces and use characteristics of components that are not intentionally exposed. If a closely coupled component requires a modification, the effect of the modification on the relationship between the couple components is unpredictable without examining the code of both components. Closely coupled systems are usually the result of loosely defined external interfaces, or an architecture in which the boundaries between components are not well defined.

An example of close coupling is an application that uses the content of a file written by another application. If the format of the file is not publicly exposed and defined, the developers of the component that wrote the file will assume they are free to modify the structure and semantics of the file as they see fit and not worry about affecting other components. If a component is closely coupled, it may rely on the structure and semantics of the file. If the developers of the first component decide to modify the structure of the file, the second component is likely to fail.

Complexity is on a different level. A loosely coupled system can suffer from complexity, and a closely coupled system may not be excessively complex. Even though the internals of each component remain private, a complex system may fail when each component is performing exactly as specified, but the content and frequency of inputs and outputs change. Complexity describes a situation where the output and input of applications are intricate, where the action of one application is dependent on inputs from several other applications.

Complexity addresses the nature of the exchanges of information between components. In a simple system, a change in the input to a component causes a predictable change in the output of a component receiving the output of the first component. In a complex system, a change in the input to the first component causes an output that is dependent on outputs from several other components. In simple situations, coders can rely on simple tables of possible inputs and outputs. However, most applications today are affected by more than simple input. Often the timing and sequence of inputs are factors. Also, the number and possible values of inputs can make preparation and evaluation of input and output tables impractical because the tables become intractably large. Typically, in complex situations, the performance of the system is predictable most of the time, but when the circumstances are extreme, the system may not perform predictably. Unfortunately, the extreme situations are often precisely when we hope that a system is rock-solid.

Support

Every time an application is integrated into an environment, more integration pathways are opened. Service desks are frequently an integration center because issue creation and response become more automated, as they must in a growing enterprise. Integration with the security system resets passwords. Integration with performance management generates automatic warnings at the service desk of slowing processes. Integration with the HR system puts a user in the service desk user-list automatically when the employee is hired. Often, each connection is made as a need arises and without consideration to an overall design.

As a system becomes more complex, detecting, diagnosing, and remediating issues become more difficult. For example, a service desk application may be designed to automatically generate a service desk incident document when the performance of a set of applications dips below a minimum level. Here’s a trivial example of complexity: one of the instrumented applications begins to thrash, consuming resources at an unexpectedly high rate, but no longer passing tasks to a second application. The resource consumption of the second application drops because it is receiving zero requests from the failing application. Consequently, the idle second application and the thrashing first application average out to normal consumption. In isolation, neither high nor low resource usage indicates a problem in either application. To respond correctly, the integrated support system has to have logic that recognizes the dependency between the two applications. But this is questionable architecture because removing or changing either of the applications could require changes in the logic in the integrated support application. Good design can mitigate this kind of dependency, but designers must be aware of the potential problems.

Unpredictability

One of the worst consequences of complexity is unpredictability. When applications are connected, predicting and understanding the consequences of modifying one application can become difficult. The more complex the connections, the more difficult prediction becomes. The worst part is that the unpredicted effects of a change may appear remote from the change.

For example, when two applications are integrated, one application may receive data from the other, and the receiving application may have to wait for data from the other; in other words, one application may be starving the other application for data. Data starvation can be difficult to analyze and remediate. A complex application may have hundreds of threads, each spending most of its time waiting for data or events. Detecting the threads on the critical path at any given time is difficult. If the holdup is discovered, there are several solutions. One solution is to modify the first application to supply data more readily. This will solve the problem, but fixing a defect in one application by changing another application only masks the vulnerability of the second application. On a different day, the second application could fail because yet another element of the environment slows down. Another approach is to modify the receiving application to be resilient to data starvation. This is only one example of the ways in which applications may become coupled when they are integrated. These kinds of relationships contribute to the complexity of the system.

Proliferation and Brittle Systems

Unfortunately, some increase in complexity is inevitable. Relationships often are complex, and knowledge is required to handle the system well. This is part of the overhead of business expansion.

As the enterprise grows, integration becomes more complex. This complexity can become a business-crippling problem because complexity can make systems brittle. A brittle system will fail catastrophically when small, apparently inconsequential changes are made.

Expanding or changing a brittle system to meet new needs is likely to affect portions of the system that have only a tenuous relationship with the change. When a system is brittle, even the simplest fix or update can demand a high level of technical expertise, possibly detailed knowledge of the entire system, to avoid disaster. This consumes inordinate time and resources. The most skilled members of the development team are diverted from critical forward-looking projects and forced to deal with the simplest problems. This can result in a development team that is turned on its head: the best and most knowledgeable developers perform maintenance that ought to be simple, while challenging new projects are assigned to junior engineers, whose inexperience contributes to future complexity. As complexity increases, brittleness becomes harder to avoid. A small change in a component may cause a massive change in the behavior of the system.

Integration typically involves resource constraints as well as data constraints. Communication between applications is typically easier and faster when applications run on the same server. However, on the same server, one application can be starved for CPU due to the activity of the other application. Again, the system may have to be managed by load balancing or other means to avoid contention, which may have been preventable with better integration design.

The sophistication and internal complexity of applications is not so much a concern here. Complex applications may be involved, and they usually increase the integration challenge, but we are concerned here with the system: a group of applications with intertwined inputs and outputs. Understandably, when developers design an application, they tend to concentrate on the functionality of the application they are building and pay less attention to the external applications they may interact with. Designing external interfaces intentionally is usually the solution to this problem. Service Oriented Architecture (SOA) has put new emphasis on external interfaces, which has helped with integration complexity. However, the burgeoning of external interfaces helps only one aspect of the problem. Interfaces isolate the internal code of an application from integrated applications. However, this does not necessarily decrease the complexity of dependencies among applications.

Typically, integrators enter the scene long after the application is developed. When interfaces are clearly defined, their task is much easier, and many blunders of the past are circumvented. The problem of complex dependencies has to be addressed with a thorough understanding of what the applications do and how they interact in production.

Careful planning can ameliorate the effects of a growing system that encompasses expanding functionality.

ITIL and Complexity

When applications are designed to be integrated and the integrated system is designed as an integrated whole rather than a patchwork of applications, the resulting system is usually less complex, consequently more robust, and less brittle. ITIL practices begin with strategic planning of a coherent system that is tailored exactly to the business needs of the enterprise. With strong leadership that understands both business and technical challenges, the ITIL continual improvement cycle is effective in reducing complexity. However, the approach is dependent on leadership that can bring many different perspectives to the table and produce plans that satisfy all the stakeholders and yet retain a simple and vigorous architecture. Finding and empowering this kind of leadership is a formidable challenge.

Most organizations do not implement ITIL practices all at once. Most often, they implement a single practice or a single group of practices at a time. Many sites begin to adopt ITIL by implementing a service desk that follows ITIL practices. In many cases, implementing an ITIL service desk first has improved service levels and effectively proved the worth of ITIL practices in the organizations that adopt ITIL practices. However, single applications or groups of applications do not automatically help reduce system complexity.

The ITIL continual improvement pattern establishes a managed service portfolio with an infrastructure designed to support the entire portfolio instead of infrastructure to support individual services. This pattern has the potential to rationalize, simplify, and streamline integration. The pattern implies that the integration requirements of each component of the system will be examined and optimized at a high level.

Unfortunately, even where the value of a strategic plan is appreciated, there are obstacles. Advanced planning is often difficult in a fluid corporate world where business units are acquired, sold off, and merged. The IT system must change rapidly, incorporating new or different technology, replacing technology that has left with a departing business, and fusing together similar but not identical technologies. When time pressure is great, there often is little time or expertise to invest in designing a strategic portfolio with simplicity built into the infrastructure architecture.

Organizations that recognize the danger in growing complexity can act to reduce complexity or slow its rate of increase. One solution is to choose a single source for an integrated suite of products. This solution uses the engineering skills of the authors of the suite to help tame complexity and supplements the skills of the enterprise IT team.

An alternative approach is careful architecture by the enterprise IT team, developing a plan that avoids redundancy and unnecessary connections between applications while promoting consistency and reliability. This approach puts responsibility for a strong integration design on the enterprise IT team. Not all IT teams are willing or able to take on that responsibility, but the return is independence and flexibility. One element of this approach is to include clouds in the system design.

Often, the two approaches are different roads to the same end. A team that starts out aiming to plug carefully chosen products into their own architecture may find that they are consistently choosing products from a single vendor or a small group of vendors. A team that decides to leave responsibility to their vendors will usually find that there are aspects of their business that no vendor can help with and they have to come up with their own integration plan.

Cloud Systems

A cloud can be used in many ways to support integrated enterprise IT services. Stripped of all the business characteristics such as metered usage that go with external ownership, a cloud is a pool of computing resources, usually remote. These resources are in the form of computing machines, data storage units, and network facilities. Both computing and storage are usually virtual; cloud consumers do not interact directly with physical resources. Instead, the cloud provider offers an interface that acts like familiar operating systems and storage appliances such as disk drives. The simulation of the familiar environment is exact enough that consumers can run the applications they ran on their local premises on the cloud with few modifications.

The familiar patterns of cloud services (Infrastructure as a Service, Platform as a Service, and Software as a Service) offer ascending levels of abstraction to the consumer. Each level of abstraction presents a new set of benefits and challenges that apply to enterprise integration.

The initial impetus for a growing organization to begin to use cloud implementations is often to acquire computing capacity quickly and cheaply. This is especially true for technology-oriented organizations whose expansion is directly tied to expanding computing capacity. Using metered IaaS resources that can be expanded and contracted rapidly and easily, these companies can stretch their investors’ dollars and advance quickly. These organizations are lucky: they do not have a legacy of systems that must be integrated into cloud expansion. Their choices will be relatively simple, and they have the luxury of building on a cloud platform with the potential to avoid many of the issues that IT has dealt with for years.

Advantages of Cloud for Integration

The three axes of enterprise IT growth described apply also to cloud implementations. Cloud-based applications, particularly SaaS applications, solve some integration issues for both large and small businesses. In fact, one of the issues facing large enterprises is the possibility of abandoning an integrated system that is the product of large sunk investments in both money and innovation over decades for an off-the-shelf service that solves integration issues easily and cheaply from the viewpoint of the end user.

Figure 8-1 depicts a traditional branch accounting system that is reminiscent of mainframe architectures and was typical in the early days of desktop accounting.

Each branch had its own accounting system. Headquarters also had its own accounting system with the same functionality as the branch systems. A corporate consolidated accounting system took input from each of the branch systems and produced combined statements that reflected the entire enterprise. Like mainframe systems that accumulated transactions through the day and then ran a batch job to post those transactions to the master record, these early distributed systems periodically updated corporate systems at headquarters. In its earliest form, the update might involve physically transporting a tape to the central system and returning a tape with updates on things like prices to be applied to a remote system. Later, the transmission was via a modem acoustically coupled to the telephone system. Eventually, these ad hoc methods were replaced by leased communication lines, private networks, and eventually public networks, such as the Internet. With faster data transport, batch updates evolved into transactional updates.

There are several problems with this approach. Managers in branch offices had to wait for the delivery of corporate reports to view their branch in the corporate context, comparing the performance of their branch with other branches and similar tasks. Infrequent updates of the corporate system meant that the corporate books were only periodically coordinated with dynamic situations. For maintenance, each branch required staff with the technical skills to administer and maintain the branch application. When a patch or update was needed, the patch or update had to be applied at each branch.

The cloud equivalent deploys a single copy of the application on a central cloud. End users access the application from their devices via HTTP, as illustrated in Figure 8-2. This is an improvement over Figure 8-1 Using a cloud, there is only one instance of the application to maintain, back up, and provide with disaster recovery and failover. Employees can use the application off the premises, and they can use a range of devices to access the application. In other words, employees are not tied to specific hardware or locations.

Cloud implementations usually use HTTP or HTTPS as the transport between the end user and the application. Consequently, large segments of the implementation are off-the-shelf, often open source, software. This decreases development costs and allows developers to concentrate on business logic rather than lower-level architecture.

The application can be supplied as a service from a third party. In that case, the service provider takes responsibility for maintaining the application in the cloud. The provider is usually responsible for maintenance, backup, disaster recovery, and failover. Maintenance usually includes updates and patches, and the provider takes responsibility for the performance of the application in the cloud.

Applications treated as a service do not necessarily involve a third party. An enterprise can build SaaS architecture in a private or public cloud, operate the application as a service to their employees, and gain many of the same advantages as SaaS applications from third parties.

Expanding the perimeter of the business presents one type of integration challenge that cloud implementations often ease, but a different set of challenges arise as the IT environment of the business becomes more complex, increasing both the number of applications and transaction volume that must be integrated into the fabric of the business.

For standard applications such as customer relationship management (CRM), human resources management, or service desk, there are SaaS applications available that will work well for them. All of these applications need to be integrated; onboarding a new employee into the HR SaaS application should not have to be repeated when the same employee has to open a service desk ticket. A new sales rep onboarded into the HR SaaS application should automatically appear as a sales rep in the CRM system.

Integrating these systems in a SaaS cloud is more dependent on the cloud SaaS vendor than installations on the local premises. Although integration of off-the-shelf software is dependent on the provisions made by the vendor for integration, the consumer still controls the environment in which the software runs.

The appearance of cloud-based systems has simplified many challenges for both smaller and larger businesses.

Small and medium businesses have especially benefitted from SaaS, which offers applications to small and medium businesses on affordable terms that are equal in complexity and power to those used by larger enterprises. Many of these SaaS applications are designed to meet enterprise integration requirements.

Nevertheless, even if a business chooses only to use SaaS, they still need to be aware of cloud integration challenges to plan their entire system well and choose the right products to include.

The previous chapter discussed a single type of integration: service management applications. Integrated IT is a bigger challenge: service management applications are only a subset of the total system, and the interchange of data and control in a complete IT system is more varied and complex. The SKMS system of the previous chapter is a data collection system; a few service management applications involve control as well as data, but service management is usually devoted to exposing data to service managers and planners. When we step beyond managing services and begin to concern ourselves with delivering services, we enter a more complex environment of integration.

Other chapters have discussed why enterprises need to be integrated. Even a novice wagon-master knows that the wagon train moves faster and farther when all the horses pull in the same direction, and every business manager knows that an enterprise at cross-purposes with itself is not likely to succeed for long. Integrating IT means establishing the communications paths that make IT services work together.

Building an Integrated Cloud

Building an integrated IT system begins with planning. In an ITIL environment, planning is the purview of Service Strategy and Service Design. The core of ITIL Service Strategy and Service Design is the service portfolio, which lists in detail all the internal and external services offered by the IT department. The service portfolio is oriented toward business decisions. It details how the services support lines of business and provides the information necessary for executive management to make business decisions about adding new services and decommissioning services that no longer serve a business purpose.

Top-Down Design

This information is useful as a starting point for designing a technical architecture that supports strategic business plans for IT services efficiently. The service portfolio usually will provide a framework that determines the volume, geographic spread, and complexity of the IT services. In many cases, the service portfolio either spells out or implies the integration requirements for the system. Integration is more easily simplified from the top-down view of the service portfolio than the bottom-up view built, while integration requirements are identified as they appear in production.

This top-down approach has clear advantages, but it can easily turn into a waterfall methodology in which barriers are erected between development and design. These barriers, which are erected with all good intention, prevent developers from responding to a changing IT environment.

Software architects should be aware of the possibility of slipping into a waterfall mode and act to avoid its barriers.

Software design is not like designing a bridge or a skyscraper. Software is infinitely malleable, even in its smallest details. A bridge designer is constrained to a few types of girder connectors because the connectors are manufactured and too expensive to build to order. The designer must choose among the connectors available that meet the engineering requirements, which are clearly understood and specified. These choices have to be made before the first girder is erected. After the choice is made, changes will be expensive and require a formal change order with several levels of approval.

The software designer also has prebuilt components in the form of software libraries and standard modules, but they are not as constraining as the component of a physical structure. Their strengths and limitations are not spelled out in tidy numeric tables. Software code is much more easily changed than a steel connector, and developers are often able to make substantial improvements with little effort by changing the details of designs. In addition, the product built from a software design is not as easy to project as the product of a bridge blueprint. Often, only after a software project has begun do some defects or deficiencies become clear. Not to take advantage of the opportunities to correct or improve the design when opportunities appear can result in projects that do not meet their promise.

Consequently, a pure top-down design is often problematic in software but still has important advantages. Balancing the two extremes is one of the key challenges to software architecture.

Planning

An integrated business system has both requirements that are specific to their business and requirements that are common to most systems. Business rules can be complex with many exceptions and special circumstances. They are also likely to change as a business evolves. Examples of business-specific requirements are the details of writing and tracking rental agreements in an equipment rental business where the business rules may differ for each type of equipment. Temporary special deals and negotiated agreements may be common, which make business rules not only complex but also dynamic. However, even an exceptionally atypical business will have architectural requirements for integration that they share with most other businesses.

Architectural Fundamentals for Integrated Systems

The fundamentals in this section specifically apply to the integration of applications and components into integrated IT systems. These fundamentals apply to clouds as well as systems installed on the premises. Architects are often challenged to combine systems on the premises. To this, they may have to add applications deployed on SaaS, PaaS, IaaS, and on-premises applications into a single system.

Each application in the system must communicate with other applications in such a way that data is available when and where it is needed

This is the basic tenet of integration. Data must be available where the business needs it. That may mean that the output of one application flows into another or that managing the system requires data from one (or several) to be available for management viewing. This requirement is fraught with difficulties. If there is no overall plan, the common tendency is to patch together connections between applications as needed in the easiest way available. When a connection to an additional application is needed, the easiest way to connect often changes. Consequently, a new communications channel is opened. Opening new channels invites complexity. This is especially bad where adding a new application adds its own complexity in addition to integration complexity.

A master plan can be helpful. A spoke-and-hub⁵ or message-bus⁶ design pattern for integration can provide the discipline to keep the number of data channels down and make data available where it is needed without a special code for each application.

A spoke-and-hub architecture has a central hub where all integration information is collected and distributed as needed to the integrated applications. A bus architecture has an information bus. Each application puts its information available for integration on the bus and may receive information from other applications off the bus. Both of these architectures are easier to understand than complex point-to-point integration. These architectures also make it easier to add new integrated applications without adding complexity. See Figure 8-3.

Internal members of the organization and consumers of the organization’s services must have access to integrated data and services

It is understood that integrated data should be available to the people and processes that want and need the information. Unfortunately, this often is not as easy as it ought to be. The data may be present and even flow from application to application. However, the data channels used by individual applications may not be accessible. For example, if two applications communicate via a hard-coded socket-to-socket link, tapping into the information flow between the applications will probably require substantial refactoring and possibly additional communications links. If the information flowing over the link is in a form that depends on implementation details of the connected applications, data interfaces may have to be redesigned to decouple the applications. Spoke-and-hub and bus architectures avoid this difficulty by providing a uniform communications channel that is designed for additional components or applications to tap into without disturbing individual applications or the fundamental architecture.

Integration should not open security holes

There is always a possibility that integration will reveal information or enable functionality that should not be accessible. The authors of an application are usually keenly aware of the security requirements for the data and functionality they manage and assume that data and functionality exposed for integration will be treated with the same awareness. However, the authors of an integrated application may be much less aware of the issues and inadvertently disclose information they should not.

An example of this can occur when a multi-application dashboard is added to a system. Usually a dashboard is designed so that the user can choose from a range of data to display from each application for a dashboard tailored exactly to their needs. However, not all users may be authorized to view all the data. An ordinary user able to configure the dashboard to display private HR information, such as Social Security numbers, is a security breach. The dashboard becomes a backdoor into HR that circumvents the HR authorization system. The dashboard must be designed to display only data that all dashboard users are authorized to view, or the dashboard must be smart enough to check the user’s authorization before displaying data.

Generally, a user who is denied access to some data or aspect of a service ordinarily should also be denied access to the same data or functionality through other integrated services. There may be exceptions to this principle when the integrated application has authorization rules that are allowed to override authorizations in other applications. Architects and developers should be aware of the potential for opening security holes and act to prevent them. A design that has a single security system that is called on to manage all authentication and authorization following manageable policies that reach to integrated data and activities will be much easier to manage than a design in which security is built into each application.

In addition, architects should be aware that an integrated application that handles sensitive integrated data must be protected from outside access as carefully as the originating application. Mobile apps provide a good example of this. Suppose a mobile app is planned for mortgage brokers to enable work outside the office. The mobile app is expected to integrate with corporate servers and provide a full range of data to the mobile broker. However, a mobile app usually emphasizes hassle-free access, not ironclad, security. If a developer does not make a special effort to secure the data and functionality provided by the app, the mobile app could invite devastating security breaches by exposing private data to anyone who happens to pick up the phone with the app installed.

User privacy required for business purposes or mandated by laws and regulations must not be violated by integration

Most systems, especially financial, medical, and systems that manage data from the European Union, must guarantee that access to data follows exacting privacy requirements. For example, regulations may require data to be encrypted as well as protected from unauthorized access. There may be special requirements for the disposal of data and other aspects of data management. Integration architects must note that an application that is integrated with a regulated application is also subject to regulation. The system must not only protect privacy under normal operations. It must also protect privacy from unintended or malicious access.

These requirements are best treated as guidelines to follow at all stages in developing the system. They have wide consequences for application design and implementation. For example, the security requirement may seem obvious, but implementing authentication and authorization uniformly across several applications can place detailed requirements on application design that reach deep into the workings of the code.

New features must be cheap and easy to integrate without affecting production services

Adding new applications and services is easy at the beginning but often becomes difficult and expensive as the system matures.

Point-to-point integration is often a seductive mistake. With only a few applications to manage, a quick integration (say reading a table in from another application to integrate the two applications) may be quick and easy. Convincing upper management to invest in a hub-and-spoke or bus architecture is a hard sell. The immediate return on the investment is low compared to the quick alternative.

If the enterprise continues to grow in volume, geographic spread, and complexity, as many successful enterprises do, eventually the system may become so cumbersome that adding or modifying an application is no longer easy and quick. Instead, it takes a master architect who understands the history of the system to tap into the application. The effects of the addition are often unpredictable, and making the addition may take weeks of tinkering to get the whole system working properly.

At this point, the enterprise must decide to pay the price of an architectural solution, which will be much higher than it was when the system was simple, or they must be content with the system as it stands.

The user experience, both internal and external, of the integrated system must aid rather than hinder growth in productivity

Perhaps this principle should be unstated, but it is occasionally ignored. Integration must improve the system. Systems can be improved in several ways. They can become more efficient by requiring fewer system resources to accomplish the same tasks. They can become more transparent by exposing data and functionality that can be used by functions such as audit and security. They can be made more reliable or easier to maintain. However, the most important way a system improves is by enabling greater productivity.

Any system that hinders productivity growth should be rethought and redesigned. This applies equally to integration. Integration can lead to displays of irrelevant information, may degrade performance, or may cause system unpredictability. These problems often arise when integration is pursued for its own sake. Integration must be justified by business requirements, not architectural elegance or because it is easy. When integration begins to hinder productivity, a review of the enterprise’s approach to integration should be considered. Often this will be a job for an ITIL strategy team.

Architectural Fundamentals That Affect Integration

The fundamentals outlined in the previous section all apply specifically to enterprise integration. Other fundamentals apply to all IT systems that effective integrated systems also rely upon.

Avoid needless data duplication

This is an informal statement of a principle related to the relational database third normal form requirement. Third normal form requires all the data in a row is dependent only on the key and not depend on other elements of the row. Stated formally, third normal form may be confusing, but it is easily spotted in practice. When a column in a table has repeated data that could be placed in another table to avoid the repetition, the data is probably out of third normal form.

Following third normal form prevents inconsistencies between tables. Avoiding data duplication between applications prevents inconsistencies in a similar way. Figure 8-4 illustrates abstracting user phone extensions maintained in two applications to a single directory application. Abstracting the extension to the directory avoids the possibility that one application has the correct extension and the second application has an out-of-date extension. The correct extension need only be maintained in the directory application. By not duplicating the phone extension data in applications A1 and A2, this architecture both simplifies administration and avoids inconsistency in the integrated application.

The principles for integration and relational databases are analogous, not identical, mainly because references such as embedded foreign keys are usually more quickly resolved in a database management system than between integrated applications. In Figure 8-4, the usernames, Fred and Sally, are the embedded foreign keys because they can be used to look up the phone extension.

Even in relational databases, schemas are sometimes denormalized in order to improve performance by accepting some consistency issues in return for decreased overhead with fewer lookups. For application integration, the performance issues can be much more severe because querying another application may be more resource intensive than a database lookup. Often application queries can increase rather than decrease complexity and brittleness in a system because they complicate the mesh of dependencies. In Figure 8-4, the duplicated phone extension introduces the possibility of inconsistent and incorrect extensions in tickets, but moving the extension to a directory application could slow down ticket creation because an additional remote call is necessary to get the extension to be included in the ticket.

There is no clear-cut answer to choosing when to duplicate data. Performance, accuracy, and complexity all have to evaluated and balanced as well as possible.⁷ Often a caching scheme can help. In a hub-and-spoke architecture, caches can be part of the hub. Bus architecture can have a cache attached to the bus.

Scalable and elastic systems to handle varying workloads are required in most enterprises

This fundamental truth will be addressed more in Chapter 11 because elastic design is a basic virtualization and cloud characteristic. The favored method for building scalable systems today is to build parallel processes that can be scaled up and down by adding and subtracting processors and processes running on the processors. This is the basic principle behind large datacenters with thousands of computers running parallel processes.

Unfortunately, integration and parallelization don’t always get along well. A parallelized task can be no faster than its slowest thread. Consequently, a thread waiting for data from an external integrated process is likely to hold up the entire task. There are various ways to avoid this kind of situation. One method is to cache data from the external process so it is always instantly available to the task’s threads. In a hub-and-spoke or bus integration architecture, caches can be built into the hub or the bus to buffer the effects of slower processes.

When designing an integrated system, data availability can be important for supporting acceptable performance levels as volume scales up. Although initial designs may not need facilities for high-speed availability, designs should allow for future requirements.

The system must be reliable. Outages must be brief and seldom, even in the midst of disaster

Integration and system reliability can be in opposition. The more integrated a system is, the more each component relies upon other components. Unless measures are taken to avoid it, the failure of one component may affect several others. Each of these may affect others, and eventually the entire system may stall. Alternate sources, system redundancy, strategic data caching, and awareness of availability versus consistency all can help increase reliability. It is also important to consider failover and disaster recovery. The key problem in failover is transferring state from the failing system to the failover system. An integration hub or bus can be designed to help with this by keeping clues to the integrated data available to the integrated processes at any time. This facility can also be useful in disaster recovery.

Maintenance must be minimal. Service outages for maintenance are not desirable

Software maintenance includes a wide range of activity and becomes more important as enterprises grow. One important aspect of maintenance is keeping up with software revisions. In a small business, replacing an application with a revision may be annoying, but it can usually be done without devastating effort. As enterprises grow and become more integrated, revisions can become more difficult. If the integration is not done through a documented interface, integration code often has to be rewritten and retested, which can be quite painful. Instead of running an install script that smoothly and silently replaces the previous version with no or only a short pause in production, someone has to design and write new code, create a test bed, and complete testing. Then, an error-prone manual installation of the new code threatens to further halt production. Large-scale integrated applications can take days to install and bring into full production if efficient upgrading is not given consideration in the fundamental architecture and each individual integration.

Parallelism provides a fundamental solution to maintenance issues. When several parallel processes are running, the whole system can be designed to keep running while individual processes are shut down for maintenance. This pattern can also be followed even when there is usually only one process. If an updated process and the production process can run at the same time and the production load transferred to the updated process, maintenance can proceed with minimal interruption.

Cloud Requirements

Cloud changes the business of IT significantly, but reliable integrated systems have the same basic requirements on the cloud as they have on the enterprise premises.

The fundamentals discussed in the previous two sections apply to applications and components running on the enterprise premises as well as clouds. The foundation for efficient and reliable traditional systems does not differ from the foundation for cloud-based systems.

Nevertheless, clouds have important special characteristics. Clouds are almost entirely virtual, and they are separated from the enterprise by network links, often over the Internet.

Virtual machines themselves run almost identically with their physical counterparts. However, because they are virtual, they can be added or removed from the system more easily and more rapidly than their physical equivalents. That means that the system, particularly the integration system, must be prepared to deal with a more dynamic environment. A physical system, which depends on hard-coded addresses to maintain connections between processes, is slow and annoying to change when reconfiguring the system, but it works. In a dynamic virtual environment, it can be a disaster. Consequently, message bus or spoke-and-hub patterns that manage dynamic addressing are even more important in cloud deployments.

Virtual systems can also be quickly reconfigured to be distributed to remote datacenters. This can be especially useful during failover and disaster recovery, but the movement may also introduce changes in latency between processes, and network reliability issues may appear, which the integration system may need to deal with.

Network performance and interruptions are more significant in cloud deployments. Generally, provisions for supporting availability and restoring data consistency after network issue must be more robust when a system is deployed on a cloud. Mobile devices maintain availability and eventual consistency by caching on the local device and then reconciling the cache with data in the cloud. The consistency and availability challenge on mobile devices is formidable because wireless connections to mobile devices are more vulnerable than conventional networks. However, as reliable as a network may be, in most cases applications are required to be available and consistency, must be restored after an interruption. Robust cloud-deployed systems must take this into consideration.

Conclusion

Although these requirements are diverse and complex, they are not impossible to meet. In fact, they are goals that have directed enterprise application design for a long time, and most design patterns used in enterprise applications are intended to support these requirements. Cloud implementations face all the problems of implementations limited to the enterprise premises. However, in some cases, some of the effort required for integration is taken on by the cloud provider. In other cases, the cloud architecture itself helps with good integration design. In all cases, sound integration depends on understanding the enterprise integration requirements and establishing an architecture that will support those requirements as the enterprise grows.

¹Kevin Ashton is usually attributed with coining the IoT phrase when he was working on automated inventory control for Proctor & Gamble. See www.rfidjournal.com/articles/view?4986. Accessed September 2015.

²Onboarding is convenient jargon for the entire process of taking on a new member of an organization. It usually starts at hiring and includes orientation and training, providing necessary tools and facilities (such as a cubicle, chair, and computer), enrolling in human resources and payroll accounting programs, opening computer system accounts, establishing authentication and authorizations for security, and so on. In IT discussions, onboarding often focuses on establishing IT accounts and security. The reverse of onboarding is off-boarding.

³A terabyte is a trillion bytes. A petabyte is a quadrillion bytes. A few years ago, the entire Library of Congress was estimated to contain 3 petabytes of information.

⁴See Marvin Waschke, Cloud Standards, Apress, 2012; pp 124-131 have a discussion of the Consistency, Availability, Partitioning (CAP) theorem and its implications for system communications. The CAP theorem asserts applications integrated over an unreliable network must balance data consistency with data availability. In other words, if completely accurate data must be available at every instant, the network must be perfect. If a perfect network is not available, consistency and availability must be balanced. Since perfect networks are only theoretically possible, availability and consistency are always a compromise. Most Internet applications settle on “eventual consistency,” which places availability above consistency but guarantees that the data will eventually be consistent. Mobile devices with intermittent or varying connections exacerbate the need for eventual consistency.

⁵For more discussion on spoke and hub architectures, see www.enterpriseintegrationpatterns.com/ramblings/03_hubandspoke.html. Accessed September 2015.

⁶For a detailed discussion of the message bus pattern, see Gregor Hohpe and Bobby Wolfe. Enterprise Integration Patterns. Boston: Addison-Wesley, 2004 (Kindle location 3386). Note that a message bus and an enterprise service bus (ESB) are not identical. An ESB provides access to services, frequently via SOAP web services. A message bus provides access to data and events, not necessarily services.

⁷When storage was expensive, data duplication was avoided to save storage space. I cannot recall a single instance in my experience when a decision to deduplicate data solely to save storage space was a good decision. The code complexity and maintenance never justified the cost savings. Of course, the fact that the price of storage has decreased continuously for a long time has something to do with this, but not entirely. Infrastructure costs seldom exceed the cost of complexity. There are many good reasons for avoiding duplication, but saving storage is seldom one of them.

In distributed environments, including clouds, eventual consistency, discussed in note 4, puts a premium on availability over consistency but insists that the system eventually attain consistency. This applies to caches as well as data stores.