The preceding chapter described some of the guiding requirements on enterprise integration. This chapter describes some of the architectural patterns that implement these requirements. Software architects often describe systems with “design patterns.” The contents of this chapter are best understood as design patterns.

Design Patterns

Design patterns are a common way of describing the principles underpinning software architecture and are a basic tool for planning software architecture. The method was originally created to express basic design components of dwellings and other structures, but the method applies well to software architecture.

A design pattern is a solution to a frequently occurring architectural problem. The pattern describes the problem and solution in a form that architects can use to identify recurring problems and implement common solutions in widely varying contexts. In software, design patterns are not tied to any programming language, operating system, or computing platform. The classic textbook of software design patterns is Design Patterns: Elements of Reusable Object-Oriented Software by Erich Gamma, Richard Helm, Ralph Johnson, and John Vlissides.¹

Although the title of the classic text may suggest that design patterns are tied to object-oriented programming, they are, in fact, programming paradigm neutral. Objects may be the natural idiom for some design patterns, but most are equally applicable to other paradigms such as resource- or service-oriented programming.

Software design pattern books frequently use a set format, usually derived from the format used in the Design Patterns book and intended to make it easy for architects and developers to find and apply them to their work.

Enterprise IT architects have found the design pattern approach useful. The same enterprise architecture issues appear repeatedly, but enterprises vary widely, and other approaches to software reuse often are not sufficient. For example, code libraries for low-level programming constructs are efficient and widely used, saving millions of lines of code and promoting robust and efficient applications. Higher-level business libraries have been much less successful because business rules and conditions vary so widely. Business rules are often made up on the spot when business deals are negotiated. They address complex circumstances and business pressures. The resulting rules are often innovations that will not fit into established libraries. This wild variation can make business programming more difficult than other areas of software design.

Nevertheless, the same patterns and solutions recur on a high level, even in business. Reusable code often cannot address these recurring situations because the code may need to be radically different to address the specifics of innovative rules and complicated circumstances. However, these issues can be addressed with design patterns that describe solutions on a high level, which can be implemented with appropriate site-specific code. Developers can write appropriate code more quickly and with fewer defects by following guidance provided by patterns and solutions. Also, code that follows an established pattern is more easily and quickly understood by other programmers.

An example of a design pattern is a pattern to address idiosyncratic units of measure. Some businesses use units of measure that are not unique to their business. One business may group small parts in lots of 100. Other businesses group them on lots of a gross, 144. The obvious approach is to write code to deal with translation from a unit of 100 to another unit of a gross as they are needed to compare parts counted in gross to parts counted in hundreds. It is simple code and quickly written, often in the comparison function itself. However, a design pattern suggests a better approach: convert to a common unit and use only the common unit in code. Do all calculations and comparisons in the common unit. Experience has shown that this is a better approach because new units can be added without disturbing existing code. If a vendor, for whatever reason, decides to group parts into lots of 1,024 items, as computer engineers count bits on a memory chip, all that is necessary is to add a 1,024 conversion to and from the common unit. The rest of the code will then work properly with the new vendor.

This conversion design pattern can be implemented in any programming language on any platform. The pattern applies whenever incompatible units are used for quantities that must be combined. The discussion in the previous chapter on transforming disparate data sources to a single common model instead of transformations between each model is an example of this pattern. In fact, this design pattern is one of the keys to extensible integration.

A number of design patterns are useful in creating an integrated implementation of an IT system. Here I will discuss integration patterns on a higher level, focusing on the interaction between architectural design and business. The foundation for this focus is the design patterns that can be used to implement the interactions. There are excellent sources on software design patterns that are helpful to developers, but here I will be not be so concerned with implementation and more concerned how the patterns relate to business requirements.²

Integration Patterns

Small and uncomplicated systems integrate easily, but as an organization expands, simplicity evaporates. When the number of data sources to integrate goes up, integration may seem to get turned on its head. Patterns that were simple to implement become nightmares. Patterns that seemed to be complex wastes of time begin to beckon like oases in a desert.

The applications in some large enterprises can almost document the history of computing. Some mainframe applications may be decades old. The authors of some applications may have written in mainframe-era FORTRAN, COBOL, and PL/I. Distributed applications that are more recent are likely to be written in C, C++, C#, or Java. Application architectures also often combine Service-Oriented Architecture (SOA) based on SOAP or REST, open source J2EE applications, proprietary integration architectures and tools, and other homegrown or standard architectures. Applications themselves can consist of code written by the IT department (often with the authors long gone); others are off-the-shelf products, perhaps several releases out-of-date, and some are bespoke applications from external vendors. In extreme cases, the source code for old production applications can be lost or exist only in difficult-to-follow assembly language.

This mélange of applications contributes to the complexity that grows as organizations grow. Criticism is easy, but most organizations grow in this way for good reason. Separating the structure of the IT infrastructure and the organizational structure is difficult. Lines of business grow according to the laws of economics and commerce, not hardware and software architecture. Business units want software tailored perfectly to their own needs and maintained to meet their schedules and requirements. Business units clamor for divergent requirements that actually point to the same application, but recognizing this can be nearly impossible for both executive management and the IT department. At the time, separate systems, perhaps built in radically different ways, may seem to be the best solution. And of course, the adage “If it ain’t broke, don’t fix it” is always attractive, especially to those who do not or cannot anticipate the catastrophes waiting ahead. The long-term implication is chaos, but at every step, the decisions seem like the right thing to do.

The implementation of enterprise integration faces three fundamental technical challenges. Data must be transmitted from application to application. The data must be delivered to the applications that need it. Receivers must understand the data they receive. Often, data must be transformed into a form usable by the receiver. If data does not flow correctly and understandably, integration does not exist.

Data Transmission

The basis for any form of application integration is interapplication data transmission. The methods available to architects have increased with the expansion of technology. As the IT expands and the level of integration increases, different methods become appropriate.

Data Serialization

Data serialization is the process of transforming data structures into a form that can be stored or transmitted over a communications link. Serialization is a part in all data transmission. There are many possible formats for data transition. Wikipedia lists almost 30 well-known serialization formats.³ When proprietary and one-off programmer-written formats are included, the number must stretch into the thousands. The large number of formats is one of the challenges to integration.

One way to serialize data is to dump data structures exactly as they are laid out in memory into a byte stream and store it in a file or transmit it over a communications link. This is efficient: the dump is fast, the data is represented compactly, and little code is needed. However, memory dumps are seldom suitable for integration because the receiving application must know the exact, byte-for-byte structure of the transmitted data in order to use it. The byte-for-byte structure is dependent on the application data structure, the programming language used in the application, and the memory conventions of the hardware the application was running on. Even minor code changes to the dumping application can change the data structure. A change in the code standard or compiler can break the integration. Moving from one type of hardware processor to another can also break communications. The problems arising from this level of coupling are seldom offset by the speed and efficiency of a memory dump and load back into an identical data structure, but when speed is paramount, it may be the best choice.

Standard and commonly used formats, such as comma-separated values (CSV), Extensible Markup Language (XML), and JavaScript Object Notation (JSON), are an important step forward. These formats do not depend on implementation details or hardware. They all can be parsed using standard parsers. All three have rules for structuring data that do not depend on the internals of the sending or receiving application.

CSV is simple, both to parse and to write, but it conveys the least information. CSV requires each unit of data, datum if you will, to be written out in plain text and each separated from the next with a comma. CSV format does not say anything about content of the datum, except for some rules for handling commas that are in the datum and are not datum separators. Data type, format, and semantics are not touched.⁴ For the reading application to use the data, there must be metadata that describes the data. Often with CSV, there is no attempt to formalize the metadata. Typical older applications bury the metadata in engineering notes or it is passed along as verbal lore.

XML took an important step beyond CSV and CSV-like conventions. The XML authors attempted to develop a self-describing data format. Whether they succeeded or not is a matter of perspective.

Figure 9-1 shows clearly one advantage of CSV: it is more compact than XML. The CSV order uses 2 lines and 49 characters; the XML uses 18 lines and 297 characters.

Formatting the XML for human readability emphasizes the volume of the XML message, but it also highlights some of the advantages of XML. The XML, although somewhat redundant with end and beginning tags, conveys more information than CSV.

The XML tags show that the document is a list of orders and clarifies the meaning of each field. The first field in the CSV record could be a date, but it could also be a serial number or some other value. In the XML, it is unambiguously a date.

Furthermore, CSV is limited in the data structures it can convey. In this example, the lines of the order repeat: “fish hook” and “pole” are both items. There is no clue in the CSV file itself of the repeating fields. In fact, expressing repeating fields in this way is common, but it is a stretch. Most generic CSV parsers will be confused, and RFC 4180 does not mention the practice. If an application must have a structure more complex than a simple flat file, CSV is usually inadequate.

XML has few limitations on the complexity of the data structures it can express. With the addition of an XML schema (a formal grammar describing the hierarchy, type, and cardinality of each element in the XML document), an XML document can be automatically checked for validity, verifying both structure and data types. For instance, in Figure 9-1, if the order-date were not a date in an acceptable format or the quantity not a numerical type, the validator would reject the file. Where sophisticated security is necessary, a tight XML schema and validation can prevent some kinds of attacks.

The sophistication of XML is also its most important defect. It is relatively hard to learn and interpret; XML Schema more so. Many developers are impatient with the finicky verbiage required to write XML and complicated schema rules. Nonetheless, XML and XML Schema have been widely accepted in web services and SOA.

JSON is similar to XML and based on JavaScript. It is more compact and simpler than XML but not as expressive, a compromise between simple CSV and complex XML. JSON has supplanted XML in areas where simplicity is more important than support for more complex data structures. Also, there was, at least until recently, more standard tooling available for working with XML than JSON, but that is in flux.

However, neither XML nor JSON solves the semantic problem. Standard XML and JSON parsers have eliminated custom parsers for each application that passes data, but that is not all that is needed for integration. In Figure 9-1, “order” may seem clear, but a purchase order and a product order must be accounted for differently. Without some agreement on the exact meaning of the XML tags, they are only approximate and can be misinterpreted. Without additional information, often called metadata, the file cannot be properly interpreted and processed. Sometimes the metadata is expressed in formal documents, but the metadata is also often communicated casually. Casual communication could be as simple as a phone conversation between developers. Casual communication works, but it depends on human memory and note taking and breaks down as a system grows.

XML and JSON are not only used for formatting shared files. In fact, they are probably more used for messaging than anywhere else.⁵

Data Transformation

Data cannot always be consumed directly by receivers. A classic example is big-endian versus little-endian byte orders. Processors in the Intel x86 family store the most significant byte of a word in the highest address. Processors derived from the IBM mainframe architectures generally store the most significant byte of a word in the lowest address. For data from a big-endian system to be understood correctly on a little-endian system, the data must be transformed from big-endian to little-endian. Without transformation, the letters in a string of text are garbled from one processor architecture to another. Numeric representations and calculations are also mixed up. Until the data is transformed, it cannot be processed.

When integrating systems, transforming between big-endian and little-endian is usually a minor issue because the transformation is often built into hardware. In addition, the transformation is well-known, and a simple algorithm will perform the transformation correctly.

When transferring data and events between applications, transformation is much more difficult. It depends on the form the data is in at the source and the form expected at the target. It may involve semantics: one application may refer to a network interface controller (NIC), while another might call the same thing an Ethernet port. The transformation engine must be able to perform translations like this. Sometimes applications use different units to express the same value. In other cases, two applications may deal with the same virtual or physical object but refer to them in a different way; for example, one application may have a name for the object that identifies it, while another application may use a machine-readable serial number or some other designation. These and more issues have to be resolved in data transformation.

File Sharing

File sharing is generally easy to implement, and the coupling between the sender and receiver is loose. This is especially for files formatted with XML or JSON because parsing is standard, although semantic metadata is still an issue. In addition, the sender and receiver can write and read at any time. They do not depend on either being active, and as long as the format and metadata are clear, neither needs to understand anything of the other’s inner-workings.

File sharing is the probably the oldest form of integration. With a few exceptions for simple embedded controllers, all operating systems support some form of files: real or virtual contiguous sequences of bytes that are durable and later retrievable. The life cycle of a file is independent of applications that write or read the file. A file may continue to exist after the application that wrote it has terminated and may exist before an application that reads it has started. An application may write a file that another reads.

These are important properties for integration. An application that is sending data through a file does not need to know anything about the receiver of the data. The receiver also does not need to know about the application that wrote the file, although it must understand the layout of the file on some level. It has to be able to parse the file, that is, to break the file up into individual units of data as the sending application intended, and it must understand the semantics of the data.⁶ For example, if the producing program intended a data element to be a 4-byte string designating a user account code, the consuming program must not interpret it as a 32-bit integer containing a market closing price.

File sharing is used as a method of integration more often than it may appear. For example, when users download apps, they are participating in a form of file sharing. A file written in the development environment of the author is transferred to the device. Next, software on the device runs to install the app and prepare it for use. The downloaded file, or some portion of it, remains on the device and is executed when the app runs. This form of file sharing works because the data is extremely “bursty”—one large burst on the initial download and then an occasional patch or update usually separated by days or weeks. File sharing handles this kind of integration between the author’s development platform and devices well. Notice, however, that file sharing would absorb unacceptable bandwidth if updates occurred every minute. Other methods are better for activity that is more granular. For example, most apps have a separate communications channel for the app content that is faster and more continuous than file sharing.

Speed is a disadvantage to file sharing. Files are slow to read and write, and they are usually written and read only at relatively long intervals. Thus, file sharing is unsuitable for data that changes faster than a file can be written and read. File sharing is perfectly adequate for a financial application that, when the markets close, writes a file that is read by several other applications to prepare reports for the next morning. But a performance monitoring application that requires hundreds of samples a second and expects up to the second accuracy cannot rely on file sharing to collect data.

Another problem with file sharing is sometimes called impedance mismatch. When data from a slow application is combined with a faster application, stale information can be combined with fresh information yielding poor performance or misinformation. Attempting to combine data from a slow-paced financial application with a fast-moving performance monitor might be confusing. Combining up-to-the second performance measures with data that could be as much as 24 hours old could present a distorted picture.

For example, the cost of performance at one time of day could appear different than another even though they are the same because financial data gets less accurate as it ages. Therefore, integrators need to be cautious when combining data transfers of different speeds and frequency. This kind of mismatch can occur with any communications method, but the mismatch occurs commonly when the source is an older application that writes files rather than uses a more rapid communications method and the receiver is a more streamlined application.

Database Sharing

When relational databases were new, an enterprise with a single normalized relational database⁷ shared by all applications represented the holy grail of integration. It is no longer held in such esteem.

Such a system would have no data duplication, and all data would be equally accessible to all enterprise applications. Using database transaction–enforcement tools, the data would always be consistent. A single version of the data would be available to all applications at the same time. In theory, a clear corporate data dictionary and schema would make application design simple. Maintaining a single database should be easier than maintaining a database for each application.

Disadvantages

Shared databases are excellent in concept but hard to realize. The shared database strategy requires a comprehensive and deep agreement between groups in the enterprise on data formats and semantics. In practice, agreement on the contents and structure of a simple table such as a list of employees can require months of negotiation. In the end, no one is completely satisfied. Within months, the most dissatisfied are likely to start their own tracking in a spreadsheet or personal database. There is a good chance a department’s hand-tailored private data store will work better for them than the generic corporate data. The department will then be patted on the back for their improved performance, and the enterprise is on a slippery path back to individual data silos.

After the arduous and politically prickly task of designing such a database is done, new problems arise. If the enterprise is large, database administrators often have difficulty tuning the performance of a database implementation that serves many different applications. Different applications use the tables differently; an index design and a page size that improve performance for one application may degrade performance for another. Reaching a compromise that satisfies all the consumers is often difficult.

When multiple applications are sharing data, the opportunities for lock contention increase.⁸ In a heavily used transactional database, lock contention is often a significant performance issue. In less-complex and less-used databases, the classic write starvation problem is usually minor. Write starvation occurs when the opportunity to get a write lock is prevented for long periods by read lock holders. The write lock is on hold until all read locks are released. On heavily used databases, write starvation can be a substantial issue. When a table is used in a different way by different applications, write starvation becomes more frequent.

If the database mechanics challenges are met, maintenance and update can still be an issue. The downside of data consistency is close coupling. A common database can be a vicious form of close coupling between applications. Changing a common table (the user table is a good example again) requires the agreement of every application that uses the table. Sometimes adding a new column can be relatively painless, but something such as changing a string length can cause a calamitous series of hidden defects and, in some circumstances, security flaws.

Changes affecting several development groups are seldom politically acceptable. The requirement often has to be addressed with a kluge or workaround. Good design seldom wins over political expediency. When a change is carried out that requires a number of applications to examine or modify code, all the applications are held hostage by the last application to complete the change, which wreaks havoc with scheduling.

The shared database architecture for enterprise integration is elegant but is unfortunately unwieldy in practice, especially in enterprises that have a heterogeneous collection of applications and development groups that follow different coding and design practices.

Common database projects were frequent in the past. Unfortunately, many have failed or were eventually abandoned. In general, they have been replaced by federated approaches that preserve individual databases but maintain active lines of communication. When requirements are cohesive and coordinated and all the parties work exceptionally well together, a shared database can be an excellent solution. Even under those circumstances, business events such as mergers and acquisitions can be difficult to cope with. The upshot is that this strategy usually does not work in the long term and is difficult to implement well in the short term.

The approach works best where a relatively small group of applications share a common semantic and structural view of the data. Under these circumstances, a shared database can be an excellent architectural choice.

Messaging

IP-based messaging is the form of process communication that is used frequently today. Messages transmit data streams, events, and control from application to application, sometimes directly and often via an intermediary queue.

File and database sharing are fundamentally similar. Both push and pull data from a persistent repository. In each case, data is inserted or extracted from a repository. When the repository is a database, the interface is usually through a data access language such as Structured Query Language (SQL). When the repository is a file system, the interfaces are usually composed of operating-system read and write functions and a formatting and parsing facility. Communication takes place through the repository, and the connected applications never have to communicate directly.

Messaging is different. In its fundamental form, no repository is required. One application sends a message directly to another. The receiver reads the message immediately. The messaging pattern does not require storing the message at either end. Both the sender and the receiver can persistently store the message content, but there is no requirement, and storage is incidental to the pattern.

The message pattern has both advantages and disadvantages. It is flexible, fast, and scales up well. A robust messaging application implementation can handle massive quantities of data. Messaging systems can be loosely coupled. Well-designed messaging architectures permit applications to be added and removed easily.

A fundamental shortcoming of messaging is that pure messaging is unreliable without additional mechanisms. If the receiving application is not prepared to receive the message, the message is lost. The link between applications may fail. The message may be garbled in transmission. Neither the sending nor receiving application is likely to know of message failures. In practice, messaging is almost never implemented without infrastructure to manage its reliability. Messaging is still the communications method of choice for most recent applications but almost never without some form of support for greater reliability.

Today, file sharing often takes place via messaging in which entire files or segments of files are transferred via a messaging system. Applications communicate with shared databases via messaging. The same messaging mechanism that transmits messages from network node to network node will also transmit messages between processes running on the same node, giving the design great flexibility in distributing processes. Often, commercially available messaging services can be used. These are often reliable and efficient.

Basic TCP/IP and UDP/IP Communication

Transmission Control Protocol (TCP) and Internet Protocol (IP) combine into the most common messaging protocol in use on the Internet. IP is the addressing function that directs data packets from one source to another. The familiar “example.com” is an IP address translated into a human-readable form. TCP is a protocol that guarantees that data will arrive in the order sent and free of transmission errors.

TCP and IP are two layers in the stack of software that supports the Internet. Since they are usually often used together, TCP/IP has almost fused into a single word. TCP/IP is implemented on almost every hardware platform and on all devices connected to the Internet.

User Datagram Protocol (UDP) over IP is less common than TCP/IP but still important. TCP provides reliable messaging service, but reliability adds overhead, which can reduce performance. UDP does not guarantee either error-free transmission or that data will arrive in the order sent. Therefore, UDP is less reliable, but it is faster. In some situations, such as streaming data, UDP’s performance outweighs its unreliability.⁹

Synchronous vs. Asynchronous

Interprocess messaging is divided into synchronous messages and asynchronous messages. Both are used frequently, and both are useful in the right place. Synchrony and asynchrony are independent of protocol, although they are usually implemented on TCP/IP.

Synchronous Messaging

Synchronous messages are something like a telephone service. The caller speaks (sends a message) and waits for a reply. If the wait is too long, the caller hangs up and may or may not call again later. This method has pros and cons. The caller always knows the exact state of the communication. If they are calling to shut down their television cable service, they know the service is properly shut down when the service representative tells them it has been shut down.

Instant certainty that the service has ended may be important to some customers, but other customers might choose some uncertainty in return for avoiding a wait, especially when the service center is busy and the wait is long. Sending an e-mail is an alternative to calling. E-mail is a form of asynchronous messaging. The sender writes a message, presses the send button, and goes on to other business, occasionally checking their inbox for a reply to the e-mail. This is not as immediate and simple as a phone call, but the sender probably will get more work done.

A common form of synchronous messaging is the remote procedure call (RPC). An RPC looks similar to an ordinary function call, but a separate process, usually running on a remote machine, executes the body of the function. An ordinary function executes within the calling process. The program calls a function and gets back a reply, often a status. If the status is successful, the code moves on to the next step; if it is not successful, execution shifts to some sort of error handling. Within the calling code, function execution is just another step in the program, no different from executing an assignment or a line of computation. However, executing an RPC can take time—lots of time. From the viewpoint of a user of an interactive application, the program may go into suspended animation while it waits for the RPC to return.

RPCs are not the only form of synchronous messaging, but all synchronous messaging is challenged to deal with server processes that may be slow or have other problems. There are solutions. For example, the local part of the RPC may return with an error if the remote part does not reply within a time limit. A time limit, with many variations including automatic retries, helps, but waiting for the time limit and retries is still annoying. When there is no user interaction, the waiting usually shows up as performance degradation.

Waiting may or may not be acceptable. Users are accustomed to pausing for a few seconds when the cursor turns to an hourglass, but more than a few seconds calls for special handling with progress bars, expected completion times, and all the other ploys that have been devised to convince users to remain patient.

Synchronous messaging can be especially annoying when working with remote applications or servers. Network latency, the time it takes to communicate with the application running remotely, is always present and can be substantial. Networks are inherently unreliable, and there is always a risk that messages will be lost or impossible to deliver. Often, remote applications provide complex services that may take many steps and require remote interactions and long-running computation. If the application were deployed locally, the interaction might be broken into steps, but when network latency and unreliability intrude on each step, minimizing the number of steps is often advisable. This means that a calculation in one long running step is faster than breaking it into smaller steps with better user feedback. In this case and others, asynchronous messaging may be a better alternative.

Synchronous messaging is usually much easier to code and understand than asynchronous messaging. When the pauses that usually accompany a synchronous interface are negligible or not an issue, a synchronous interface may be a simple solution. Another case where synchronous messaging plays an important role is in real-time applications in which actions must be performed in a limited time. In real time, it is often better to abandon a synchronous call than to go on to another activity while waiting for an asynchronous reply. For instance, when streaming audio or video, it can be better to abandon an effort to get data, replace the missing data with an approximation, and move on. The approximation is a compromise, but it is better than a gap.

Asynchronous Messaging

Asynchronous messaging is an alternative to synchronous messaging. The sender of an asynchronous message does not wait for a reply. The sender hands off the message to the messaging subsystem, and the sender proceeds to its next task. The sender does not get their information sooner, but they, or the application, are able to move on to other processing instead of being blocked.

“Fire-and-forget” is a simple way to use asynchronous messaging. The message system fires off a message, hoping it will reach its target and succeed but not caring if the message reached its target or executed successfully. The code calling the message system does not wait for a reply. If this is acceptable, it can be fast and efficient. A classic regular postal service works this way. Mail is dropped into a mailbox without waiting for a confirmation. If the receiving post office cannot process the mail, it may be eventually returned to the sender, but return-to-sender mechanism is not expected to be prompt or reliable. If a postal service user wants to be sure their mail is received, the user upgrades to registered mail. E-mail is also largely a fire-and-forget service.¹⁰

Fire-and-forget is efficient and easy to implement, but situations may require a more sophisticated strategy. However, in software, less is often more. A fire-and-forget takes fewer resources and is usually executed more quickly than other strategies. For example, a fire-and-forget strategy is appropriate when messages are sent regularly and a missed or out-of-order message is not a significant error. This can be the case for event management systems.

Other asynchronous strategies include callbacks. When the sender’s message contains an address of some kind where a reply message can be sent and trigger some code, the address is called a callback. Using a callback strategy, the sender sends a request with a callback address. For example, a client might send a request to a storage server to load several gigabytes of data into storage from a local address. This may take several hours. When the load is complete, the storage server sends back a message to the sender’s callback address confirming the load and containing security credentials for the client to use when accessing the data. Code on the client side then processes this information and stores the credential for future use.

The callback strategy for asynchronous messaging is extensively used and can be powerful, but it is usually more complex to implement than may appear. When the remote system sends a message to the callback, the callback code begins to execute. Callback execution may be in an interrupt or in another thread. The callback code must be designed to be thread-safe or re-entrant if the results of the callback are to be processed in the context of the original message. For example, in the previous storage example, the data may have been stored for a single user. If the user is to get the credentials to access the stored data, the callback code must be aware of the user associated with the original message. The sender could include a user ID in the original message to be returned with the callback message, but that is not always possible. If the callback is designed to have access to the context of the sender of the message, the context will tell the callback which user gets the credentials. If several callback messages can start the same callback code, care must be taken to match the message to the correct context.

This can be complicated, and errors can be difficult to detect and fix. For some developers, the transition from straight-line coding to interrupt-driven code is difficult. In interrupt-driven code, execution is not from line to line; instead, execution may shift to another code block as external events occur.

Synchronous and asynchronous messaging are patterns for building effective communications systems, and TCP/IP is the protocol most of these communications systems are based on. However, there are higher levels of reliability that should be addressed by an enterprise integration system.

Assured Delivery

A reliable messaging service such as TCP/IP is not enough to support reliable integration. TCP/IP guarantees that data will be delivered in the order in which it was sent, and the content will be unchanged. It does not guarantee to the sending application that the data was received, nor does it guarantee to the receiving application that it has not missed messages that it was supposed to receive.

The limitations of TCP/IP are limitations imposed by its level in the data transmission stack. For successful messaging, the problem must be addressed on a higher level. A classic example is a production machine controller that sends out a critical warning message to a service desk that will automatically generate an issue. TCP/IP guarantees that the information will be delivered accurately, but what happens if the service desk receiver is down when the message is delivered? The message is lost, the issue is not generated, and a catastrophic failure ensues, which could have been prevented if the issue were written. The failing machine could keep resending the message until the service desk sends an acknowledgment, but a simple controller may not be able to accept an acknowledgment or resend and may even have failed itself. Alternatively, a messaging service could queue messages until the receiver was ready to receive them. By delegating assured delivery to the message service, the system becomes less coupled because the service desk does not need code to send an acknowledgment tailored to the receiver nor does the sending application have to manage sending the message to match the service desk requirements.

Messaging services frequently have assured delivery options. A robust messaging service with assured delivery options is often the most scalable and reliable choice for integration messaging.

Data Routing

Data must not only be transmitted efficiently but must also be routed to the correct recipients. One form of routing involves inspecting the contents of messages and determining the recipient from the content. Another common routing activity is to reduce a swarm of identical, similar, or logically related messages to a single message. Sometimes messages are discarded when they are not useful. In other cases, messages may be broken up into chunks that are of interest to different recipients. In some messaging architectures, all messages are directed to all recipients, and processing logic in the recipient filters the input down to the messages the recipient can use. This arrangement has the advantage that the logic to determine the recipient’s input is with the recipient, not spread through the routing system; however, it increases the traffic into the recipient. A combination of complex routing and intelligent recipients is an interesting compromise.

No matter how simple or complex the routing, when using TCP/IP to transmit data, the sender must know the IP address of the receiver or the domain name of receiver and resolve the name via the Domain Name System (DNS).

Hard-Coded Addresses

The simplest way to access a receiver’s address is to hard-code the address directly into the sender’s communications stack. This certainly works, but it has drawbacks. Changing the receiver involves code changes. If the sender’s code is compiled, the code has to be recompiled. Even if the code is interpreted rather than compiled, good practice requires the code changes be recorded into the software configuration system. Many sites will require at least a partial rebuild of the system to ensure consistency with the configuration whenever the content of the software configuration changes. In addition, the new software must be tested, and there is always the risk of failure, taking you back to square one. This is a lot of work and risk for a simple change.

Despite the drawbacks, hard-coded addresses are a simple solution for a small system that does not change often. Hard-coding generally implies a point-to-point system where routing and addresses never change. With only a few integrated applications, this system can be adequate. There is an added requirement that IP addresses do not change. This puts a constraint on changing the hardware, but where software and hardware seldom change, the simplicity of hard-coding may outweigh its inflexibility. Larger and more dynamic IT installations have to develop systems that are more easily changed.

Configuration Mapping Files

A simple solution is to provide a configuration-mapping file that each application reads. The file can map a single sender to each of its receivers, or the file can contain mappings between several senders and their receivers. When a sending application is ready to send to a receiver, it resorts to information from the configuration-mapping file to find the correct IP address.

A configuration-mapping file avoids the pain and risk of hard-coding, but it still has limitations. Keeping configuration files up-to-date can be an issue, especially when the system is distributed over many network nodes. In that case, when the configuration changes, files on several hosts may have to change. This is an added administration burden and an opportunity for a failure. Placing mappings for more than one sender into one file that maps senders to their receivers reduces the number of mapping files to maintain, but a single file means sending applications that use the file must have similar code to read the mapping file, which may not be possible with legacy and third-party applications. In a large and heterogeneous system, the mixture of multi-application configuration files, single-application configuration files, hard-coding, and other mechanisms can become an error-prone tangle. Forgotten configuration files can be a nagging problem for system administrators. Avoiding them is sometimes a significant achievement.

In addition, to avoid the overhead of repeatedly reading the same file, applications often are coded to read configuration files into memory at startup. In that case, some means must exist to cause the configuration files to be reloaded when they change. Sometimes a brute-force stop and restart is enough, but often the refresh must occur without pausing production.

The file refresh and reload issue can be addressed in several ways. One approach is to deal with configuration files in the same way that patches and updates are managed by Windows or Linux. These operating systems have code installed that responds to files that are downloaded from central repositories. The installed code manages the updates and patches, making required settings and signaling applications to reload or the system to reboot. These approaches are effective, but they are also major engineering efforts and may be outside the scope of an IT department to develop for their own system. Also, configuration changes can be relatively slow. This may be an issue when systems change dynamically in a virtualized system. In a virtual system, the sources and targets for integration messages can undergo constant and rapid change. Continually changing and redistributing files can become impractical.

Publication-Subscription

Other approaches have different advantages. One of the most common integration routing architectures is called publication-subscription (or pub/sub).

Pub/sub is modeled after a magazine clearinghouse that distributes several different magazines. When the customer initiates a subscription with the clearinghouse, they choose the magazines they want from the magazines the clearinghouse offers. The customer contacts the clearinghouse, not the individual magazines. Extending the metaphor, a magazine that has joined the clearinghouse does not deal directly with subscribers. The clearinghouse takes care of the overhead of directing the magazines to the right customers, and the customer deals only with the clearinghouse, not each individual magazine. Thus, magazine publishers don’t need to know the addresses of their customers, and the customer does not need to know the addresses of the magazines. Both publisher and subscriber need to know only the single address of the clearinghouse.

Pub/sub message routing works similarly. Usually, a broker process acts as the clearinghouse.¹¹ Applications register with the broker, offering messages to which may be subscribed to and subscribing to messages that interest them.

The details of the registering with a broker can be challenging. In a simple form, an application simply registers its address and a unique identifier for each type of message it has to send by sending a registration message to the broker at a known address. Subscribing applications register their address and the messages they want to receive. The subscribers must know the identifiers of the types of message they want to receive. They must also know how to process the associated type of message. In this simple scenario, applications must still know some details of the senders and their messages in order to use the messages, but address management is left to the broker. The only address the applications need to know is the address of the broker.

Pub/sub is occasionally criticized for semantically coupling senders and receivers. Ideally, decoupled applications should be able to register for messages without knowing anything about the source of the message, and the senders of messages should not have to consider their receivers. This scenario is attractive: applications could register with the pub/sub broker and immediately begin receiving all the relevant messages, and other applications would begin to receive relevant messages from the new application. Ideally, when an application adds, removes, or updates services, the system would automatically adjust to the new situation. Unfortunately, this scenario is more aspirational than real.

However, the aspirations are approachable. For example, consider MIME types as they are used with HTTP. MIME types, also called Internet media types, were developed to describe data content of files attached to e-mail and are often associated with a file extension. MIME types are registered with the Internet Assigned Numbers Authority (IANA), the same authority that manages domain names and IP addresses. Using the IANA registry, anyone can see an authoritative definition of the type.

MIME types appear in the headers of HTTP packets. The MIME type tells the receiver of an HTTP packet about the format of the payload of the HTTP packet it receives. MIME types make it possible for web browsers to know to parse an HTML document as HTML rather than some other format like pure text.

MIME types vary widely. Some are general, and an application can expect almost anything in the content, such as an HTML type, which is fine for browsers displaying HTML. As HTTP has grown to be used for more than exchanging documents and has become a protocol for exchanging information between applications, MIME types have become useful for specifying finer-grained types.

In some ways, the World Wide Web¹² is an ideal application integration system. Individuals and web sites that know nothing about each other use published MIME types and a standard message format and protocol to exchange information. This is close to complete decoupling of receiver and sender and represents one pattern for decoupled pub/sub integration. The pattern has two essential elements: a shared protocol like HTTP and public message types.

Although setting up a pub/sub broker requires investment, it is scalable and more easily reconfigurable. The broker can be either the hub of a spoke and hub system or a component on a message bus. For many sites, hooking applications up to a broker is a major accomplishment and a great improvement because it shifts address management to a central broker that is relatively easily managed. The applications still may be semantically coupled, but using standards like MIME types reduces coupling.

Sites that go further with public message types and high-level standard protocols may gain even greater benefits. SOA is a step toward easier integration because they are standard architectures, protocols, and published message types.

Data Translation

If everyone spoke the same language, communication would be easier, though perhaps not as rich. However, supporters of universal languages such as Esperanto and Interlingua argue for the value of a neutral language more or less easily understood by everyone. A similar situation occurs in enterprise application integration. There are the equivalents of international languages; Distributed Management Task Force Common Information Model¹³ is one example that is widely acknowledged.

As long as an application communicates with a small group of other applications whose developers and administrators are all familiar with the messages and models of the entire group, this works. However, as new applications are added to the mix and personnel changes, fewer and fewer technicians are prepared to understand communications between the applications.

At that point, the value of a fully decoupled sender and receiver architecture becomes evident. Unfortunately, the pub/sub system described earlier may be impossible to implement under time and budget constraints.

This leads to data translation. A data translator takes a message generated by one application and translates it into a form usable by another application. The translation can translate formats. For example, a translator may transform a file in CSV form to XML. The translation can be complex, changing terminology, performing calculations, and executing business logic to convert data prepared for one purpose into a form suitable for a different purpose.

At its extreme, each pair of applications may need a special data translator, leading to a situation, which should be familiar to readers by now, where the number of translators increases exponentially with the addition of each new application. Even with a pub/sub system that manages application addresses and message types, each application may need an array of translators to deal with each message type it receives. The application may not know where the message is coming from, but it still must know how to decipher the message.

There are models that can help with translation. Consider that an international language like Esperanto can still be useful when only a few speakers use it. In a room full of people who speak different languages, a few Esperanto speakers can act as mediators. If one of the Esperanto speakers speaks language A and another Esperanto speaker speaks language B, the A speakers and B speakers can communicate via the two Esperanto speakers.

Chapter 7 discussed the CA Unified Service Model and its use as an intermediary between data sources and targets. This approach can be generalized in an integration model. Instead of building a data translator for each application pair, each data source converts its data to the agreed-upon “Esperanto” form. Each data receiver builds a translator from Esperanto to its native internal form. This plan eliminates the unique translators for each pair of applications.

This pattern isolates the individual applications from the universal model, which reduces the amount of reworking of each application in order to participate in the integration system.

There are also some downsides. The central model has to be comprehensive enough to serve the needs of all applications, which is not an easy accomplishment. If the model is not comprehensive enough, someone has to build a one-off specialized translator or the model has to expand. Deciding which route to take is delicate. A one-off translator may be the best solution if the requirements are truly unique. There is little point in expanding an entire model for a one-time event. On the other hand, one-time events are often not as unique as they were initially thought to be.

Conclusion

Designing data transmission, transformation, routing, and translation systems requires knowledge of the IT domain and the content of the applications in the domain as well as the technical advantages of various integration patterns and how to implement the patterns in hardware and software.

This is no small challenge. Hardware and software alternatives rise and fall in prominence continually. Enterprises that can accurately predict what their business will be doing five years from now are rare. The big picture is only one thing that an architect designing an integration system must understand. What will be the direction of application design in the domain of the enterprise? What kind of automation will new lines of business require? How will automation change in current lines of business? There are probably no answers to any of these questions, only better or worse guesses. That is no reason for an architect not to try. Architects are more likely to be right if they guess the future and guess wrong than they are to be right if they flail in the dark.

¹Eric Gamma, Richard Helm, Ralph Johnson, John Vlissides, Design Patterns Elements of Reusable Object-Oriented Software, New York: Addison-Wesley, 1994.

²There are a number of excellent references on design patterns for enterprise and cloud applications. Martin Fowler, “Patterns of Enterprise Application Architecture,” New York: Addison-Wesley, 2003, is an excellent starting point. Gregor Hohpe and Bobby Wolfe, Enterprise Integration Patterns Designing, Building, and Deploying Messaging Solutions, New York: Addison-Wesley, 2004, takes off where Fowler leaves off. Bill Wilder, Cloud Architecture Patterns, O’Reilly: Cambridge, 2012, carries design patterns into the cloud era. Although Cloud Architecture Patterns draws on Microsoft Azure for examples, it is much more than an Azure handbook.

³Wikipedia. “Comparison of data serialization formats.” February 8, 2015. http://en.wikipedia.org/wiki/Comparison_of_data_serialization_formats.

⁴CSV is mostly an informal standard. There is an Internet Engineering Task Force (IETF) request for comment, RFC 4180, http://tools.ietf.org/html/rfc4180, but it is informational, not a recommendation. CSV is so common and so simple that most developers don’t think about standards for CSV.

⁵XML, JSON, and Yet Another Markup Language (YAML). All perform similar roles. XML is the oldest and expressive but complex and relatively hard to work with. It is used in many enterprise systems and has an extensive array of tooling for parsing, generation, validation, and translation. However, XML is not as popular as it once was. JSON is easy to understand and works naturally with JavaScript. Simplicity has made JSON the language of choice for many new applications, especially those based on REST. YAML is relatively new. It has the same basic syntax as JSON, but it has features that fill in many of the deficiencies of JSON versus XML. YAML may be the compromise of the future.

⁶Simple file locking is an extreme example of file sharing. The application locking the file or the operating system writes a file with the same name but a different extension as the file to be locked. This extra file signals other applications that the file is not to be touched until the lock file disappears. The existence of the file conveys the essential information. The file is empty or the contents are irrelevant in the simplest case.

⁷Relational database normalization is a difficult subject. A fully normalized database contains no duplicated information, and each row in the database is independent following rules derived from relational algebra. Replacing good database design with rote normalization can lead to poor performance and complex schemas. Consequently, database normalization is not as well thought of as it was in the past.

⁸Transactional database locking is complex, and several methods exist. For this simplified discussion, read locks block changes but not views, and write locks block both changing and viewing. A write lock cannot be taken when a read lock is in place. A row lock may be either a read or write lock and locks a single row. A table lock is like a row lock but locks an entire table. A database lock locks all tables. Some databases can hold only a limited number of locks of any kind. Locks are promoted to reduce the total number of locks.

⁹It’s worth noting that implementing checks and responses in the communicating applications can render an unreliable protocol such as UDP reliable. Sometimes this is an effective tactic.

¹⁰Under the covers, the message exchanges between e-mail servers are more complicated than “fire and forget,” but after the user clicks the send button, the e-mail is out of their control.

¹¹A separate broker process may not be necessary. The broker function can be distributed if there is a means to distribute all the subscription to each integrated application, and each application has code to use the subscription information. This avoids the broker becoming a bottleneck. Alternatives include broker clones that distribute the load.

¹²Strictly speaking, the World Wide Web is a sprawling information system that runs on the Internet. The Internet is the interconnected mass of networks that enables widespread communication. The components of the World Wide Web include browsers such as Internet Explorer, Firefox, and Chrome; HTTP servers like Apache and Internet Information Services (IIS); and a myriad of web sites.

¹³See www.dmtf.org/standards/cim.