The Challenge of Siloed Applications

The challenges presented by siloed systems start innocently enough. The problems with siloed applications begin as a minor irritation or inconvenience. But over time, the problems escalate from inconvenience to calamity. And there is never any easing of the problems. It is eternal escalation.

Fig. 5.1.1 depicts the siloed applications that management wakes up to one day.

So, what are the problems that eternally escalate in the face of siloed applications? There are many.

• Maintenance. Once an application is built, it needs constant maintenance. And over time, the need for maintenance never de-escalates. The completion of one maintenance project spawns three other new projects for maintenance. And as there are more applications that arise, the need for maintenance grows exponentially.
• Integrity of data. It is one thing to have data. It is quite another thing to have data that can be believed. Multiple siloed applications breed disintegrity of data. Marketing says we are losing money. Finance says we are breaking even. Sales says we are making money. Who to believe is like drawing straws. One day, we believe one source. The next day, we believe the next source. And nobody is saying the same thing as anyone else. Who to believe is a constant political battle within the walls of the organization. The problem is not in having data. There is plenty of that. Instead, the issue becomes which data to actually believe. Making good corporate decisions becomes difficult in the face of data that cannot be believed.

Fig. 5.1.2 shows that there are multiple versions of the same data within different siloed applications.

Not only do siloed applications have conflicting values of information, but also trying to resolve the issues across the different applications is an intractable problem. There are a multiplicity of reasons why the unbelievability of data across siloed applications presents an intractable problem. But the single greatest reason for the intractability stems from the fact that data cannot be meaningfully shared from one siloed application to the next.

Fig. 5.1.3 shows this issue.

Certainly, data can be shared from a mechanical standpoint. But merely passing data from one application to another does no good when the data mean something different in each application. As a simple example of why data are not interchangeable among applications, consider the following example. Suppose that there are three siloed applications—application A, application B, and application C.

Suppose that all three applications have a field of data—amount of sale—and in each amount of sale is a dollar value. It seems like it would be easy enough to exchange the values between the applications. But a closer examination turns up the fact that application A measures American dollars, application B measures Australian dollars, and application C measures Canadian dollars. At first glance, there is consistency among the applications. But a closer examination shows that it would be a very misleading thing to do to merely start exchanging dollar values between the different applications. The data are not the same at all.

Unfortunately, the differences between applications run much more deeply than a conversion of money. There are many other reasons why merely transferring data values from one application to the next is a dangerous thing to do.

Older technology can become an issue. When an application is built, it is built in the technology that is available at the time of development. Unfortunately, the siloed application often times outlives the technology it was built in. Because a siloed application is extremely difficult to change, the siloed application becomes “trapped” inside older technology.

There are then many reasons why siloed applications in corporations become an ever-increasing vexation.

Building Siloed Applications

So, exactly how did corporations get into the dilemma that they find themselves in when it comes to their siloed applications?

The sojourn to siloed systems starts innocently enough. Simply stated, developers were building applications in what they thought were the best methods and approaches for development at the time.

One of the fundamentals of application development was the tenet that applications are to be built from end user requirements. That notion sounds simple and straightforward. But time has shown that this simplistic approach has some major shortcomings.

Fig. 5.1.4 depicts this simple notion—that applications are built based on end user requirements.

Indeed, whole books and methodologies are built on this simple notion. And to an extent, these books and methodologies are correct. Applications must be based on end user requirements.

However, there is a major flaw with this line of thinking. The flaw is that only the direct user of the systems is used to determine the requirements. The thinking is that if ALL the people—direct users and indirect users of the application—are considered, it will take a very long amount of time to build the application. Therefore, when gathering the requirements of the system, only the direct users of the system are considered. In doing so, the gathering of requirements takes a finite amount of time.

So, who are these indirect users of the application? Typically, the indirect end user of the application includes (but is not limited to) marketing, sales, finance, and accounting.

And when the application was built and just being installed, the direct end user was quite pleased. But shortly thereafter, the dissatisfaction begins to come when the indirect users of the system began to voice their complaint. Some of the things the indirect user of the data complained about were as follows:

• Data definitions that were different from those used by the developer
• Accessibility to the data found in the application
• Difficulty of access to the data found in the application
• Accuracy of the data found in the application
• Timeliness of access to the data found in the application
• Organization of the data found in the application
• And so forth

The list of complaints of the indirect end user was long.

Because the data were so hard to get to and so foreign to the thinking of the indirect end user, the indirect end user sets out to build THEIR own application. And in doing so, the indirect end user escalated the issues evolving around siloed systems. Now, there were even MORE siloed applications.

Fig. 5.1.5 shows that—once built—the application did not solve any of the demands for data of the indirect users.

And the hunger for data and information spawned more applications, and each of the new applications exaggerated the dissatisfaction with siloed applications.

What Does a Siloed Application Look Like?

So, what exactly do these applications that turned into silos actually look like? What are some characteristics of these applications?

Most of these siloed applications were the very first applications that were built or were otherwise acquired by the corporation. As a result, these were the very first applications that inhabited the corporation.

In many cases, these applications were the applications that directly connected the corporation to its customers. Typical of these applications were ATM processing, bank teller processing, airline reservation processing, and so forth. These early applications were critical because they were the day-to-day face of the corporation to its customer base.

These early applications had a direct effect on the day-to-day business of the corporation. For example, when the bank teller system went down, the bank had to cease doing business until the system came back up.

Current Valued Data

One of the common expectations was that the data found in the application were what can be called “current valued data.”

Fig. 5.1.6 shows current valued data.

Current valued data are data that are accurate as of the moment of access. In order to achieve this high degree of accuracy, the application must support online transaction processing.

As a simple example of current value data, consider the account balance for a husband and a wife. At 8:00 a.m., their account balance has $5000. At 9:15 a.m., the wife does an ATM withdrawal of $500. At 9:15 a.m. and 5 seconds, the account balance is corrected to $4500.

The husband checks the account balance at 10:00 a.m. and sees that the account balance is $4500. The husband withdraws $175 as of 10:01 a.m. At 10:01 a.m. and 5 seconds, the account now has $4325.

At 4:15 p.m., the wife deposits a check into the account for $2000. As of 4:15 p.m. and 5 seconds, the account balance is now $6325.

At any point in the day, the husband, the wife, and the bank can check the account balance and see what the balance in the account is. The account balance is accurate as of the moment of access to the data.

Current value data are data that are accurate as of the moment of access. Applications that interact directly with the customer are typically populated with current value data.

The converse of current value data is data that are not accurate as of the moment of access. As a simple example of noncurrent value data, consider the stock market where values are captured at the end of the trading day. Suppose there is an application that captures data as of the end of a trading day.

At 9:00 a.m., you look at your favorite stock. You found that its closing value the day before was $76.10. However, the market has been open for several hours now. The stock may well be trading higher or lower than $76.10. But your stock value won’t be updated until the end of the trading day.

Minimal Historical Data

Another characteristic of the data found in siloed applications is that there is relatively little historical data found in the application. If there is any amount of historical information found in the application, it is summarized.

There is a very practical reason why there is little historical data found in an application. The reason for the paucity of historical data is the need for efficient transactional performance. In a customer facing application, it is typical to do transactions. In order to execute transactions quickly, it is necessary to wade through and process the minimum amount of data. When an application stores massive amounts of historical data, the efficiency of transaction performance is impeded. Therefore, online transaction processing applications shed as much data as possible as soon as possible. In doing so, they optimize the performance of the system.

In order to understand this principle, consider your bank account in the bank. Is it reasonable for you to find the current account balance? Yes, of course it is. Is it reasonable for you to go and look up a transaction that you made last week? The answer is yes, of course. Now, suppose you are going through an IRS audit. Suppose you need to find a check you wrote 5 years ago. Is it reasonable that the 5-year-old check is in the online processing portion of your database? No, it is not reasonable. In order to find your 5-year-old check, the bank will have to look through audit and archive records.

When it comes to detail, applications typically store a month's worth of data or maybe even a quarter's worth of data depending on the nature of the business being accommodated. Beyond that, the data are stored elsewhere.

Fig. 5.1.7 shows that online transaction processing applications focus on very current information.

High Availability

Another characteristic of siloed applications is the fact that application data often have the need for a high degree of availability. For systems that are customer facing, anytime the system is down and dysfunctional, the customer grows dissatisfied with the company whose business the application was built for. For example, suppose your ATM machine is down. You cannot transact your activity, and you grow dissatisfied with the bank that owns and manages the ATM machine. As a consequence, applications are up and running as frequently as possible. Applications try to achieve 100% uptime. The reality is that no system is up and available 100% of the time. That nevertheless is the goal.

Fig. 5.1.8 shows that applications try to achieve a 100% availability.

Overlap Between Siloed Applications

Another consequence of the building of siloed applications is that there is a high degree of overlap—redundancy—among the many siloed applications. Because the applications are built for individual requirements, it is almost inevitable that there be significant data and process overlap between the different siloed applications.

Fig. 5.1.9 shows the overlap between the different applications.

Frozen Business Requirements

Another common characteristic of siloed applications is that the siloed applications have their business requirements “frozen in time.” The siloed applications are built around requirements of the business that are known as of one moment in time. Once the application is built, the siloed application is very difficult to change. Unfortunately, business conditions change whether the underlying application changes or not. Business change is simply a fact of life.

The result is that one day, the business wakes up and finds that they have a different set of business requirements than those represented by the siloed application. The siloed application represents business requirements of a previous decade. But the siloed application is so difficult to change that new, more modern business requirements are not satisfied by the siloed application.

For this reason, siloed applications are said to be “frozen in time” when it comes to the business that is represented by the siloed application.

Fig. 5.1.10 shows a siloed application that has been frozen in time.

Dismantling Siloed Applications

Unfortunately for most organizations, dismantling the older siloed application is not an option.

Fig. 5.1.11 shows that dismantling the siloed application is not something that is even considered.