We can say what things used to be like, what they are like, and also what they will be like. These statements we can make are statements about, respectively, the past, the present and the future. In a table in a database, each row makes one such statement. In conventional tables, however, the only rows are ones that make statements about the present.

These things we say represent what we claim is true. Of course, as we saw in Chapter 12, we can equally well say that they represent what we accept as true, agree is true, assent to or assert as true, or believe, know or think is true. For now, we'll just call them our truth claims, or simply our claims, about the statements made by rows in our tables.

Besides what we currently claim is true, there are also claims that we once made but are no longer willing to make. These are statements that, based on our current understanding of things, are not true, or should no longer be considered as reliable sources of information. It is also the case that we may have statements—whether about the past, the present or the future—that we are not yet willing to claim are true, but which nonetheless are “works in progress” that we intend to complete and that, at that time, we will be willing to claim are true. Or perhaps they are complete, and we are pretty certain that they are correct, but we are waiting on a business decision-maker to review them and approve them for release as current assertions. The former is a set of transactions about to be applied to the database. The latter is a set of data in a staging area, either waiting for additional work to be performed on it, or waiting for review and approval.

So if statements may be about what things were, are or will be like, and claims about statements may have once been made and later repudiated, or be current claims, or be claims that we are not yet willing to make but might at some time in the future be willing to make, then the intersection of facts and claims creates a matrix of nine temporal combinations. That matrix is shown in Figure 13.3. ²

The reason we are interested in the intersection of facts and claims is that rows in database tables are both. All rows in database tables represent factual claims. One aspect of the row is that it represents a statement of fact. The other aspect is that it represents a claim that that statement of fact is, in fact, true. This is just as true of conventional tables as it is of asserted version tables.

When dealing with periods of time, as we are, the past includes all and only those periods of time which end before Now(). The future includes all and only those periods of time which begin after Now(). The present includes all and only those periods of time which include Now().

Every row in a bi-temporal table is tagged with two periods of time, which we call assertion time and effective time. Consequently, every row falls into one of these nine categories. Conventional tables contain rows which exist in only one of these nine temporal combinations. They are rows which represent current claims about what things are currently like. But since conventional tables do not contain any of the other eight categories of rows, their rows don't need explicit time periods to distinguish them from rows in those other categories. And in conventional tables, of course, they don't have them.

Both the assertion and the effective time periods of conventional rows are co-extensive with their physical presence in their tables. They begin to be asserted, and also go into effect, when they are created; and they remain asserted, and also remain in effect, until they are deleted. They don't keep track of history because they aren't interested in it. They don't distinguish updates which correct mistakes in data from updates which keep data current with a changing reality, ultimately because the business doesn't notice the difference, or is willing to tolerate the ambiguity in the data.

So conventional tables, all in all, are a poor kind of thing. They do less than they could, and less than the business needs them to do. They overwrite history. They don't distinguish between correcting mistakes and making changes to keep up with a changing world. And these conventional tables, as we all know, make up the vast majority of all persistent object tables managed by IT departments.

We put up with tables like these because the IT profession isn't yet aware that there is an alternative and because, by dint of hard work, we can make up for the shortcomings of these tables. Data which falls into one of the other eight categories can usually be found somewhere, or reconstructed from data that can be found somewhere. If all else fails, DBMS archives and backups, and their associated transaction logs, will usually enable us to recreate any state that the database has been in. They will allow us to re-present six of the nine temporal categories we have identified. ³

The three categories that cannot be re-presented from backups and logfiles are the three categories of future claims—things we are going to make our databases say (unless we change our minds) about what things once were like, or are like now, or may be like in the future. Future claims often start out as scribbled notes on someone's desk. But once inside the machine, they exist in transaction datasets, in collections of data that are intended, at some time or other, to be applied to the database and become currently asserted data.

In the previous chapter, we called the eight categories of data which are not current claims about the present, pipeline datasets, collections of data that exist at various points along the pipelines leading into production tables or leading out from them. As physically separate from those production tables, these collections of data are generally not immediately available for business use. Usually, IT technical personnel must do some work on these physical files or tables before a business user can query them for information.

This takes time, and until the work is complete, the information is not available. By the time the work is complete, the business value of the information may be much reduced. This work also has its costs in terms of how much time those technicians must spend to prepare that data to be queried. In addition, even without special requests for information in them, these physical datasets, taken together, constitute a significant management cost for IT.

With multiple points of rest in the pipelines leading into and out of production database tables, there are multiple points at which data can be lost. For example, data can be accidentally deleted before any copies are made. For datasets in the inflow pipelines, and which have not yet made it into the database itself, the only recourse for lost data is to reacquire or recreate the data. If prior datasets in the pipeline have already been legitimately deleted (legitimately because the data had successfully made it to the next downstream point), then we may have to go all the way back to the original point at which the data was first acquired or created. This can impose significant delays in getting the data to its consumers, and significant costs in reacquiring or recreating it and in moving it, for a second time, down the pipeline. And this risk is quite real because, prior to making it into the database, the backups and logfiles which protect data once it has reached the DBMS are not yet available.

By internalizing these datasets within the production tables whose data they contain, we eliminate the costs of managing them, including the costs of recovering from mistakes made in managing them. We now turn to the task of re-presenting what were physically distinct managed objects, external to production tables. We re-present them as queryable objects, showing how queries can produce result sets containing exactly the data that would have been in those physical datasets, had we not internalized them.

We emphasize once more that most business queries for temporal data will not focus on data from a single one of these eight internalized pipeline datasets. Together with currently asserted current data, these eight other categories of temporal data constitute a partitioning of all bi-temporal data. Like the Allen relationship queries we will discuss in the next chapter, we focus on these queries in spite of the fact that they are not real-world business queries. We focus on them because, as a set, they are guaranteed to be complete. If these eight categories of pipeline datasets can be internalized, then we can be certain that any real-world business dataset—one destined to update a production table, or one derived from a production table—can also be internalized. In the next chapter, once we have seen that any Allen relationship against asserted version data can be expressed in a query, we will be similarly certain that any query whatsoever can be expressed against asserted version tables.

In each case, we will illustrate these queries in the context of CREATE VIEW statements. From the point of view of the semantics involved, there is no difference between direct queries and SQL VIEW statements. But actual VIEW statements lend a little more substance to the notion of re-presenting internalized pipeline datasets as queryable objects.

The cost of managing physical pipeline datasets is high. This cost is seldom discussed because it is universally thought to be just an inevitable cost of doing business. Bringing down this cost is a matter of doing all those various things that IT management has done for decades, and continues to do. Quality control procedures are put in place so errors don't creep into our databases and later have to be backed out. The platform costs of storing, transforming, and moving data into and out of pipeline datasets are controlled by minimizing redundancy, and by moving datasets up and down the storage hierarchy. Software that sets up and runs production schedules minimizes the human costs of scheduling work involving these pipeline datasets.

But the work of managing pipeline datasets is tedious. And whenever the management of these datasets is a one-off kind of thing, i.e. whenever the development group has to manage these datasets rather than the IT Operations group that handles scheduled maintenance, errors in managing them are not uncommon.

Asserted Versioning does not offer a way to more efficiently manage pipeline datasets. It offers a way to eliminate them and, consequently, eliminate the totality of their management costs! There will always be some circumstances in which data must be manipulated in external pipeline datasets. But these can become the exception rather than the rule.

In place of these pipeline datasets, Asserted Versioning stores the information contained in those pipeline datasets internally, within the production tables that are their sources and destinations. Pending transactions can be stored within the production tables themselves. Posted transactions can be, too. Data staging areas can also exist as semantically distinct sets of rows, physically contained within production tables. Pipeline datasets, then, cease to exist as distinct physical objects. They become virtualized, as semantically distinct collections of rows all physically existing within the same tables, re-presented in different views.

We may think that the principal cost elimination benefit of internalizing pipeline datasets is that it reduces the number of distinct datasets that programs, SQL and production scheduling software have to identify and manage. This is a reduction in the cost of the mechanics of pipeline datasets. Instead of assembling data from multiple tables, it already exists all in one place.

But the more significant cost reduction has to do with the semantics of pipeline datasets. With all data about the same things in the same place, we will, all of us, find all of it when we go looking for it. The most junior member of the business community will find the same set of data for his queries that the most senior member does. There won't be differences in completeness of the source data, or quality of that data, as there so often are in today's business world and today's collections of business data.

When we need any of this data, we won't have to go looking for it. All of the data about what we once thought was true, or what we currently think is true, or what we are not yet willing to assert is true, will be available by simply changing the assertion point-in-time selection criterion on views and queries. By changing that predicate in a WHERE clause to a past point in assertion time, we will be able to access the internalized re-presentation of posted transactions. By changing the predicate to a future point in time, we will be able to access the internalized re-presentation of pending transactions.

By the same token, we will be able to access historical data about what things used to be like from the same table that contains data about what they are like right now, and that may also contain data about what those things are going to be like sometime in the future. Again, it will be as easy as changing a predicate in a WHERE clause.

Glossary entries whose definitions form strong inter-dependencies are grouped together in the following list. The same glossary entries may be grouped together in different ways at the end of different chapters, each grouping reflecting the semantic perspective of each chapter. There will usually be several other, and often many other, glossary entries that are not included in the list, and we recommend that the Glossary be consulted whenever an unfamiliar term is encountered.

We note, in particular, that none of the nine types of pipeline dataset are included in this list. In general, we leave category sets out of these lists, but recommend that the reader look them up in the Glossary.