An organization starts with the capability level 0: incomplete. It indicates that processes are not, or only partially, performed. It also indicates that at least one specific goal of the process area is not satisfied. There are no generic goals for this capability level because partially performed processes should not be institutionalized [11].

The organization can proceed to capability level 1: performed if all generic goals of level 1 are satisfied. This level requires that processes are performed and produce the needed output. However, this level doesn’t require that the process itself be institutionalized, which means that process improvements can be lost over time [11].

The capability level 2: managed requires a process that is planned and executed in accordance with policy. This managed process employs skilled people who have access to the right resources and are able to produce controlled outputs. All relevant stakeholders are involved in the process, which is monitored, controlled, reviewed and evaluated on a regular basis.

The next capability level an organization can accomplish is capability level 3: defined. It is characterized by a defined process which is a managed process derived from the organization’s set of standard processes and derived to the needs of the circumstances. The process has been tailored according to the tailoring guidelines of the organization and maintains a process description. In addition, it contributes process-related experiences back to the overall process organization.

Capability level 4: quantitatively managed is a defined process (see capability level 3) that uses statistical and other quantitative methods to control selected subprocesses. These methods are used to identify processes that produce a higher number of defect outputs or outputs with a lower quality [10].

The highest capability level that an organization can reach is capability level 5: optimizing which focuses on the institutionalizing of an optimizing process. This process requires that the organization constantly measures its (quantitatively managed) processes, analyzes trends, surveys technical practice, addresses common causes of process variation and then adapts the processes to changing business needs [10].

The first maturity level 1: initial indicates an organization with ad-hoc and chaotic processes. There is no stable process environment provided by the organization. The success of the organization depends on the skills and engagement of individuals rather than defined and established processes. Organizations at maturity level 1 can deliver working products. However, they often exceed the budget and original delivery schedule. These organizations usually overcommit, abandon their processes in times of stress and have problems in repeating past successes [12].

Organizations at maturity level 2: managed have processes that are planned and executed in accordance with policy and that involve all relevant stakeholders. Skilled people with adequate resources produce controlled outputs under a monitored, controlled, and reviewed process that is evaluated on a regular basis. Existing processes are retained during times of stress [12].

Maturity level 3: defined indicates well-characterized and understood processes which are described in standards, procedures, tools, and methods. Organizational standard processes are established and improved over time. The major difference between maturity levels 2 and 3 is that standards, process descriptions and procedures at maturity level 2 can be different at each specific process instance. However, in maturity level 3, standards, process descriptions and procedures are tailored from the standard processes of the organization [12].

At maturity level 4: quantitatively managed, organizations use quantitative objectives for quality and process performance for managing their projects. Selected subprocesses are measured by collecting and statistically analyzing specific measures of process performance [12].

Organizations at maturity level 5: optimizing continually improve their processes using a quantitative approach to understand the variation of their processes and their process outcomes. The focus is on the continual improvement of process performance by incrementally improving processes and used technology [12].

Table 3.1 shows the activities and tasks of the Data Vault 2.0 methodology and how they relate to the maturity levels of CMMI.

Organizations can use this table to focus on the given Data Vault activities when they try to advance through the maturity levels. For example, parallel teams should only come into the focus of the organization when they have already achieved CMMI maturity level 4 (quantitatively managed) and try to achieve level 5 (optimizing). This follows the recommended practice as outlined earlier in this chapter, to advance through the maturity levels instead of directly trying to become an optimizing organization from the onset. While the first approach requires more time and a careful development of organizational capabilities, the risk is much lower than going directly after maturity level 5.

As a result, agile methodologies have been developed to make software development more flexible and overall more successful. Scrum is one example of such agile methodology and is described in the following sections. It was introduced in the late 1990s [15,16] and has become “the overwhelming [agile] favorite.” [17]

User requirements are maintained as user stories in a product backlog in Scrum [18]. They are prioritized by business value and include requirements regarding customer requests, new features, usability enhancements, bug fixes, performance improvements, re-architecting, etc. The user stories are implemented in iterations called “sprints” which last usually two to four weeks. During a sprint, user stories are taken off the product backlog according to the priority of the item and implemented by the team [19]. The goal of Scrum is to create a potentially shippable increment with every sprint, which is a new release of the system that can be presented to the business user and potentially put into production [20] (Figure 3.7).

This requires that a user story be implemented and tested as a whole, including all business logic that belongs to this story. All stakeholders, such as business users and the development team, can inspect the new, working feature and provide feedback or recommend changes to the user story before the next sprint starts. Scrum supports the reprioritization of the product backlog and welcomes changing business requirements between the sprints [20] (Figure 3.8).

This helps to improve the outcome of the project in a way that meets the expectations of the business user. To ensure this, the customer should become as much a part of the Scrum team as possible [21].

The next sections explain the elements of Scrum in more detail.

In the first project base course, each iteration is only one day long. After the iteration, all stakeholders review the current “potentially shippable” system and check whether it still meets the expectations. If one of the stakeholders of the project identifies an issue in the project that requires correction, this correction will be relatively quick. For example, a business requirement can be modified and the system changed in the next iteration. In many cases, the change will be relatively small because it was identified early in the process. In the worst case, the whole iteration needs to be rolled back and performed again. In this case, only one day of work is lost.

If the iteration cycle (the sprint) is longer, for example a month, more requirements, expressed as user stories, will be implemented. Because there are often requirements that depend on each other, changing a requirement with many dependencies will take much more corrective effort than in the first case with a shorter sprint length.

Each user story is prioritized before adding it to the product backlog. Developers strictly have to pick up user stories from the backlog in the order of priority [14]. At the beginning of each iteration, the user stories that should be implemented are taken off the product backlog and moved to the sprint backlog (see Figure 3.7 for details) [18]. The user story is designed, implemented, tested and integrated into the product as a whole. Problems are fixed immediately in order to maintain a “potentially shippable” product [14]. Figure 3.10 shows how team members work off the product backlog.

The work items that have a high priority (and therefore will be picked next by a team member) have to be detailed and well-defined because they describe the upcoming task for the developer. On the other hand, user stories with lower priority can be less detailed. These user stories are often not well elaborated by the business user and require more definition. The typical process is to remove such a user story, split it up into multiple stories, define them in more detail and put them, prioritized, back into the product backlog [18]. It is also possible to reprioritize existing user stories between sprints to adapt the project to changing demands by the business. A common practice is to derive (at least the initial) user stories from a requirements specification written by the product owner using traditional methods [22].

Not only that – running agile projects also requires an agile organization. If the organization thinks waterfall with “over-the-fence” throwing of artifacts between departments (e.g., from development to test to business) and putting changes into production requires a minimum of 3 weeks, your team will not perform very well from an agile perspective. As already stated in this chapter, the goal is to put the changes that are being developed in one iteration into production in the same sprint. That goal might not hold true at all times, for example if it is not possible to scope a change down to a manageable piece that can be developed and deployed within one sprint. In this case, multiple sprints might further develop the artifact until it is ready to deploy. However, the standard should be to deploy within the sprint where the artifact is being developed. There is no “agile big-bang” – that is, developing the data warehouse over sprints and deploying it at the end of the project in two years.

CMMI can help to evolve both the team and the organization to achieve agility. When following the recommendations outlined in section 3.1.1 (subsection titled “Advancing to Maturity Level 5”), organizations develop these skills over time, leaving team members time to learn, use and improve their agile skills. In fact, Scrum builds on CMMI level 5 (optimizing) where teams have the goal of improving their abilities over time. We will discuss this in more detail in section 3.3.

Most agile projects have in common the understanding that, to achieve the presented goals, the business intelligence (BI) team has to assume several responsibilities:

The presented responsibilities are directly derived from Scrum principles but apply to all agile projects. Therefore, the Data Vault 2.0 methodology can only be successful if the development teams adhere to these responsibilities.

To support an agile approach to requirements gathering, requirements are gathered along the way, unlike classical data warehousing where these requirements are gathered at the beginning of the project. The approach that has worked best in our projects is to use Raw Marts and quickly roll out data into the requirements meeting for review. These Raw Marts are used for creating reports or cubes to a limited number of business users who attend the requirements meeting and are not intended for distribution. This is because the Raw Marts contain raw data with or without incomplete implementation of business rules. The exercise is to show these reports to the users and ask them: “what is wrong with this report?” It turns out that business users can easily point to the problems within the report and, by doing so, provide all the business rules IT needs to implement the final report.

The procedure for this approach to requirements gathering is as follows:

After these business rules have been implemented by IT, the business side of the project can review and test the outputs and ask for further modifications if they are not yet satisfied with the end result. However, these modifications become change requests and are implemented in subsequent sprints. The described agile requirements-gathering process helps business users to express their business rules. For many of them, the traditional focus on requirements documents is too abstract and prevents the required identification from identifying issues with draft reports.

A recommended practice is to record these requirements meetings and set up a Wiki or other Web 2.0 enabled Web site that is available to everyone in the organization. The recordings from the meeting, including a description of the found business rules, should be posted on the Web site to ensure the transparency of the requirements-gathering process. Web 2.0 mechanisms enable participants to post comments or even modify the business rules according to their understanding. This approach makes sure that the requirements are correct in the first place. If a lot of discussion occurs on the Web site, another requirements meeting may be necessary to clarify any open issues before implementation should begin. Having these discussions before the actual implementation has taken place means a huge benefit and productivity boost to the organization and is a contributing factor to the overall success of the project. In order to make the right assumptions about the functionality that the team is able to complete in one sprint, which is important for scoping, the team has to be able to make correct estimates about the effort it takes to complete certain functionality. This topic is covered in the next section.

With the use of function points, FPA is independent from technology-dependent metrics, such as lines of code (LOC) or other metrics that require a specific tool or platform to be measured [24]. Lines of code measures penalize high-level languages [25]. For example, the same functionality requested by a business user might require 100 lines of code in C# but 500 lines of code in C. Function points, however, measure the functionality that has been delivered to an end user or will be delivered in the future [23].

Figure 3.12 shows the functional characteristics of a software system in the airline industry.

The functional characteristics of software are made up of external inputs (EI), which is the data that is entering a system; external outputs (EO) and external inquiries (EQ), which is data that leaves the system one way or another; internal logical files (ILF), which is data manufactured and stored within the system; external interface files (EIF), which is data that is maintained outside the system but necessary to perform the task. The assessment of function points also includes the complexity of the general system.

In function point analysis, systems are broken into smaller components for better analysis [26]. The next section introduces the high-level steps to count function points and perform a function point analysis.

The described series of steps is valid for all software development efforts, including data warehousing projects. The next sections describe how this procedure can be applied in such projects.

These challenges make the previously described process, which is used extensively in classical software engineering, a tough, if not impossible, task. This is due to the process orientation of software versus a subject-oriented view of data warehouses [28]. As a result, the process has to be adapted to be used in data warehousing.

Figure 3.11 shows that each capability is composed of staging tables, objects in the core data warehouse and the data mart. That includes all logic implemented in ETL, such as business rules. Also included are graphical elements such as reports, dimensions and measures in the OLAP cube and items on the dashboard.

The reason why staging tables, which are actually within the boundary of the data warehouse functionality, are considered as external interface files (as opposed to internal logic files) is that they primarily hold external data as a direct copy (enriched with some simple metadata). However, if internal processing is applied to these tables, they are considered as internal logic files [28].

For the same reasons, target tables are considered as internal logic files and not external to the boundary: they are updated with data processed within the boundaries of the data warehouse function [28].

Mappings in the data warehouse implement the required logic by extracting data from source tables, transforming it, and loading it to the target tables. They are considered as external input (EI) because they alter the system’s behavior by implementing the business rules [28].

Data warehouse mappings usually make extensive use of lookup tables which map codes with more descriptive data. For example, surrogate keys are mapped against business keys to support the understandability for end-users. Because they provide such data, these tables have more value to the business user than providing technical advantages. Therefore, they are considered as internal logic files (ILF) [28].

Process alerts notify external entities of the ETL process statuses and are considered as external output (EO) [28].

The figure shows the causes and effects of complexity factors on typical ETL projects. However, not all these factors are in correlation with the actual effort. They depend on the organization and project teams. To find out the factors that have an effect, a correlation analysis, for example using a step-wise forward regression analysis, is required.

A sample equation for a specific data warehouse project regression analysis could look like the one presented in Equation 3.1:

Sample equation for regression analysis [58]:

The actual equation for the project’s regression analysis would also contain the regression coefficients that are depicted as characters A to H in Equation 3.1.

The estimation process should support the IT team when the business asks for an estimated timeframe or effort for the requested functionality to be delivered. The ability to provide a profound answer is a core requirement from CMMI, in order to achieve capability levels higher than the initial/performed capability level. For example, FPA can be applied in capability level 2 (managed) in order to quantify the size of functional requirements in requirements management, estimate the effort and cost in project management, report function point data to management, etc [30].

In order to perform an estimation process using FPA, it is required to keep accurate metrics about past data warehouse development efforts to be able to use the experience from the estimation process for future estimations. This information is used to adjust the level of effort to the organization. The person hours depend on the complexity of the functionality. It should be clear that difficult functionalities will take longer to implement than easy functionalities (per calculated function point). Table 3.3 shows an example that is valid within one particular project.

Note that Table 3.3 has to be adjusted to each project under estimation and these values also change over time as developers are becoming more experienced or, due to turnover, lose experience in the team.

As we have seen in the previous discussion, the number of function points depends on the functional characteristic of the software to be built. To use FPA in the Data Vault 2.0 methodology, the functional characteristics of software (external inputs, external outputs, external inquiries, internal logical files, and external interface files) had to be adapted to reflect Data Vault projects. The following functional characteristics of data warehouses built with Data Vault are defined:

In the same way, it is possible to define other functional characteristics, e.g. for point-in-time (PIT) tables, bridge tables, Business Vault entities or OLAP cubes. In general, Data Vault 2.0 loading routines are EtL patterns (small-capital “t” for transform): there is only minimal transformation of the raw data required to load the Raw Data Vault. Therefore, the function points for those transformations should be low, leading to high levels of optimization, low levels of complexity, and simple function point counts.

Consider the exemplary list of function points per work item shown in Table 3.4, used to calculate the levels of effort.

The example shows that each type of functionality has to be estimated independently from the other types. Column Item indicates the type of functionality. The second column indicates the complexity factor of the item. The shown values are typical. However, it is also possible that there are satellites in the easy and the moderate category (some satellites have a low number of attributes, which makes the load process much easier than the average). For that reason, multiple rows are added for satellites with varying complexity factors. The estimated function points indicate the number of function points counted for the item type. The Estimated Total Hours column is the product of the Estimated Function Points column in Table 3.4 and Person Hours per Function Point in Table 3.3.

Note that there is a distinction made between thin and wide satellites. This distinction refers to the number of columns and not the physical width of the table, which depends on the number of columns and the width of each column in bytes. In general, the more columns a satellite has, the more complex load logic is required, the more spelling errors might occur, and the more chances to load the wrong fields to the wrong targets or miss a column comparison occur. This is why this table distinguishes between such loads.

Similarly, the same concept is applied to unit tests in the second part of Table 3.4. For each item and the complexity for testing, the function points are estimated and enhanced with the total estimated person hours, which are based on Table 3.3 again. Therefore, the second part of the table turns out to be very similar to the first part. However, it adjusts for different efforts between implementation and testing. Remember that this table has to be built individually for each project and updated over time. There might be additional differentiation required between less or more complex work items (such as the satellite loads in the example in Table 3.4). To get the process started, it is important to follow the general procedure as outlined before to come up with function point estimates that relate to the actual effort behind the items being built and to track the person hours required for implementation and testing behind these function points. Therefore, tracking the hours per function point becomes an important step in the first cycle of your development effort, before you can actually estimate effort in the future. This is due to the fact that estimates are driven from past experiences within a given context. It is not possible (without complex transformation) to use the experience from one context (e.g., one organization) to estimate the effort in another context, because there are many differences between organizations – for example, the maturity level, the experience levels of development staff and management, and the ratio between internal and external staff (who bring specialized experiences to the firm).

These general rules should provide guidance to development teams when deciding on the function points associated with individual functional characteristics. Note that the estimated person hours for development but also for maintenance are drastically reduced when automating the Data Vault loading process, a concept that is beyond the scope of this book.

When organizations advance through CMMI maturity levels, they will recognize that estimation based on function points becomes easier over time. This is due to consistent and repeatable business processes which are easier to predict because experiences from the past can be actually applied to such processes. Without such consistent execution, the estimates are not well founded because the circumstances of process execution change all the time. Well-defined standards help organizations to achieve such consistent and repeatable business processes. FPA also helps to promote measurable and optimizable components, which are required to perform estimates. This goal is supported by separation of the individual components that have to be delivered to business. Without this separation, estimates would overlap functionality and therefore make the estimates less accurate. Identifying the components also reduces the confusion over what is delivered and when. Because each component has to be identified and described, it is also possible to schedule the delivery of each component.

The overall goal of the estimation process is to make the data warehouse development effort more predictable. It helps to justify the costs of the data warehouse project and secure further funding from the business. Because correct estimates build trust into the development team and the business, they are therefore a central part of the Data Vault 2.0 methodology.

Once the team has decided what and how much it is going to deliver in the upcoming sprint, the focus goes over to project execution, which is covered in the next section.

The major challenge in this phase is to consolidate an unambiguous collection of the user requirements, which is often difficult because different user classes often have contradicting requirements. Therefore, this phase is often the most difficult and time-consuming phase in the waterfall model [32]. In order to collect system requirements, analysts use various traditional and modern methods. Traditional methods include the following [34]:

Modern methods include the following: [34]

These methods have in common to identify and gather the requirements from business users in order to provide them to the next process steps, which are described in the following sections.

Another area of interest is the selection of the required hardware and software which is based on the performance, security and storage requirements of the whole system. In some projects, there is an existing infrastructure for data warehousing available. In such cases, new databases have to be created for the new project and the infrastructure provider will check whether there is enough capacity in the existing infrastructure to serve the new project. In other projects, where data warehousing is introduced to the organization, new infrastructure has to be created. This task should not be underestimated and often becomes an independent project with the procurement of hardware and supporting services, such as maintenance of the future data warehouse (think of backup and restore services, deployment of new features, bug tracking, end-user support, network maintenance, etc.).

There are various tools included in Microsoft SQL Server 2012 to build the data warehouse. Most of these tools can be found in Microsoft Business Intelligence Development Studio. This development environment includes the following tools: [37]

In addition, most teams use additional tools from Microsoft or third-party application vendors. Tools often used include:

Once the tests have passed, the modules are added to the system by integration. After the integration of these units has been completed, the whole system is tested again. When all units at the bottom that can be integrated with each other are integrated, the integration team moves to the next level and integrates the larger parts. The new larger unit is tested again to check whether all subunits work together seamlessly [32].

Because such a testing approach requires running many tests over and over again, tools for test automation are often used in this phase.

In order to support the operations and maintenance team, the data warehouse team which develops the solution has to provide both an end-user and administration documentation, including specific instructions for the maintenance of the data warehouse (such as loading new data deliveries or customizing existing reports).

It is possible to collect a large number of measures regarding the processes in software engineering [46]:

And because the owners of the processes have the same interests as the ones in their manufacturing counterparts, e.g., improving the quality of the process output, improving performance, better meeting customer needs, etc., Six Sigma can be applied to software engineering processes as well. However, because of differences between software engineering and other engineering disciplines, additional thought is required [46].

Often, the overall process cycle time is much longer in software engineering than in creating machine-manufactured goods. Therefore, Six Sigma projects might take longer or might have a greater risk, due to less data for statistical analysis [46].

The intensity of human interaction is much higher than in many other manufacturing areas. It also involves creative elements throughout the project life-cycle. Six Sigma teams might focus on the repetitive tasks within the project (such as inspections) or they might focus on the human factors. In any case, it is important to carefully normalize the data to make sure that the comparisons are valid [46].

In software engineering, there is only one master copy produced. The duplication of this master copy is simple and can be easily made without variation of the produced output. However, developing the subcomponents of the software is done only once and there might be variation between the individual subcomponents of the final product. Another source of variation is the implementation of the software into the user’s environment [46].

Top level management commitment is required because Six Sigma is a strategic management decision that has to be initiated by top-level management. In order to become a success, all elements of the framework, including the improvement strategy, require top-level management commitment [40]. Special attention by the top management should be given to the training program and project team activities, because they are seldom successful without strong top-level commitment. It should be clear that true commitment is required and not just empty promises. Instead, pragmatic management is required that drives the initiative for several years [32].

However, not only is the top management committed to the Six Sigma initiative success. Instead, full stakeholder involvement is required. All employees, suppliers, customers, owners and parts of the close society should become involved in the Six Sigma improvement process. The majority of activities are performed by employees who need support by top management, for example in the availability of training courses, project team activities and evaluation of process performance. Key suppliers to the organization are encouraged to start their own Six Sigma initiatives and are supported by information sharing and participation at in-house trainings. Financial support to smaller companies is also not uncommon [32].

The success of any Six Sigma initiative depends on skilled stakeholders, such as employees. A comprehensive knowledge of the improvement methodology, the process performance and statistical tools is required. It is also important to understand the processes of the project team activities and how to deploy customer requirements. It should be clear that this skillset is not readily available within the organization and has to be built using training schemes and other knowledge transfer. For this purpose, Six Sigma provides standardized training courses with various levels. These levels are denoted by the belt rank system from martial arts: there are White Belts, Green Belts, Black Belts, Master Black Belts and Champions [32]. Importance is given to Black and Green Belts because they become the center of the Six Sigma team.

The job of the Black and Green Belts is to keep the project focused [47]. The following procedure is recommended for project team activities [32]:

To uncover new areas for process improvement, Six Sigma depends on a pragmatic system for measuring performances. It reveals poor process performance and helps to identify future problems that have to be dealt with. The measurement depends on characteristics of the product that are tracked over time and consolidated. Results are typically visualized in trend charts and other graphical illustrations [32].

As already indicated, the improvement strategy is based on the DMAIC steps for improvement. The next section covers them in detail.

The first step in the approach is to define the problem and clearly describe the impact on customer satisfaction stakeholder’s employees, and profitability. The project members define the following: [48]

It is important in this step to gather and document the customer requirements and, once understood, send them towards the operational level where project goals and objectives are set. There are several techniques that support the project team, including [48]:

The second step is to measure the current performance to identify opportunities for improvement. After changes have been done, the business can measure its success by comparing the new performance with the past performance baseline. Several statistical tools are available for the measurement: including averages, standard deviation and probability distributions.

After having identified issues in the process, the next step is to search for the root cause during the analyze phase. Opportunities for improvement are prioritized by two dimensions: their contribution to customer satisfaction and the impact on profitability [48].

The improvement step implements the opportunities identified in the previous step. Project members develop solution candidates and select the solution with the best results and performance. While other improvement frameworks develop solutions by varying one variables of the process at a time, Six Sigma is using statistically designed experiments to vary multiple variables simultaneously and obtain multiple measurements under the same experimental conditions [48].

The last step of DMAIC improvement is the control step. Its goal is to control the improved processes and to make sure that the Six Sigma initiative sustains. If, however, the results of the improvements made in the last steps are at risk, the DMAIC improvement can start again, as Figure 3.21 shows.

With this approach in mind, there is not much difference between software development and data warehouse development in respect to Six Sigma. Therefore, the same concepts as regarding the application of Six Sigma to software apply (see previous section titled “Applying Six Sigma to Software”).

Both methodologies rely on objective assessments of the data quality. Data quality on the other hand, has many dimensions that are of interest for business users, and therefore can be used to classify the quality of data (or information). Table 3.10 lists common data quality dimensions.

The assessment of these data quality dimensions can be task-independent or task-dependent. Task-independent assessments require no knowledge about the context of the application and can be applied to any data set. Task-dependent dimensions on the other hand require knowledge about the business rules of the organization, regulations of the company or legal authorities [54].

The first phase is the definition phase, where the data is analyzed and business requirements are gathered. The information manufacturing system which processes the information is also defined. The output of the phase is a logical and physical design of the information product with attributes related to quality and a quality E/R model which defines the information product and the information quality. The second phase is the measurement phase that defines the metrics for the information quality and uncovers problems in information quality after an analysis. The third phase is the analysis phase that analyzes the information quality problems found in the previous phase and identifies the root cause of errors. The fourth phase is the improvement phase where key areas for improvement are selected along with strategies and techniques. These strategies and techniques are applied within the definition phase of the TDQM when the cycle starts over again [56].

The input to the definition phase is the data definition from the operational systems, stakeholder perspectives and project and context information from the data warehouse. Relevant data quality dimensions of the data warehouse are identified including to relations to objects in the data warehouse. Business users and other stakeholders weight the quality dimensions. The measurement phase uses the quality dimensions from the definition phase and identifies dependencies among them. In the next phase, the analysis phase, the results from the measurement phase are used to identify critical areas by comparing the data quality values and the data quality requirements of the business users. The list of data quality dimensions that require improvement are used in the improvement phase to improve the data [57].

TQM has several primary elements that are important when implementing a customer-focused continual improvement process [58]:

The meetings discussed in the introduction to this section should integrate these elements in order to be successful from a TQM perspective.

Sometimes, when erroneous information is found in reports or OLAP, the organization decides not to follow the recommended TQM approach to identify the root cause of the error and fix it, probably in the source system or the business processes. Instead, the organization decides to fix the error somewhere between the source system and the front-end reports. If such an approach is followed, the only acceptable way to fix the error in the data warehouse is to apply soft business rules on the way out of the Raw Data Vault, for example using the Business Vault or when providing the information mart. Chapter 13, Implementing Data Quality, demonstrates how to implement data quality in the data warehouse. However, in the light of TQM, the goal of the effort is to achieve a closed loop process. This is done by involving the business user in aligning the data set across the source systems and correcting the error in the source system instead of in the data warehouse.

But TQM is more than just fixing data quality (DQ). While it involves data quality activities, it is the ability to see the error in the error marts, take the data into question, and then issue change requests back to the source systems in order to correct the process, or the data, or both. Without closed loop processing (as described previously: users fix and align the data sets in the source), it is nothing more than pure DQ. Instead, TQM requires that people are involved and the loop between source system and presentation layer is closed by providing feedback and removing the errors, thus closing the gaps that are found in the system as a whole.

Chapter 9 will discuss master data management (MDM), which is used to enable business users and requires that “management” and governance work together. The same principles apply to closing the gaps and aligning all the source systems to the vision of master data that the business users designate. The same chapter covers how managed self-service BI is used by business users to directly interact with the data in the enterprise data warehouse, making corrections in real-time, with the feedback loop becoming a publication of the messages (resulting from managed self-service BI), directly from the data warehouse and feeding back to the operational systems through the Service Oriented Architecture (SOA) and the Enterprise Service Bus (ESB). This requires write-back capabilities to enable business users to make corrections in source systems.