The first phase of the survey questionnaire had two main components: the determination of the category of the fault reported in the MR and the testing phase in which the fault was found. In determining the fault, two aspects were of importance: first, the development phase in which the fault was introduced, and second, the particular type of the fault. Since the particular type of fault reported at this stage of the survey tended to be application or methodology specific, we have emphasized the phase-origin nature of the fault categorization. The general fault categories are as follows:

The other main component of the survey concerned the phase of testing that uncovered the fault. The following are the different testing phases:

The time interval during which the faults were found (that is, when the fault MRs were initiated and when they were closed) was retrieved from the MR tracking system database.

Ideally, the testing phases occur sequentially. In practice, however, due to the size and complexity of the system, various phases overlap. The overlap is due to several specific factors. First, various parts of the system are modified in parallel. This means that the various parts of the system are in different states at any one time. Second, the iterative nature of evolution results in recycling back through previous phases for various parts of the system. Third, various testing phases are begun as early as possible, even though it is known that that component may be incomplete. Looked at in one way, testing proceeds in a hierarchical manner: testing is begun with various pieces, then subsystems, and finally integrating those parts into the complete system. It is a judgment call as to when different parts of the system move from one phase to the next, determined primarily by the percentage of capabilities incorporated and the number of tests executed. Looked at in a slightly different way, testing proceeds by increasing the size and complexity of the system, while at the same time increasing its load and stress.

Table 25-1 summarizes the fault MRs by fault category. The fault MRs representing the earlier part of the development or evolution process (that is, those representing requirements, design, and coding) are the most significant, accounting for approximately 33.7% of the fault MRs. Given that the distinction between a design fault and a coding fault required a “judgment call” on the part of the respondent, we decided to merge the results of those two categories into one: design/coding faults account for 28.8% of the MRs. However, in the process structure used in the project, the distinction between requirements and design/coding is much clearer. Requirements specifications are produced by systems engineers, whereas the design and coding are done by developers.

The next most significant subset of MRs were those that concern testing (the testing environment and testing categories)—24.8% of the MRs. On the one hand, it is not surprising that a significant number of problems are encountered in testing a large and complex real-time system where conditions have to be simulated to represent the “real world” in a laboratory environment. First, the testing environment itself is a large and complex system that must be tested. Second, as the real-time system evolves, so must the laboratory test environment evolve. On the other hand, this general problem is clearly one that needs to be addressed by further study.

“Duplicate” and “no problem” MRs account for another significant subset of the data—29.8%. Historically, these have been considered to be part of the overhead. Certainly the “duplicate” MRs are in large part due to the inherent concurrency of activities in a large-scale project and, as such, are difficult to eliminate. The “no problem” MRs, however, are in large part due to the lack of understanding that comes from informal and out-of-date documentation. Obviously, measures taken to reduce these kinds of problems will have beneficial effects on other categories as well. In either case, reduction of administrative overhead will improve the cost effectiveness of the project.

“Previous” MRs indicate the level of difficulty in finding some of the faults in a large, real-time system. These problems may have been impossible to find in the previous releases and have only now been exposed because of changes in the use of the system.

Figure 25-1 charts the requirements, design, and coding MRs by testing phase. We have focused on this early part of the software development process because that is where the most MRs occurred and, accordingly, where closer attention should yield the most results.

Figure 25-1. Fault categories found by phase

Please note that the percentages used in Figures 25-1 and 25-2 are the percentages of those faults relative to all the faults, not the percentages relative to just the charted faults. The phases in Figure 25-1 appear as sequential when in actual fact (as is almost always the case in software systems’ development) there is a lot of parallelism, with phases overlapping significantly. With hundreds of software engineers developing hundreds of features concurrently, the actual project life cycle is nothing like the sequential waterfall model, even though software development proceeds through a set of, often iterative, phases. That is the reason for Figure 25-2, which shows the same data relative to a fixed timeline.

Figure 25-2. Fault categories found over time

There are two important observations. First, system test (ST) was the source of most of the MRs in each category; capability testing (CT) was the next largest source. This is not particularly surprising, since that is when we were looking the hardest for, and expecting the most, faults. System test is where the various components are first integrated and is the most likely place to encounter the mismatches and misassumptions among components. So while all testing is looking for faults, system test is where we look the hardest. Capability testing is akin to unit testing in an appropriate context—internal and initial interaction faults are looked for. The capability test context limits the kinds of faults that can be exposed.

Second, all testing phases found MRs of each fault category. It is also not surprising that coding faults are found over the entire set of testing phases. One obvious reason for this phenomena is that changes are continually made to correct the faults that are found in the various earlier testing phases. Moreover, while it is disturbing to note that both design and requirements faults continue to be found throughout the entire testing process, we feel that this is due to the lack of precision and completeness in requirements and design documentation and is a general problem in the current state-of-practice rather than a project-specific problem.

The time values in Figure 25-2 are fixed intervals. From the shape of the data, it is clear that System Testing overlaps interval t4. It is unfortunate that we have only the calendar data (that is, the time boundaries on the MRs), as a correlation with effort data [Musa et al. 1987] (that is, the actual amount of time spent in that time period) would be extremely valuable.

For the requirements, design, and coding fault categories over time, Figure 25-2 shows that all fault types peaked at time t4 and held through time t5, except for the coding faults, which decreased.

The following general observations may be drawn from this general survey of the problems encountered in evolving a large-scale, real-time system:

These observations are limited by the fact that the tracking of fault MRs is primarily a testing activity. It would be extremely useful to observe the kinds and frequencies of faults that exist in the earlier phases of the project. Moreover, it would be beneficial to incorporate ways of detecting requirements and design faults into the existing development process.