The system discussed in this chapter is a very large-scale (that is, a million lines or more), distributed, real-time system written in the C programming language (with additional domain-specific languages as needed) in a Unix-based, multiple machine, multiple location environment.
The organizational structure is typical with respect to projects for systems of this size and for the number of people in each organization. Not surprisingly, different organizations are responsible for various parts of the system development: requirements specification; architecture, design, coding and capability testing; system and system stability testing; and alpha testing.
The process of development is also typical with respect to projects of this size. Systems engineers prepare informal and structured documents defining the requirements for the changes to be made to the system. Designers prepare informal design documents that are subjected to formal reviews by 3 to 15 peers, depending on the size of the unit under consideration. The design is then broken into design units for low-level design and coding. The products of this phase are subjected both to formal code reviews by three to five reviewers and to low-level unit testing. As components are available, integration and system testing is performed until the system is completely integrated.
The release considered here is a “non-initial” release—one that can be viewed as an arbitrary point in the evolution of this class of systems. Because of the size of the system, the system evolution process consists of multiple, concurrent releases—that is, while the release dates are sequential, a number of releases proceed concurrently in differing phases. This concurrency accentuates the inter-release dependencies and their associated problems. The magnitude of the changes (approximately 15–20% new code for each release) and the general make up of the changes (bug fixes, improvements, new functionality, etc.) are generally uniform across releases. It is because of these two facts that we consider this study to provide a representative sample in the life of the project.
Faults discovered during testing phases are reported and monitored by a modification request (MR) tracking system (such as, for example, CMS [Rowland et al. 1983]). Access to source files for modification is possible only through the tracking system. Thus all change activity (whether fixing faults, adding new functionality, or improving existing functionality—that is, whether they are corrective, adaptive, or perfective changes) is automatically tracked by the system. This activity includes not only repairs, but enhancements and new functionality as well. It should be kept in mind, however, that this fault tracking activity occurs only during the testing and released phases of the project, not during the architecture, design, and coding phases. Problems encountered during these earlier phases are resolved informally without being tracked by the MR system.
The goal of this study was to gain insight into the current process of system evolution by concentrating on one release of a particular system. The approach we used is that of surveying, by means of a prepared questionnaire, those developers who “owned” the fault MR at the time it was closed, surveying first the complete set of faults and then concentrating on the largest set of faults in more depth. This survey was the first of its type, although there have been some smaller studies using random selections. The CMS MR database was used to determine the initial set of fault MRs to survey and the developers who were responsible for closing those fault MRs. The survey identifying the fault MR was then sent to the identified developer to complete.
For a variety of reasons (schedule pressure among them), there were significant constraints placed on the study by project management: first, the study had to be completely non-intrusive; second, it had to be strictly voluntary; and third, it had to be completely anonymous. We will see in the later discussion about validity issues that these mandates were the source of some study weaknesses.
It is with this background that we present our surveys, analyses, and results.