This tells the story of over 25 years of evolution and tailoring the goals and processes for a particular environment. The continuous improvement was measured by taking values of three data points: the development defect rates, the reduced cost for development, and the improvement of reuse of code at three points in time: 1987, 1991, and 1995. Each data point represents the average of the three years around it [Basili et al. 1995]; see Table 5-1.
Table 5-1. The results of the QIP approach in the SEL
|
Continuous improvement in the SEL |
1987–1991 |
1991–1995 |
|---|---|---|
|
Development defect rates |
75% |
37% |
|
Reduced cost |
55% |
42% |
|
Improved reuse |
300% |
8% |
During this period there was a continual increase in the functionality of the systems being built. An independent study estimated it to be a five-fold increase from 1976 to 1992.
The cost of this activity was about 10% of the development costs. However, the data shows an order of magnitude improvement for that 10% cost, which is a pretty good ROI.
But the data in Table 5-1 are not the results of a controlled experiment or even a well-defined case study. One can argue that these results might have occurred anyway, even if we did nothing. There is no control group. But the people involved believe differently. We believe the learning process worked and delivered these results. No controlled experiment could have been developed for this 25-year study, as we did not know when we started what we would do and how it would evolve.
During the 25 years of the SEL, we learned a great deal about software improvement. Our learning process was continuous and evolutionary, like the evolution of the software development process itself. We packaged what we learned into our process, product, and organizational structure. The evolution was supported by the symbiotic relationship between research and practice. This relationship requires patience and understanding on both sides, but when nurtured, really pays dividends.
The insights gained from learning by application, supplemented where appropriate by pre-experimental designs, quasi-experiments, controlled experiments, and case studies, provided a wealth of practical and theoretical results that could not have been gathered from formal experiments alone. It was the large-scale learning that allowed for the development of such things as the GQM approach, the QIP, and the Experience Factory, as well as the development and evaluation of various techniques and methods.