There are two kinds of statistics, the kind you look up and the kind you make up.
Table 15-7 summarizes my results. I have marked cells where an operating system excels with a + and corresponding laggards with a –. For a number of reasons it would be a mistake to read too much from this table. First of all, the weights of the table’s metrics are not calibrated according to their importance. In addition, it is far from clear that the metrics I used are functionally independent, or that they provide a complete or even representative picture of the quality of C code. Finally, I entered the +/– markings subjectively, trying to identify clear cases of differentiation in particular metrics.
Table 15-7. Result summary
Nevertheless, by looking at the distribution and clustering of markings, we can arrive at some important and plausible conclusions. The most interesting finding, which I drew from both the detailed results listed in the previous sections and the summary in Table 15-7, is the similarity of the values among the systems. Across various areas and many different metrics, four systems developed using wildly different processes score comparably. At the very least, the results indicate that the structure and internal quality attributes of a large and complex working software artifact, will represent first and foremost the formidable engineering requirements of its construction, with the influence of process being marginal, if any. If you’re building a real-world operating system, a car’s electronic control units, an air traffic control system, or the software for landing a probe on Mars, it doesn’t matter whether you’re managing a proprietary software development team or running an open source project: you can’t skimp on quality. This does not mean that process is irrelevant, but it does mean that processes compatible with the artifact’s requirements lead to roughly similar results. In the field of architecture this phenomenon has been popularized under the motto “form follows function” [Small 1947].
One can also draw interesting conclusions from the clustering of marks in particular areas. Linux excels in various code structure metrics, but lags in code style. This could be attributed to the work of brilliant, motivated programmers who aren’t, however, effectively managed to pay attention to the details of style. In contrast, the high marks of WRK in code style and low marks in code structure could be attributed to the opposite effect: programmers who are effectively micro-managed to care about the details of style, but are not given sufficient creative freedom to develop techniques, design patterns, and tools that would allow them to conquer large-scale complexity.
The high marks of OpenSolaris in preprocessing could also be attributed to programming discipline. The problems from the use of the preprocessor are well-known, but its allure is seductive. It is often tempting to use the preprocessor to create elaborate domain-specific programming constructs. It is also often easy to fix a portability problem by means of conditional compilation directives. However, both approaches can be problematic in the long run, and we can hypothesize that in an organization such as Sun, programmers were discouraged from relying on the preprocessor.
A final interesting cluster appears in the low marks for preprocessor use in the FreeBSD kernel. This could be attributed to the age of the code base in conjunction with a gung-ho programming attitude that assumes code will be read by developers at least as smart as the one who wrote it. However, a particularly low level of namespace pollution across the FreeBSD source code could be a result of using the preprocessor to set up and access conservatively scoped data structures.
Despite various claims regarding the efficacy of particular open or closed source development methods, we can see from the results that there is no clear winner (or loser). One system with a commercial pedigree (OpenSolaris) has the highest balance between positive and negative marks. On the other hand, WRK has the largest number of negative marks, and OpenSolaris has the second-lowest number of positive marks. Looking at the open source systems, although FreeBSD has the highest number of negative marks and the lowest number of positive marks, Linux has the second-highest number of positive marks. Therefore, the most we can read from the overall balance of marks is that open source development approaches do not produce software of markedly higher quality than proprietary software development.