Appendix A. How SIG Measures Maintainability

SIG measures system maintainability based on eight metrics. Those eight metrics are discussed in Chapters 2 through 9. Those chapters include sidebars explaining how SIG rates source code properties relevant to those guidelines. These ratings are derived from the SIG/TÜViT¹ Evaluation Criteria for Trusted Product Maintainability. In this appendix, we provide you with additional background.

Together with TÜViT, SIG has determined eight properties of source code that can be measured automatically. See “Why These Ten Specific Guidelines?” for how these properties have been chosen.

To assess maintainability of a system, we measure these eight source code properties and summarize these measurements either in a single number (for instance, the percentage of code duplication) or a couple of numbers (for instance, the percentage of code in four categories of complexity, which we call a quality profile; see “Rating Maintainability”).

We then compare these numbers against a benchmark containing several hundreds of systems, using Table A-1 to determine the quality level on each property. So, if the measurement for a system is among the top 5% of all systems in the benchmark, the system is rated at 5 stars for this property. If it is among the next best 30%, it rates 4 stars, and so forth. This process of comparing quality profiles for each system property against the benchmark results in eight star ratings, one for each system property.

We then aggregate the ratings to arrive at one overall rating. We do this in two steps. First, we determine the ratings for the subcharacteristics of maintainability as defined by ISO 25010 (i.e., analyzability, modifiability, etc.) by taking the weighted averages according to the rows of Table A-2. Each cross in a given row indicates that the corresponding system property (column) contributes to this subcharacteristic. Second, we take a weighted average of the five subcharacteristics to determine an overall rating for maintainability.

This describes the SIG maintainability model in a nutshell, since there is more detail to it than what we can cover in this appendix. If you would like to learn more about the details of the maintainability model, a good start for elaboration is the following publication:

• Visser, Joost. SIG/TÜViT Evaluation Criteria Trusted Product Maintainability. http://bit.ly/eval_criteria

Background on the development of the model and its application is provided in the following publications:

• Heitlager, Ilja, Tobias Kuipers, and Joost Visser. “A Practical Model for Measuring Maintainability.” In Proceedings of the 6th International Conference on the Quality of Information and Communications Technology (QUATIC 2007), 30–39. IEEE Computer Society Press, 2007.

• Baggen, Robert, José Pedro Correia, Katrin Schill, and Joost Visser. “Standardized code quality benchmarking for improving software maintainability.” Software Quality Journal 20, no. 2 (2012): 287–307.

• Bijlsma, Dennis, Miguel Alexandre Ferreira, Bart Luijten, and Joost Visser. “Faster issue resolution with higher technical quality of software.” Software Quality Journal 20, no. 2 (2012): 265–285.