4
ESTABLISHING A SOFTWARE MEASUREMENT PROGRAM*
This chapter provides an overview of software measurement and an infrastructure for establishing a software measurement program. It is recommended to start small and build on success. It is also recommended to combine a software measurement program with a software process improvement initiative so that the measurement program is sustainable. As far as possible, establish automated mechanisms for measurement data collection and analysis. Automated methods should be a support resource of the measurement process rather than a definition of the process. Regularly collect the core measurements and additional measurements specific to the local goals in the organization. Plan and schedule the resources that will be required to collect and analyze the measurement data within the organization’s overall software process improvement efforts and the specific organization’s projects. Evolve the measurement program according to the organization’s goals and objectives. Provide a mechanism for projects and the organization’s software process improvement group to consolidate software project measurements.
The following four steps illustrate a comprehensive process for establishing a software measurement program.
- Step 1: Adopt a software measurement program model
- Identify resources, processes, and products
- Derive core measurement views
- Step 2: Use a software process improvement model
- Establish a baseline assessment of the project/organization
- Set and prioritize measurable goals for improvement
- Establish an action plan with measures
- Accomplish actions and analyze results
- Leverage improvements through measurement
- Step 3: Identify a goal-question-metric (GQM) structure
- Link software goals with corporate goals
- Derive measures from attribute questions
- Establish success criteria for measurement
- Step 4: Develop a software measurement plan and case
- Plan: what, why, who, how, when
- Case: measurement evidence and analysis results
An organization may decide to implement a subset of these activities. Organizations should tailor their use of the activities as necessary to meet the organization and project goals and objectives. Each of these four major activities is described in the following subsections.
An organization or a project must understand what to measure, who is interested in the results, and why. To assist this understanding, it is recommended that a software measurement program model be adopted such as illustrated in Figure 4.1.
The measurement program model provides a simple framework for specifically identifying what software attributes are of potential interest to measure, who the various customers of measurement results might be, and why such measurement results are of interest to those customers. The measurement program model includes the general software objects of measurement interest such as resources, processes, and products. The measurement customers include the end-use customer, software organization and project management, and software application personnel. These customers need software measures for different reasons. Their viewpoints drive the eventual measurement selection priorities and must be integrated and consistent to be most effective.
To establish a successful measurement program (e.g., one that is used for organization and/or project decision-making and lasts more than 2 years), it is necessary to have a basic understanding of measurement.
Figure 4.1
Software measurement program model.
Resources, Products, Processes
Software objects such as resources, products, and processes have attributes that characterize software projects and are therefore of interest to measure. A software measure is an objective assignment of a number (or symbol) to a software object to characterize a specific attribute.
Resources are inputs to processes. Such inputs specifically include personnel, materials, tools, and methods. Resources for some processes are products of other processes. An attribute of great interest that is relevant to all of these types of resources is cost. Cost is dependent on the number of resources and the market price of each resource. For personnel, the cost is dependent on the effort expended during the process and the market price value of each person assigned to the process.
Processes are any software-related activities such as requirements analysis, design activity, testing, formal inspections, and project management. Processes normally have time and effort as attributes of interest, as well as the number of incidents of a specified type arising during the process. Certain incidents may be considered to be defects in the process and may result in defects or faults in products.
Products are any artifacts, deliverables, or documents that are produced by software processes. Products include specifications, design documentation, source code, test results, and unit development folders. Products normally have size and inherent defects as attributes of interest.
Direct and Indirect Software Measurement
Direct measurement of a software attribute does not depend on the measurement of any other attribute. Measures that involve counting, such as the number of source lines of code (SLOC) and the number of staff hours expended on a process, are examples of a direct measure. Agile methods might use direct measures such as lead time, engineering time, time to change, time to deploy, and time to roll back.
Indirect or derived measurement involves more than one attribute. Rates are typically indirect measures because they involve the computation of a ratio of two other measures. For example, software failure rate is computed by dividing the count of the failures observed during execution by the execution time of the software. Productivity is also an indirect measure since it depends on the amount of product produced divided by the amount of effort or time expended.
Two other very important aspects of the measurement assignment are preservation of attribute properties and mapping uniqueness. The mapping should preserve natural attribute properties (e.g., order and interval size). If another assignment mapping of the attribute is identified, there should be a unique relationship between the first mapping and the second mapping. It is very difficult to ensure that measures satisfy these preservation and uniqueness properties. This document will not consider these issues in any detail.
Views of Core Measures
The three views (strategic, tactical, and application) of the core measures illustrated in Figure 4.1 identify important attributes from the viewpoints of the customer, project management, or applications engineers, respectively. It is extremely important for the measurement program to be consistent across the three views of core measures. There must be agreement and consistency on what measures mean, what measures are important, and how measures across the three views relate to and support each other.
Strategic View
This view is concerned with measurement for the long-term needs of the organization and its customers. Important measures include product cost (effort), time to market (schedule), and the trade-offs among such quality measures as functionality, reliability, usability, and product support. It may be critical to an organization to establish new customers and solidify old customers through new product capabilities—with limited reliability and usability, but with a well-planned support program. Time to market is usually a critical measure, and may become one of upper management’s most important measures.
Agile methods might focus on value: Why are we doing the project? Cost: Can we afford the project and technical debt? Is the execution risk acceptable?
Tactical View
This view is concerned with the short- and long-term needs of each individual project’s management goals. The project measures that support the tactical view should be able to be aggregated to show a relationship to the organization’s strategic goals. If not, then individual projects will appear to be “out of sync” with the organization. The primary measures of interest to project management are schedule progress and labor cost.
Application View
This view is concerned with the immediate resource, process, and product-engineering needs of the project. Resources (e.g., personnel and support equipment) are of some interest in this view, but the engineer is primarily interested in the process activities to produce a high-quality product. The engineering definitions of process and product quality should be consistent with project management or upper-level organization management understanding. Product size, complexity, reliability, and inherent defect measures are important to the engineers because they indicate achievement of functional and performance requirements.
Use a Software Process Improvement Model
In order for a software measurement program to be successful, the measurement activities should be conducted within the environment of continuous software process improvement. Without such an environment, measures will not be seen as value-added and the measurement program will not be sustainable. Two models are important to a software process improvement initiative and the integration of software measurement, as illustrated in Figure 4.2. The initiate, diagnose, establish, act, and leverage (IDEAL) model provides an organization with an approach to continuous improvement. The capability maturity model (CMM) can be used to establish a measurement baseline.
The IDEAL model provides a framework for conducting process improvement activities at the organization level and the project level. The IDEAL model is similar to the plan/do/check/act model.
Organization Software Measurement
Figure 4.2
Software process improvement models.
During the initiate stage, the organization goals and measures for the improvement are defined along with success criteria. The diagnose stage includes baselining the organization’s current process capability (e.g., using the Software Engineering Institute [SEI] CMM during a software process assessment) in accordance with the measures inherent in the assessment process. The establish stage provides focus for identifying specific improvements that will be accomplished by action teams and the measures for those improvements. Prioritized improvement actions are determined and action teams are formed to develop specific plans that address the high-priority improvements. The act stage includes implementation of the action team plan, including the collection of measurements to determine if the improvement has been (or can be) accomplished. The leverage stage includes documenting the results of the improvement effort and leveraging the improvement across all applicable organization projects.
Project Software Measurement
During the initiate stage, the project goals and measures for success are defined along with success criteria. A project software measurement plan should be developed or included as part of the software project management information (e.g., referenced as an appendix to a software development plan). The diagnose stage includes documenting and analyzing the project’s measures as a measurement case during the project life cycle in accordance with the measures in the measurement plan. The establish stage provides focus on identifying specific project or organization improvements that might be accomplished. Prioritized improvement actions are determined and assigned to project or organization level, as appropriate. For more mature organizations, project teams can accomplish the improvements during the project. For less mature organizations, the identified improvements will serve as lessons learned for future projects. Action teams are formed (by the project or organization) and a plan is developed to address the high-priority improvements. The act and leverage stages of the project are limited to making midcourse project corrections based on the measurement information. Such measurement data and the actions taken are recorded in the measurement case. The project’s measurement case then becomes the complete documentation of the project management and engineering measures, any changes to the project direction based on measurement analysis, and lessons learned for future projects.
Software Engineering Institute Capability Maturity Model
The SEI CMM serves as a guide for determining what to measure first and how to plan an increasingly comprehensive improvement program. The measures suggested for different levels of the CMM are illustrated in Table 4.1. The set of core measures described in this document primarily address Level 1, 2, and 3 issues.
Level 1 measures provide baselines for comparison as an organization seeks to start improving. Measurement occurs at a project level without good organization control, or perhaps on a pilot project with better controls.
Level 2 measures focus on project planning and tracking. Applicable core measures are the staff effort and schedule progress. Size and defect data are necessary to understand measurement needs for Level 3 and Level 4 and to provide a database for future evaluations. Individual projects can use the measurement data to set process entry and exit criteria.
Table 4.1 Relationship of Software Measures to Process Maturity
| MATURITY LEVEL | MEASUREMENT FOCUS | APPLICABLE CORE MEASURES |
| 1 | Establish baselines for planning and estimating project resources and tasks | Effort, schedule progress (pilot or selected projects) |
| 2 | Track and control project resources and tasks | Effort, schedule progress (project by project basis) |
| 3 | Define and quantify products and processes within and across projects | Products: Size, defects Processes: Effort, schedule (compare above across projects) |
| 4 | Define, quantify, and control subprocesses and elements | Set upper and lower statistical control boundaries for core measures. Use estimated vs. actual comparisons for projects and compare across projects. |
| 5 | Dynamically optimize at the project level and improve across projects | Use statistical control results dynamically within the project to adjust processes and products for improved success. |
Level 3 measures become increasingly directed toward measuring and comparing the intermediate and final products produced across multiple projects. The measurement data for all core measures are collected for each project and compared with organization project standards.
Level 4 measures capture characteristics of the development process to allow control of the individual activities of the process. This is usually done through techniques such as statistical process control where upper and lower bounds are set for all core measures (and any useful derived measures). Actual measure deviation from the estimated values is tracked to determine whether the attributes being measured are within the statistically allowed control bounds. A decision process is put into place to react to projects that do not meet the statistical control boundaries. Process improvements can be identified based on the decision process.
Level 5 processes are mature enough and managed carefully enough that the statistical control process measurements from Level 4 provide immediate feedback to individual projects based on integrated decisions across multiple projects. Decisions concerning dynamically changing processes across multiple projects can then be optimized while the projects are being conducted.
Identify a Goal-Question-Metric (GQM) Structure
One of the organization’s or project’s most difficult tasks is to decide what to measure. The key is to relate any measurement to organization and project goals. One method for doing this is to use the goal-question-metric (GQM) paradigm, illustrated in Figure 4.3 with a partial example related to software reliability.
This method links software goals to corporate goals and derives the specific software measures that provide evidence of whether the goals are met. Since such measures are linked directly to organization goals, it is much easier to show the value of the measurement activity and establish success criteria for measurement.
Figure 4.3
Goal-question-metric (GQM) paradigm.
The GQM method to software measurement uses a top-down approach with the following steps:
- Determine the goals of the organization and/or project in terms of what is wanted, who wants it, why it is wanted, and when it is wanted.
- Refine the goals into a set of questions that require quantifiable answers.
- Refine the questions into a set of measurable attributes (measures for data collection) that attempt to answer the question.
- Develop models relating each goal to its associated set of measurable attributes.
Some attributes of software development, such as productivity, are dependent on many factors that are specific to a particular environment. The GQM method does not rely on any standard measures and the method can cope with any environment.
This activity may be conducted concurrently with any other software measurement activities and may be used to iteratively refine the software measurement program model, core measurement views, and process improvement efforts.
Develop a Software Measurement Plan
The software measurement program activities provide organization and project-specific planning information and a variety of measurement data and analysis results. These plans, data, and results should be documented through use of a software measurement plan and software measurement case.
A software measurement plan defines
- What measurement data are to be collected.
- How the data are to be analyzed to provide the desired measures.
- The representation forms that will describe the measurement results.
Such a plan also provides information on who is responsible for the measurement activities and when the measurement activities are to be conducted. A software measurement plan should be developed at an organization level to direct all measurement activity and at a project level to direct specific project activity. In most cases, a project’s software measurement plan can be a simple tailoring of the organizational plan. The organization’s software measurement plan can be a separate document or it might be an integrated part of the organization’s software management plan or software quality plan.
A software measurement plan at either the organization or project level should relate goals to specific measures of the resource, process, and product attributes that are to be measured. The GQM method can be used to identify such measures. Improvement in accordance with the SEI CMM key process areas should be an integrated part of the derivation. The identified measures may be a core measure or derived from one or more core measures.
The following activities are key to developing a software measurement plan:
- Establish Program Commitment: Define why the program is needed, obtain management approval, and identify ownership.
- Determine Goals and Expected Results: Use software-process assessment results to set the improvement context.
- Select Project Measurements: Apply the GQM method to derive project measures.
- Develop Measurement Plan: Document the measures to be collected, data collection, analysis and presentation methods, and their relationship to an overall improvement program.
The software measurement case documents the actual data, analysis results, lessons learned, and presentations of information identified in an associated software measurement plan. The following activities are key to developing a software measurement case:
- Implement Measurement Plan: Collect and analyze data, provide project feedback, and modify project/program as necessary.
- Analyze Measurement Results: Store project measurement results, analyze results against historical project results.
- Provide Measurement Feedback: Report results of analysis as project lessons learned, update measurement and process improvement programs, and repeat the process of developing/updating a measurement plan and case.
Example Measurement Plan Standard
This document contains an example of a standard defining the contents and structure of a software measurement plan for each project of an organization. The term measurement plan will be used throughout.
1 Introduction
This standard provides guidance on the production of a measurement plan for individual software projects.
1.1 Scope This standard is mandatory for all projects. Assistance in applying it to existing projects will be given by the organization measures coordinator.
2 Policy
It is policy to collect measures to assist in the improvement of
- The accuracy of cost estimates
- Project productivity
- Product quality
- Project monitoring and control
In particular, each project will be responsible for identifying and planning all activities associated with the collection of these measures. The project is responsible for the definition of the project’s objectives for collecting measures, analyzing the measures to provide the required presentation results, and documenting the approach in an internally approved measurement plan. The project is also responsible for capturing the actual measurement information and analysis results. The form of this actual measurement information could be appended to the measurement plan or put in a separate document called a measurement case.
3 Responsibility and Authorities
The project leader/manager shall be responsible for the production of the project measurement plan at the start of the project. Advice and assistance from the organization measures coordinator shall be sought when needed. The measurement plan shall be approved by the project leader/manager (if not the author), product manager, organization measures coordinator, and project quality manager.
4 General Information
4.1 Overview of Project Measures Activities The collection and use of measures must be defined and planned into a project during the start-up phase. The haphazard collection of measures is more likely to result in the collection of a large amount of inconsistent data that will provide little useful information to the project management team, or for future projects. The following activities shall be carried out at the start of the project:
- Define the project’s objectives for collecting measures.
- Identify the users of the measures-derived information, as well as any particular requirements they may have.
- Identify the measures to meet these objectives or provide the information. Most, if not all, of these should be defined at the organization level.
- Define the project task structure, for example, work breakdown structure (WBS).
- Define when each measure is to be collected, in terms of the project task structure.
- Define how each measure is to be collected, in terms of preprinted forms/tools, who will collect it, and where/how it will be stored.
- Define how the data will be analyzed to provide the required information, including the specification of any necessary algorithms, and the frequency with which this will be done.
- Define the organization, including the information flow, within the project required to support the measures collection and analyses activities.
- Identify the standards and procedures to be used.
- Define which measures will be supplied to the organization.
4.2 Purpose of the Measurement Plan The project’s measurement plan is produced as one of the start-up documents to record the project’s objectives for measures collection and how it intends to carry out the program. The plan also
- Ensures that activities pertinent to the collection of project measures are considered early in the project and are resolved in a clear and consistent manner.
- Ensures that project staff are aware of the measures activities and provides an easy reference to them.
The measurement plan complements the project’s quality and project plans, highlighting matters specifically relating to measures. The measurement plan information can be incorporated into the quality and/or project plans. Information and instructions shall not be duplicated in these plans.
4.3 Format Section 5 defines a format for the measurement plan in terms of a set of headings that are to be used, and the information required to be given under each heading. The front pages shall be the minimum requirements for a standard configurable document.
4.4 Document Control The measurement plan shall be controlled as a configurable document.
4.5 Filing The measurement plan shall be held in the project filing system.
4.6 Updating The measurement plan may require updating during the course of the project. Updates shall follow any changes in requirements for collecting measures or any change to the project that results in change to the project WBS. The project leader/manager shall be responsible for such updates or revisions.
5 Contents of Measurement Plan
This section details what is to be included in the project’s measurement plan. Wherever possible, the measurement plan should point to existing organization standards, and so on, rather than duplicating the information. The information required in the plan is detailed under appropriate headings in the following section.
For small projects, the amount of information supplied under each topic may amount to only a paragraph or so and may not justify the production of the measurement plan as a separate document. Instead, the information may form a separate chapter in the quality plan, with the topic headings forming the sections/paragraphs in that chapter. On larger projects, a separate document will be produced, with each topic heading becoming a section in its own right.
Thematic Outline for a Measurement Plan
Section 1 Objectives for Collecting Measures
The project’s objectives for collecting measures shall be described here. These will also include the relevant organization objectives. Where the author of the measurement plan is not the project leader/manager, project management agreement to these objectives will be demonstrated by the fact that the project manager is a signatory to the plan.
Section 2 Use and Users of Information
Provide information that includes
- Who will be the users of the information to be derived from the measures
- Why the information is needed
- Required frequency of the information
Section 3 Measures to Be Collected
This section describes the measures to be collected by the project. As far as possible, the measures to be collected should be a derivative of the core measures. If organizational standards are not followed, justification for the deviation should be provided. Project-specific measures shall be defined in full here in terms of the project tasks.
A GQM approach should be used to identify the measures from the stated project objectives. The results of the GQM approach should also be documented.
Section 4 Collection of Measures
Provide information that includes
- Who will collect each measure
- The level within the project task against which each measure is to be collected
- When each measure is to be collected in terms of initial estimate, reesti-mates, and actual measurement
- How the measures are to be collected, with reference to proformas, tools, and procedures as appropriate
- Validation to be carried out, including details of the project-specific techniques if necessary, and by whom
- How and where the measures are to be stored—including details of electronic database/spreadsheet/filing cabinet as appropriate; how the data is amalgamated and when it is archived; who is responsible for setting up the storage process; and who is responsible for inserting the data into the database
- When, how, and which data are provided to the organization measures database
Section 5 Analysis of Measures
Provide information that includes
- How the data are to be analyzed, giving details of project-specific techniques if necessary, any tools required, and how frequently it is to be carried out
- The information to be provided by the analysis
- Who will carry out the analysis
- Details of project-specific reports, frequency of generation, how they are generated, and by whom
Section 6 Project Organization
Describe the organization within the project that is required to support the measurement activities. Identify roles and the associated tasks and responsibilities. These roles may be combined with other roles within the project to form complete jobs for individual people.
The information flow between these roles and the rest of the project should also be described.
Section 7 Project Task Structure
Describe or reference the project’s task structure. It should be noted that the project’s measurement activities should be included in the project task structure.
Section 8 Standards
The measurement standards and procedures to be used by the project must be given, indicating which are organization standards and which are project specific. These standards will have been referenced throughout the plan, as necessary. If it is intended not to follow any of the organization standards in full, this must be clearly indicated in the relevant section of the measurement plan, and a note made in this section.
In Conclusion
This final section provides examples, summarized in Table 4.2, that illustrate the use of the recommended core measures (with some minor variations) for a variety of software projects.
Table 4.2 Core Measures for Example Projects
Note
* This chapter has been adapted from the book Leading IT Projects: The IT Manager’s Guide by Jessica Keyes. Auerbach, 2008.