CHAPTER 5
Implementing a Metrics Program
The goals of a metrics system can vary in urgency, as determined by the strategic objectives of the organization (see Figure 5-1). The sophistication of the metrics system—and its funding—likewise can vary depending on whether the organization’s overall goals are to improve project-by-project performance, divisional project performance, enterprise project performance, or enterprise project management maturity. (Rad and Levin 2002)
If the goals are division-specific, then the organization might highlight the current maturity rating of the division together with the desired rating. Improvement objectives would then be stated either as an incremental advancement or as a collective achievement goal. Thus, the goal might be, independent of current status, to improve performance in all project management process areas by one maturity level following a staged model. Alternatively, the goal might be to attain a certain level (e.g., Level 4) in project performance on all projects. If the metrics goals are project-specific, then the goals might be tied either to project success ratings or to overruns in cost and schedule.
Because so many aspects of a project can be measured, the opportunities can be overwhelming. In some organizations, measuring the values of the indices of the model can become an end unto itself. To avoid this, metrics procedures should not be “check-the-box” processes, because such simplistic reports usually exist to satisfy a perfunctory scheduled review or audit. Rather, a metrics system should be regarded as a tool to show a sustained organizational benefit achieved primarily because of a project management culture.
Metrics needs to be viewed not as one-time exercises, but as continuing initiatives toward enhanced productivity and effectiveness. Thus, the focus of using a metrics system is not to provide a quick fix for projects in trouble, but to establish the pathway for a journey to excellence.
To that end, sometimes it might be necessary to reexamine current metrics with specific applicability to projects. For example, indices such as profitability usually come too late for mid-course corrections and remedial actions. Therefore, financial and non-financial information should be integrated in a usable way. In planning a metrics program, it is important to identify the most important project and process management issues, select and define the corresponding metrics, and integrate these metrics incrementally into existing processes.
Figure 5-1
Goals of a Metrics System
Sometimes metrics might be collected only to fulfill a corporate requirement to report on information on certain arbitrarily targeted dates or to meet a specific contractual reporting requirement. If the metrics system is not designed and implemented properly, the organization might not collect any data, might collect data but use only the portion that can be gleaned, or might collect data for a lot of metrics that have no valuable use. Each of these circumstances should be avoided.
Metrics must be well defined, and guidelines for their use must be fully accepted by those who will use them. The project management metrics program should be designed and implemented in such a way that project personnel will come to regard metrics as the basis for actions that support project management excellence and overall organizational improvements. Data provided by the metrics system can become the basis for informed analysis only when there is agreement on what is happening and what should be happening in projects.
With the help of a well-planned metrics system, project team members will come to recognize both the long-term benefits of metrics and the obstacles that could exist during the implementation of a metrics program. To facilitate successful implementation of a metrics system, communication about metrics initiatives must be direct, consistent, and widespread. The utility of a metrics system must be visibly communicated throughout the organization to establish commitment to sustaining the program. Whoever is responsible for the program should have strong communications skills and be able to balance the metrics program’s needs with the participants’ readiness to accept and embrace change.
Each person’s involvement in the metrics program should be known and communicated. Every effort should be made to ensure that program implementation is not an organizational secret, as that status will breed notions of conspiracy, especially if the organization is undergoing downsizing, outsourcing, or reorganization.
Project professionals who have achieved project management certification will recognize the value of a well-crafted project management metrics system. However, there might be cases where some project personnel or other project stakeholders do not fully embrace the value of metrics. Ongoing training may be necessary to keep the metrics program in focus and on track.
Implementing a metrics program generally requires a cultural change within the organization to help project professionals overcome inherent resistance to monitoring project performance and to widespread reporting of organizational results. In such cases, the initial purpose of a metrics system might be simply to provide project professionals the tools to communicate the benefits of project management to senior executives and operational personnel. Moreover, time must be set aside for stakeholder meetings to discuss the added value that a metrics system brings to the organization. A number of briefings and workshops may be required before people throughout the organization recognize the purpose of the metrics system.
METRICS PROGRAM DESIGN
Implementation of a formalized metrics system must be treated just like any other project, perhaps even more so, to highlight the advantages of effective planning and monitoring. Thus, the steps associated with a project management process must be followed in implementing a metrics program. The metrics system design must include a detailed implementation plan with procedures and policies for using the results derived from the system. Moreover, the data from the metrics system must be fully incorporated into existing policies and procedures. (Instrument 5-A)
Clever data presentation and intricate analysis methods should not be the goal; often, they are not worth the effort (Lucero and Hall 2001). Rather, the primary objective should be to allow managers to derive information as early as possible. Thus, the results of the data collection process should be reported as soon as practicable, because data that are reported too late are essentially useless.
As part of planning the metrics system, the success criteria for the metrics system, and for project management, must be developed. In tangible terms, the metrics system’s impact on the following must be identified: individual projects, programs, the project portfolio, morale improvement, and operational benefits.
Since a metrics system is a collection of relatively simple pieces of data from several facets of a project, care must be taken to ensure that the constituent indices and models of the system are directly related to the project attribute they are intended to characterize. To add utility to the indices, they could be packaged within a predictive model.
Predictive models can be mathematical or graphical in nature, depending on the organization. When considering complex environmental issues, models are far more likely to predict realistic project outcomes than any single metric. For example, the amount of money spent is only one facet of a project’s progress. The resources used, the cost of those resources, and the number of deliverables completed provide a better picture of a project’s progress. All models, even very sophisticated ones, are only partial representations of the reality they attempt to portray.
Quantitative data collected from the attributes of things or even people are usually assumed to be accurate and unbiased because they are expressed as precise numbers. However, many seemingly objective data are influenced by someone’s subjective judgment. Designers of metrics systems should be mindful that project professionals will collect many data items that reflect some degree of subjective judgment. Parenthetically, it is hoped that this subjective judgment is fully in line with the guidelines of professional responsibility.
Consider the example where data on the number of design units that have passed an integration test are presumed to be precise numbers determined by the technical characteristics of the software under test. However, the number of tests conducted depends primarily on a team member’s interpretation of the criteria used for the test. Changes to the integration test criteria could come from the team member’s interpretation of the level of assembly, the size of the software components that are integrated for each test case, the number of inputs or conditions required for each test case, and the level of conditional stress during the test (such as concurrent tasking, shared memory availability, and process utilization) (Lucero and Hall 2001). Furthermore, qualitative indices will be affected not only by these issues, but also by the fact that their measurement is far more dependent on the judgment, viewpoint, and expertise of the examiner.
Planning for metrics systems, and using their specific metrics, require a determination, in realistic terms, of the status of the organization’s environment. To characterize the organization’s attitude toward change, a determination must be made as to whether the culture is project oriented or specialty driven. The key features of current practices will form the foundation and logic for how projects are selected, how they are managed, and the specific methodologies that will be used.
Current corporate practices also must be detailed in order to determine the extent to which there is tolerance for inaccurate project data and project overruns. Top management can provide insight into the current corporate and business functions of business units and the duties of personnel within those units. In addition, key knowledge areas must be identified, primarily because the operation of the metrics system, and its constituent indices, must match the company’s strategic project management needs. Likewise, it is essential to involve the Project Management Office (PMO) in the process from the very beginning of the implementation phases, even if the metrics are not fully a part of the PMO’s function.
Metrics should inspire confidence by virtue of repeatable accuracy and being rooted in valid assumptions, concepts, and calculations. Since the utility of each metric extends beyond the boundaries of a given project, collection and reporting of data should be part of the organization’s policies.
When designing a metrics system, the goal should be to cover not only the project deliverables but also the entire enterprise. Collecting quantitative planning and progress data should not be limited to large projects; it should apply to all projects regardless of size and complexity. Naturally, the set of indices and models comprising the metrics system will vary somewhat depending on the size and nature of the project.
It is advisable to develop common definitions for key project activities and milestones. Such uniformity facilitates the development of a sophisticated Work Breakdown Structure (WBS) (based on templates from past projects), a typical WBS Dictionary, an organizational Resource Breakdown Structure (RBS), and a set of standard report templates. High-level summary reports and cross-functional reports must include detailed data on conflicts between projects arising from overallocation of resources.
If formalized metrics are not used consistently throughout the organization, they will not be employed beyond the span of the project for which they were developed. Metrics, when using standard and consistent scales organizationwide, are powerful tools for informing stakeholders of progress in meeting project objectives. They gain further utility when used to compare like items across multiple projects, as they form a logical basis for suggesting needed changes in ongoing efforts. Unfortunately, in organizations that have not reached Level 2 or Level 3 of a staged maturity scale, it is the project manager who establishes the methods for monitoring project status, and the luxury of comparing data across projects is not available.
Goldratt (1990) notes, “Tell me how you measure me, and I will tell you how I will behave. If you measure me in an illogical way, do not complain about illogical behavior” (26). This statement implies that the way measurements are taken may cause people in the organization, or the organization as a whole, to behave in a given manner. Goldratt further warns: “Change my measurements to new ones that I don’t fully comprehend and then nobody knows how I will behave, not even me” (88). Thus, as organizational processes for the new suite of metrics are being developed, earlier data collection processes must be reviewed in light of their verified utility to projects.
It is useful to identify key pieces of information that are not included in the current system in order to devise means of incorporating them into future versions. Finally, it is helpful if the implementation team validates and improves planned processes before collecting metrics data. Even then, it might be two years or more until the capture and analysis of data yield useful results.
ATTRIBUTES OF MODELS AND INDICES
The main purpose of using a single metric, or a suite of metrics as part of a comprehensive model, must be decided before collecting the related data. Definitions of each index, and eventually of each model, should begin with a precise description of the project management issue that is to be measured. The direct or indirect linkage among the metric, the project goals, and the organizational goals must be highlighted (see Figure 5-2).
The thresholds of the metric that define normalcy (i.e., the range of data that includes normal behavior) should be clearly defined. Beyond that, one threshold should define a point considered excessive, where remedial measures must be put in place to bring the project back on course. Another threshold should define exemplary performance, attainment of which should be noted, applauded, and replicated across the organization.
Figure 5-2
Characteristics of a Metric
Celebratory thresholds include early delivery, notable cost underrun, or unusual technical quality of the deliverable. On the other hand, danger thresholds should signal an unacceptable cost overrun, late delivery, and a seriously defective project product. Finally, each time a new metric is introduced, the following information should be detailed and explained fully: data collection process, data collection frequency, results dissemination process, and corresponding feedback policies.
In addition to a list of values that describe normal performance, exceptional performance, and unacceptable performance, the definition of each index must include a corresponding plan of action for cases where the threshold of tolerance for that index has been breached. For example, it might be specified that cost performance will be rated as superb if there is an underrun of 10% or more. Such performance should be applauded, with proper recognition to the project manager and team members.
On the other hand, cost performance might be rated as unacceptable if there is an overrun of 150% or more. When this threshold is breached, organizational project recovery measures must be put in place to bring the project back on a stable track. To extend this concept further, it is important to determine how many current projects map onto this range of acceptability and how many should map onto this range once the metrics system is fully in place.
Project managers should know which index has the most impact on their project’s performance monitoring. Some attributes that are easy to measure might not be particularly important. Attributes that are more difficult to measure might be more important. Therefore, metrics should be assigned a priority ranking, and all metrics should not be treated equally.
It is useful if individual metrics, or the metrics system as a whole, have predictive capabilities. Thus, the metrics system can become a predictive tool and an early warning system for the project’s mal-performance, and for those cases where there is a mismatch between the deliverable and market needs.
At a minimum, the index should have the capability to measure a specific performance attribute. The metric also should have features that can be integrated with other metrics in order to synthesize overall project performance or even overall enterprise performance. The following issues should be considered:
• Clarity: Can everyone understand what is being measured, how data are collected, and how the results should be interpreted?
• Repeatability: Can someone else perform the same measurement and get the same result?
• Traceability: Are the origins of the data identified in terms of sequence, activity, status, environment, and tools used?
• Precision: To what degree of accuracy does the metric express the item being measured?
Furthermore, a metric system should respond to the following questions for each index:
• What is the attribute or subject of the metric?
• Is the right attribute being measured?
• How often is the measurement performed?
• What are the thresholds of the metric?
• Who are the people who need to be informed of a specific metric’s values?
• How can we motivate the team with this metric?
SCHEDULE OF IMPLEMENTATION
The metrics system should be designed specifically with an eye toward the collection of data for project and enterprise success factors. The project metrics program might choose to begin its implementation with a focus on things attributes of only one project. Then, building on the success of this mini-implementation, data collection and reporting can be extended to people attributes of the project, and eventually to things-people attributes of more than one project (see Figure 5-3).
Ultimately, the metrics program can be expanded to include attributes of the enterprise that encompass all projects. Further, by conducting periodic assessments, the metrics system can benchmark the organization’s current maturity level to the target maturity level and to industry standards of best practices. These interim assessments will chart a course for the full sophistication of the organization by identifying areas that are in need of improvement and by demonstrating the effectiveness of the metrics system in achieving the ultimate goal.
During the early stages of metrics system implementation, specific expectations must be defined for the organization concerning metrics and how they will be used. The plan must specify incremental progress milestones, such as first visible impact, intermediate significant achievements, and completion target. To reap the full benefits of successful implementation of a metrics system, it is often useful to implement the system in stages, because even with sizeable investments, it will take several years before sustained improvements are observable in the organization’s project performance.
Figure 5-3 Metrics Suite
Enterprise
The overall process includes the following steps:
• Step 1: Establish the Vision and Strategy. The metrics system can be an add-on to existing methods of project management monitoring and forecasting in an organization, or it can be new and revised indices for organizational business attributes and project management. To define the metrics system strategy, the organization’s key factors for success, competitors’ programs and priorities, the organization’s strengths and weaknesses in light of the competition, and likely external changes must be assessed.
• Step 2: Prepare an Execution Plan. This plan should include a transition plan for the metrics system. The plan should detail organizational interfaces because barriers between organizational units need to be eliminated to facilitate knowledge management. The plan should guide the implementation of the metrics system, eliminate or reduce uncertainty in roles and responsibilities, document assumptions and constraints, provide a basis for monitoring and controlling the metrics system implementation, and facilitate communication with stakeholders.
• Step 3: Implement the System. The metrics system’s initial focus should be on areas involving the greatest impediments to improved project results. Since project management processes and practices will need to be modified, staff from all stakeholder organizations should actively participate in the implementation. Greater project personnel involvement will foster greater commitment, increased sharing of lessons learned, improved coordination, and early warning of problems. The implementation team must conduct reviews of the metrics system’s effectiveness in the light of continuous improvement initiatives. Then, the implementation team must communicate the results and progress to all stakeholders.
When implementing a new metrics system, or simply compiling and formalizing an existing one, it is important to strike a balance between the organizational culture and the best practices of project management. Metrics can serve as a facilitator of continuous improvement efforts because they provide an unbiased indication of the current status and a clear description of the desired future. (Instrument 5-B)
INSTRUMENT 5-A
Metrics Collection Checklist
Use this instrument to document the rationale for selecting a metric. This instrument should be reviewed and enhanced regularly, as should the corresponding records of each metric.
What is the purpose of the metric?
What best describes the metric?
Where should the metric be used?
Where should the metric not be used?
Who will collect the data?
Who will verify the data?
How should the data be collected?
How often should the data be collected?
How should the data be analyzed?
Who should analyze the data?
What is considered a normal value?
What is the lower threshold?
What is the upper threshold?
INSTRUMENT 5-B
Metrics Implementation Checklist
Use this instrument as a permanent record for the selection and use of a given metric. This form should be reviewed regularly to identify enhancement opportunities.
State the purpose of each metric.
Describe the benefits of collecting each metric to the enterprise.
Determine whether there are any special considerations relevant to the metric.
Determine whether the metric is required to be collected on every project, or whether the Project Manager has flexibility in terms of collecting and reporting the needed data.
Determine who is responsible for collecting the required data.
Determine whether those responsible for data collection will require any special training.
State the time in which the metric is to be collected.
Describe the analysis process associated with the metric.
Describe how results from the metric’s data collection are to be used.
Determine how the results will be integrated with the other metrics in the system.
Determine the ease of interpreting the results of each metric.
Determine the usefulness of the results of each metric.
Assess the cost of implementing each metric in the system.
Determine if different metrics are needed for different phases of the project life cycle.