The Confusion Surrounding Metrics

While most organizations and program managers specifically, recognize the importance of metrics in general terms, the lack of use and inconsistency of use is more prevalent in practice. We have observed what Norman Fenton and Martin Neil noted in their research, which is that metrics and activities associated with metrics “have not addressed their most important requirement: to provide information to support quantitative managerial decision-making.”² So why not? The answer is confusion. It is easy to get overwhelmed by the complexity of metrics, especially now with data becoming easier to obtain. Just because data are easy to obtain does not mean they are useful. We will make sense of translating data into useful information in the coming sections. For now, let's address the confusion as it pertains to nomenclature.

Much of the challenge in designing, developing, and establishing a useful performance management system is in part due to confusion in using performance terms such as objective, metric, measure, indicator, critical success factor, and more. Some authors use these performance management terms seemingly interchangeably, while others delineate them with a level of sterile technicality. Although authors may not agree, effectively establishing a performance management system in an organization requires a common language of terms and meanings. The following defines and explains key performance management system terms.

Objective: A specific target to be achieved most often associated with a goal or strategy. A major league baseball player may have, as an example, an objective to achieve a .300 batting average or hit 40 home runs and steal 40 bases. From a business perspective, an objective may be to achieve 10 percent new market penetration in each of the next three years.

Business Success Factor: Many practitioners use the term critical success factor (CSF). D. Robert Daniel, of McKinsey & Company, introduced the term in 1961 when his article “Management Information Crisis” appeared in Harvard Business Review.³ We, however, use the term business success factor (BSF) so as to keep focus on business outcomes—it's all about the business. This approach is consistent with Daniel's work in that he highlighted CSFs as types of data useful in measuring and informing top management activities. Therefore, critical business success factors are organizational conditions, practices, or elements necessary for success. For example, a critical business success factor may be to acquire new customers by expanding use of social media.

Metric: A quantitative measure with an agreed-upon dimension. One inch, 132 pounds, and 4 meters are all examples of metrics. Each is specific and has an agreed-upon quantitative meaning. An inch, for example, is one-twelfth of a foot or 2.54 centimeters. From a business perspective, return on investment (ROI) of 14 percent, market penetration of 21 percent, and deliverable efficiency to schedule of 98 percent, are examples of metrics.

Measurement: A quantification process, calculation, or algorithm that yields a numeric score. For effectiveness, a measurement yields not only the score, but aligns that score back to strategy and objectives. Business leaders and investment analysts are often measuring organizational return on assets, market share, and cash flow regarding organizational performance and alignment with strategy and objectives. Again, for clarity purposes, measurement is the process of measuring metrics (the actual performance) and relating them to strategy and objectives (the performance target).

Indicator: A prescribed state(s) to notify, alarm, or warn of conditions. The use of a household smoke detector is an indicator. Another example of an indicator is the engine light on the dashboard instrument panel of a car. The indicators may go off by signaling an alarm (visually or audibly), which prompts action. From a business perspective, examples include cost or schedule overruns or underruns relative to prescribed thresholds. Early warning systems such as the program strike zone and program indicators (Chapter 9) are valuable—the earlier the warning, the better.

Rationale for Using Metrics

There is one main, often cited, and logical reason for using metrics. It is common knowledge that “what gets measured gets managed; and what gets managed gets improved.” All organizations want to achieve continuous improvement. Those actively, and effectively, using performance metrics are more likely to succeed with continuous improvement. John Hauser and Gerald Katz illustrated this quite nicely in their article “Metrics: You Are What You Measure!” Here is how they explained it:

The link is simple. If a firm measures a, b, and c, but not x, y, and z, then managers begin to pay more attention to a, b, and c. Soon those managers who do well on a, b, and c are promoted or are given more responsibilities. Increased pay and bonuses follow. Recognizing these rewards, managers start asking their employees to make decisions and take actions that improve the metrics. (Often, they don't even need to ask!) Soon the entire organization is focused on ways to improve the metrics. The firm gains core strengths in producing a, b, and c. The firm becomes what it measures.⁴

As illustrated in Figure 8.1, metrics measure performance, and that performance yields output of work effort and resource investment, which in turn determines business results. The results measure the degree to which a mission is accomplished or not.

In particular, using a performance management system helps program managers and their sponsors understand the health of a program. Metrics quantify the degree to which objectives and overall strategy are being achieved. Indicators help to pinpoint if, where, and why a program has problems, which also helps to understand root causes of problems to tailor specific actions to further mitigate or eliminate them. Therefore, when done properly, devising a performance management system and employing appropriate metrics improve the probability of program success and business benefit.

It is important to note that metrics can only tell part of the story. They cannot fully identify, explain, or predict what will happen. Most potential problems need more than one metric or source of information to properly characterize or understand an issue or problem. It is a situation analogous to assembling the pieces of a puzzle. One piece of the puzzle does not depict the image of the whole puzzle.

A useful and effective metric should be timely, accurate, relevant, objective, and presented in a useful format to aid in management and decision making. Metrics should support, not replace, good management judgment.

Metrics and Program Governance

Metrics play a role in support of the organization's governance processes and procedures. As pointed out in PMI's The Standard for Program Management, “To support the organization's ability to monitor program progress and strengthen the organization's ability to assess program status and conformance with organization's controls, many organizations define standardized reporting and control processes applicable to all programs.”⁵ Metrics identified and developed jointly between the senior management team and the program governance personnel are used as part of the monitoring and control for each program and represent the unique needs for that organization.

Those governance metrics and measures, some of which are identified in this chapter, will be normally categorized to cover:

• Planning: priority setting, funding allocation
• Execution: progress tracking, early warning indication
• Completion: assessment of deliverables, benefits
• Results Evaluation: closed loop correction, strategy assessment

Every firm's senior management, along with personnel responsible for administering their governance processes, is faced with the challenge of identifying the right metrics and achieving balance between effective control and monitoring with the overhead cost required to administer them. The investment in time needed to capture data and use meaningful metrics must be balanced against the value obtained. If, as stated earlier in the chapter, what gets measured gets improved, organizations must guard against measuring and monitoring the wrong or superfluous aspects of the program.

Using Strategy as a Compass

As noted, metrics must be aligned with the specific strategy and objectives of the firm to be effective. As a reminder, it is the business strategy that dictates the configuration and focus of a company's program management function. This means that each element of program management needs to be aligned with the business strategy and reflected by the program management metrics used. To illustrate this alignment, we will look at three companies with three different business strategies as defined by Michael Porter's model, and demonstrate how the strategy drives the selection of the metrics.¹¹

Company A in Table 8.3 uses a differentiation strategy to offer its customers something different from their competitors. In particular, Company A focuses on technology innovation and fast time-to-market to achieve differentiation and gain higher profits and market share. Secondarily, Company A strives to achieve high quality and does not allow runaway cost. To measure performance to their strategy, Company A selects metrics for measuring time-to-market, percentage of new technologies developed, profitability index, and market share. Additionally, they choose customer fallout and development cost metrics to measure their secondary strategy of providing high quality and cost containment.

Company B focuses on a low-cost strategy aimed at establishing a sustainable cost advantage over its rivals. It competes in the manufacturing of fasteners used in building construction and uses a low-cost strategy to underprice rivals and capture market share. This strategy will leave Company B with a small gross profit margin per product, but profit can be achieved through a large volume of sales and lean staffing. Metrics to support this business strategy should measure development and manufacturing costs, market share, program staffing level, gross profit margin, and sales volume. Secondary objectives are to bring the product to the market in an average industry standard time and with average quality. This is complemented with the secondary priority metric of development time and defect assessment rate. Table 8.4 summarizes the alignment of strategy and metrics.

Company C aims to become a low-cost provider with high-quality features and focuses on a best-cost strategy by combining upscale features with low cost. A secondary strategy is to develop innovative information technologies within reasonable time frames. Primary metrics to measure this strategy should include defect rate assessment, customer fallout, development cost, and manufacturing cost. Secondary metrics could include percent of new technologies delivered and development time. Table 8.5 summarizes the alignment of strategy and metrics.

These examples illustrate how business strategy drives the choice of metrics. We have shown three examples of companies with different strategies and the corresponding sets of metrics chosen. Each company in the world will have its own business strategy and set of metrics to measure the effectiveness and achievement of its strategy.

Metrics Quality

As noted at the beginning of this chapter, metrics can do as much harm as they can good if they are not chosen and used properly. We express the need for quality when using metrics in order to overcome the common errors of metrics management and performance management systems. Best practice companies design their metrics systems as minimalistic and simple as possible; yet balance few and simple with the need for decision-making information. Sometimes, few are trumped by many because of the program and business need for the metrics.

As a quick reference guide (and mnemonic tool) to help overcome limitations associated with metrics, follow these “SMART” guidelines from George T. Doran.¹²

Specific: Objectives and metrics need to be clear and unambiguous. All stakeholders should be able to (consistently) define the metric relative to what is expected, why it is important, and how it is measured.

Measurable: The objectives must be timely and measurable. Therefore, data must be available for use and consistent to track (and trend) over time.

Attainable: Objectives must be realistic. Goals and objectives should include some that are high probability of occurrence and some that are a stretch for achievement.

Relevant: Goals, objectives, and metrics must be aligned from strategy through execution. In other words, they must be relevant. Additionally, to be relevant, metrics must be regularly used, otherwise the time to collect and illustrate metrics is not worth the investment of time and energy.

Time-Bound: Metrics must be grounded and the most natural way to do so is by time. Time provides a sense of urgency, especially if corrective action is needed in between metric reports.

Many organizations over-compensate when using metrics. Organizations not “SMART” about using metrics often find they are committing one or more of the common errors noted at the beginning of the chapter. When the errors occur, frustrations cascade from projects to programs to the organization at large. Quality in the use of metrics relates to those that are the fewest needed, the simplest to manage, and are used regularly in discussions and for decision making. Additionally, quality in metrics includes those that have a clear owner and point of responsibility.

Metrics and Decision Making

There are hundreds of decisions to be made during the course of a program—some large and significant, some small and incremental, and some that fall between these two extremes. It is well documented that having access to the right data at the right time facilitates effective decision making.¹³ Difficulty in finding information to support a decision, or even the perception of difficulty, can affect decision quality on a program. Decision makers often trade off decision quality versus time to find the information needed.

Having an effective metric system for a program serves to bring the data and information needed close to the people in charge of making decisions on a program. Having the right metrics is critically important, however.

Selecting the metrics to use to drive information-based decisions involves determining the data needed to measure and quantify a decision—or more accurately, the decision criteria. Further, decisions at the program level must support the business goals and program objectives. Therefore, a tried and true approach for selecting one's metrics remains valid for program management.¹⁴

1. Establish the goals and objectives.
2. Determine the necessary decisions and decision criteria.
3. Select metrics that will measure the criteria.

Utilizing this approach ensures that both program decisions and metrics support the goals and objectives of the program.