Business understanding and experience leads to the following question:What is the business rationale for using metrics?

There are at least two logical and often cited reasons for using metrics, with one less obvious reason explained later. First, as stated earlier, what gets measured gets improved. Second, metrics measure performance toward the achievement of the business results intended, as well as other program critical success factors.

Metrics measure achievement of the critical success factors of the program. Programs using comprehensive metrics to measure and monitor performance will have fewer problems, hence, higher success in accomplishment of program goals. For example, recent studies cite metrics as a key to programs and project success.^[105] Additionally, a recent study by one of the authors involving 229 programs and projects found that companies using consistent, tiered, balanced, and mutually aligned metrics outperformed companies who used sporadic, nontiered, schedule-oriented, and nonaligned metrics. The use of program metrics is an institutionalized practice of leading companies. They focus on regular, periodic measurements of program and business performance. Other companies still favor schedule-only metrics that are sporadically utilized, normally when a program is in trouble. Additionally, leading companies do not apply the same metrics throughout the PLC.^[106] Programs have different priorities and emphases in different phases of the life cycle of the program.

Figure 9.1. Best versus rest-in-class in the use of program management metrics.

Figure 9.1 demonstrates the key differences between companies that methodically design their set of program metrics (which we refer to as best-in-class companies). There are also those that take an ad hoc approach (which we refer to as rest-in-class companies). Best-in-class companies experience both a higher rate of program goal accomplishment and produce a higher-quality metric set—quality that is quantified by usefulness.

A detailed comparison between the best-in-class and rest-in-class companies revealed that the best companies engineered and installed a system of metrics that were balanced and mutually aligned. Balanced metrics are those that cover all dimensions of a program, including schedule-oriented metrics, as well as those for financial, customer, process, and human resource utilization. Balanced metrics also include both leading and lagging metrics. Leading metrics are forward looking, such as a projected finish date based upon the rate of milestone completion. Lagging metrics, such as percentage of deliverables completed, are most valuable for a retrospective view of program performance. Both leading and lagging metrics are important and useful.

Mutually aligned means that metrics are compatible, using the same baseline information. For example, performance-to-planned schedule, probability of completing the program by a certain date, and the cumulative percentage of milestones that are accomplished are all based on the same baseline—the program schedule.

Literature claims that this behavior, using balanced and mutually aligned metrics, is not by chance but by design and is aimed toward enhancing success in terms of accomplishing the program goals.^[107] Consequently, best-in-class companies translate program success criteria into specific metrics the program team can act on and also create incentives for accomplishment of the metrics. For example, the executive team at Lucent Technologies studies the best time-to-market performance measures in the industry, which are turned into targets for their own teams to beat. This is a culture of continual improvement by making program targets highly aggressive.^[108]

To reflect how metrics vary, successful companies create metric tiers. Tiering allows the companies to categorize a diverse continuum of programs and projects, while concentrating on crucial features. One company uses the following three tiers for its research and development programs: basic research programs (tier 1), technology development programs (tier 2), and specific programs (tier 3). Quality of the research and percent of goal fulfillment are examples of tier-1 metrics. Number of deployable technologies and breakeven after release exemplify tier-2 metrics. When it comes to tier 3, good examples are the number of customers who found defects and time to market.

In summary, best-in-class organizations carefully build metrics systems that align program execution with the organization's business strategy. In that effort, the emphasis is on using metrics to measure performance on a consistent basis from inception to completion of the program. They insist on measuring multiple facets, balancing metrics to obtain a holistic picture of program health, and selecting metrics that are aligned and compatible. Finally, by dividing metrics into tiers, leading companies define metrics for use by all levels of management.

This brings us to the third, and less obvious, reason for using program management metrics. One of the most fundamental duties of senior management is to account for, and use as effectively as possible, the corporate assets which he or she oversees. To date, senior managers have not designed a methodology that enables satisfactory evaluation of their program management assets, except in a passive fiduciary way. Hence, top corporate managers have limited capability to evaluate one of the strongest competitive weapons a corporation may build—its program management discipline.

The real value of the program management discipline is only obvious when we take a close look at the role program management plays in the development of product, service, or infrastructure capabilities. However, one cannot judge the intrinsic value of program management by just looking at the value of recently developed new products, services, or infrastructure capabilities. Measuring program management performance and effectiveness requires a shared understanding between all stakeholders and their participation in the evaluation process. Participative roles will vary between stakeholders, but, in general, only a high level of interaction between these members can lead to the establishment of credible metrics of value-producing performance.

We have constructed an approach that enables stakeholders to get involved in a relevant and direct fashion to ensure that program management performance and effectiveness are adequately measured—the PMVP. All stakeholders, including CEOs, senior management, program management directors, functional managers, program managers, and project managers will all benefit from use of the PMVP.

In Chapter 3, we explained the integrated management system and its subsystems that describe the program management capability of a company—strategic management, portfolio management, program management, project management, and team execution. The subsystems are used to build the PMVP, as shown Figure 9.2 Metrics related to each subsystem allow the PMVP to serve as a tool to analyze the performance of the program management function and to be used as a guide for continual improvement efforts.

Figure 9.2. The program management value pyramid (PMVP).

The PMVP's foundation is built upon three premises that should be in place to create and sustain program management as a competitive advantage for an enterprise:

These premises indicate the following: First, the ability of program management to create value for the corporation should be measurable; second, how well programs and program management are aligned with the business goals and strategies should be measurable; and third, how well program management executes its function should be measurable. If program management performance can be effectively measured in these areas, one has the ability to evaluate the overall value of program management within the firm.

The bottom level of the pyramid, project management and team execution, contains the tactical elements of the PMVP. As explained in Chapter 3, project management and team execution form the basis for planning, implementing, and delivering the interdependent elements of the product, service, or infrastructure capability. This is where the hands-on work gets completed. The functional project managers are responsible for the detailed planning and execution of the project deliverables pertaining to their respective operating functions (for example, hardware engineering, software engineering, and marketing). Metrics that assess the adequacy of the tactical elements would be those that permit measurement of parameters, which are important to practices that support the program management capability.

The second level in the pyramid relates to program management. As described in Chapter 3, it is not by accident that program management is in the center of the integrated management system and PMVP. Program Management is the "glue" that translates business strategic objectives into tangible products, services, or infrastructures that are developed by integrating the work efforts and deliverables of multiple project teams. Effective program management metrics focus on the program's ability to transcend across strategic planning and execution.

The third level is program portfolio management. Portfolio management is an effective process for choosing the most valuable programs, balancing and prioritizing them, and making sure they are aligned with the strategic goals of a company. Various metrics are available for evaluating the effectiveness of the portfolio management process within an organization.

The highest level of the PMVP is strategic management. This level deals with the question of whether program management is a viable part of a company's business goals and strategy. Some appropriate metrics are covered in the PMVP that can be used to answer this question.

Table 9.1 presents a menu of the most commonly used metrics for each level of the PMVP, as they pertain to program management. The metrics are categorized by what is being measured (financial return, strategic alignment, and so on) and by the PMVP level (strategic management, portfolio management, and program management, and project management). A single metric may relate to more than one PMVP level, which is indicated in the table.

The menu was developed from our experience with specific companies; therefore, it is not a comprehensive list of all possible metrics. Rather, we present metrics that we have found to be the most generally representative and useful for the managerial functions in the PMVP.

The heading in column one of the table, metric, identifies the metric being presented; the heading in column two, definition, provides the definition of each metric; and the heading in column three, usage, explains the purpose and how to use the metric.

In practical terms, it is highly unlikely that any one firm would, or could, use all the metrics presented in Table 9.1. It is presented as a menu of metrics. The listing should be seen as a restaurant menu that presents many dining options. From a restaurant menu you choose and order only a subset of the available menu items—seldom all of them. Similarly, from the menu of metrics you choose only those metrics that you need. We will explain later how to customize a metrics set for a specific company's needs.

We'd like to remind the reader once again that Table 9.1 is a menu of common metrics used in practice and does not represent that all the metrics should be used on a program. Each senior management team and PCT must select a set of metrics for their particular measurement needs. A common question we are asked is how many metrics do we need? The answer to this question is that it depends. A study by Business Week and The Boston Group shows that the sweet spot is somewhere between 8 and 12 metrics, which was Chu Woosik, a senior vice president of Samsung Electronics confirmed.^[109] See the box titled, "How Many Metrics Do You Need?" for additional guidance.

In developing the PMVP, our focus was on those measurements that relate to program management's contribution to building and sustaining a business-unit's competitive advantage and the improvement of business results. The PMVP provides a holistic approach for evaluating the strategy for obtaining business objectives, how well a company's development funds are invested, the value of program management to the company, the effectiveness of program management practices, and how individual programs are progressing.

The PMVP is designed to function in a top-down fashion, level by level, to create a balanced and aligned set of metrics that cover multiple dimensions of product, service, or infrastructure development. The value pyramid is also a dynamic model, meaning the metrics set created will dynamically change as a program progresses through the development life cycle, as different metrics are needed at different points in the life cycle.

The PMVP is applicable to both centralized and decentralized forms of program management structure. In a centralized form of program management, there exists an organizational home for all or a majority of program managers. In this case, the use of the PMVP will be determined between the director of program management and senior management, and the commonality of metrics across programs is established. The decentralized form means that all or a majority of the program managers are assigned to one or more business units, and metrics will be based upon the specific business needs of the business unit.

The PMVP is an effective tool for measuring and guiding continual improvement in an organization. However, utilizing the PMVP does not mean automatic improvement. Metrics can only provide data. It is up to the senior management team of a business to interpret the data presented by the metrics and turn it into information that can be used to develop the necessary improvement actions. Let's look at some of the key questions and considerations for management at each level of the PMVP.

As we mentioned earlier, each company will need a distinct set of metrics, depending on its business strategy. Therefore, the set of metrics will vary by the type of industry the company is part of and by its business strategy (for example, differentiation, cost leadership, or best cost). The PMVP does not provide a mechanism to automatically indicate which metrics to choose; this is the job of senior management.

When a business unit determines its own distinct set of metrics, it should do so in collaboration with company stakeholders. As noted in the following list, each stakeholder will emphasize different metrics, according to their needs:

In addition to selecting from a set of metrics, thought has to be given to how all metrics from the various levels of the PMVP interconnect and which are compatible with one another. It is the job of the senior stakeholders of the business to provide the interconnection and alignment strategy of the set of metrics. Let's be reminded that business strategy dictates the configuration and focus of a company's program management function. This means that each element of program management needs to be aligned with the business strategy—program management metrics included. To illustrate, we'll look at three companies with three different business strategies as defined by Porter's model, and demonstrate how the strategy drives the selection of the metrics.^[110] Company A in Table 9.2 utilizes a differentiation strategy to offer its customers something different from their competitors. In particular, Company A focuses on technology innovation and fast time to market to achieve differentiation and gain higher profits and market share. Secondarily, Company A strives to achieve high quality and doesn't allow runaway cost. To measure performance to their strategy, Company A selects metrics for measuring time to market, percentage of new technologies developed, profitability index, and market share. Additionally, they choose customer fall out and development cost metrics to measure their secondary strategy of providing high quality and cost containment.

Company B focuses on a low-cost strategy aimed at establishing a sustainable cost advantage over its rivals. The intent is to use the low-cost advantage as a way of underpricing rivals and capturing market share. This strategy will leave company B with a small-profit gross margin per product, but profit can be achieved through a large volume of sales and lean staffing. Metrics to support this business strategy should measure development and manufacturing cost, market share, program-staffing level, gross profit margin, and sales volume. Secondary objectives are to bring the product to the market in an average industry standard time and with an average quality. This is complemented with the second priority metrics of development time and defect assessment rate.

The best-cost strategy pursued by Company C combines upscale features with low cost. Company C aims to become a low-cost provider of IT products that have high-quality features. A secondary strategy is to develop innovative information technologies within reasonable time frames. Primary metrics to measure this strategy should include defect rate assessment, customer fall out, development cost, and manufacturing cost. Secondary metrics could include percent of new technologies developed and development time.

These examples illustrate how business strategy drives the choice of metrics. We showed only three examples of companies with different strategies and the corresponding sets of metrics chosen. Each company in the world will have its own unique business strategy and set of metrics to measure the effectiveness and achievement of its strategy (see box titled, "The Control Chart as a Metric for Program Team Charter").

All metrics should help improve business results. To demonstrate this point, we choose and evaluate the use of a set of metrics from the PMVP, as follows: 1)Projected future income from program road map from the strategic management and portfolio management levels, 2) portfolio risk index from the portfolio management level, and 3)percentage of program milestones accomplished from the program management and project management levels.

The purpose of metrics is to help improve business results of the organization. To serve this purpose, metrics need to be carefully developed to ensure they are compatible (or mutually aligned), balanced, consistently utilized, and tiered.

We provide a menu of commonly used program metrics that can be configured as a customized set of metrics by the management team of a business. Each business should construct its set of metrics based upon its business strategy and the specific industry in which it operates.

One convenient way to visualize a system of metrics is the PMVP that has the following four levels: strategic management, program portfolio management, program management, and project management and team execution. This pyramid provides an integrated and holistic approach for selecting and utilizing a metrics set, which evaluates the performance of the four levels of the integrated management system.

^[105]

^[106]

^[107]

^[108]

^[109]

^[110]