Chapter 6
Measuring Project Portfolio Management Effectiveness
PPM effectiveness should be measured using outcome measurements and process measurements. The outcome measurements assess PPM effectiveness based on the accomplishment of the portfolio outcomes. While measuring the ultimate outcomes of the portfolio in terms of realized benefits is necessary, the intermediate measurement to assess projected benefits and to evaluate whether or not the portfolio is managed according to its goals is also important. With the expectation that effective PPM process should lead to the accomplishment of portfolio outcomes, it is necessary that process measurements be developed to evaluate the efficacy of the processes and to evaluate whether or not the process will generate the desired outcomes.
Measuring PPM Effectiveness in the Context of Each Organization
As indicated in the previous chapter, the research evidence suggests that, to a certain extent, PPM has been practiced in the five organizations we studied, but none of them explicitly defines effectiveness in project portfolio management. Also, even though these organizations have attempted to assess their PPM, their measurements need to be further developed.
In Alpha's case, senior executives are interested in the benefits generated by the projects, especially the monetary benefits, such as revenue. Besides those benefits, Alpha's executives also pay attention to the effectiveness of the process. They assess whether or not projects go through the process in a reasonable period of time. Some measurements that they have implemented include Time-in-Stage, Time-to-Launch, and Time-in-Revision. Committed launch date achievement is another measurement implemented by Alpha. For Beta, the Development PMO emphasizes process effectiveness, especially the use of resources. Beta sees the benefits of PPM as a way to promote project visibility and the transparency of resource allocation. The ability to hit the baseline plan is another measurement of effectiveness at Beta. For Delta, PPM is assessed from two perspectives. The project management perspective emphasizes the predictability of project delivery, in terms of meeting budget, schedule, and scope expectation. From the portfolio management perspective, Delta evaluates their PPM by assessing the process outcomes such as spending budget, portfolio balance, and risk mitigation. In addition, Delta has put forth some efforts to measure project benefits, including average first benefit dollar and percentage of benefit realized. At Gamma, management pays attention to the effectiveness of project execution in terms of the project visibility and project delivery. As a government center, Lambda emphasizes process effectiveness in terms of predictability of project delivery. Besides focusing on the use of funds (obligation of money), senior management put more attention to technology transition. They assess whether or not the SC&T projects are transferable to development and operations in responding to the war fighter needs.
Toward a Measurement of PPM Effectiveness
Based on what we have learned from the case studies and our definition of PPM effectiveness, we propose a measurement of PPM effectiveness in this section. From our definition, PPM effectiveness is “The organizational capability to (1) form and govern a project portfolio such that the portfolio aligns with the organization's strategic direction, addresses risks and opportunities, and is adaptive to the internal and external changes, in order to provide short and long-term value or benefits to the organization, and (2) to manage projects in the portfolio to promote transparency, process consistency, visibility and predictability of projects in the portfolio, and to promote integrity, cohesion, and the morale of the project community.” This effectiveness definition suggests that PPM effectiveness can be assessed using outcome measurements and process measurements.
Outcome Measurements
The effectiveness of PPM should be measured based on the accomplishment of portfolio outcomes. Outcome measurements can be categorized into ultimate outcome measurements and intermediate outcome measurements.
Ultimate outcome measurements assess the short and long-term value of the portfolio to the organization. We learned from the case study that company Delta has a plan to measure the short and long-term value of their portfolio. One way to measure the value of the portfolio is to assess the financial benefits generated by projects in the portfolio. These financial benefits can be measured in terms of revenue or cost saving, for example, as the “realized benefits” after the completion of the projects. To do so, management can create actual benefit realization metrics to measure such benefits over time versus planned. Besides realizing the benefits generated by the precedent portfolio and validating the decisions previously made, the information from these metrics can educate individuals such that appropriate decisions can be made when managing the current portfolio. In many cases, the project value cannot be measured directly in terms of financial benefits (Patanakul & Shenhar, 2010). For those projects, their benefits must be clearly articulated during project initiation and must be monitored as part of portfolio governance. In some cases, those benefits may be quantifiable. Such benefits as operational efficiency, for example, are measured in terms of time savings or customer complaints. Some surrogate measurements can also be used to monitor the benefits of these projects. For example, in Lambda's case, the SC&T director pays attention to the value of the SC&T portfolio by using technology transition as a surrogate measurement. If the benefits are not quantifiable, it is also possible to develop some rating scales to assess the benefits of the projects. These quantifiable benefits and rating scales can also be used in the actual benefit realization metrics.
While the ultimate outcome measurements are used to assess the short and long-term value of the portfolio once the projects are completed, intermediate outcome measurements can be developed to assess the projected benefits during project execution, with the expectation that the realized benefits will be attained. Examples of intermediate outcome measurement are the assessment of projected revenue or projected cost savings. In Alpha's case, senior executives focus on revenue generated by the network project portfolio. They closely monitor the projected revenue when performing portfolio governance. Besides the projected benefits, another set of measurements can be developed to assess the intermediate outcomes of PPM in terms of the attainment of PPM goals. These measurements can be used to evaluate whether or not the portfolio aligns with the organization's strategic direction, addresses risks and opportunities, and is adaptive to the internal and external changes. The expectation is that the achievement of PPM goals will lead to the achievement of the realized benefits. To assess whether or not the portfolio aligns with the organization's strategic direction, metrics can be developed to assess the number of projects in the portfolio that contribute to each strategic direction. The contribution of the portfolio to the strategic direction can also be assessed in terms of spending budget (Cooper et al., 2001). Similar metrics can be created to assess whether or not the portfolio addresses risks and opportunities. For example, metrics can be developed to assess the degree of technological risks and market risks of each project in the portfolio. In terms of the adaptability of the portfolio to the internal and external changes, some scenario analyses can be performed to assess the sensitivity of the portfolio to the changes in strategic priority and competitive environments, for example. The low degree of sensitivity indicates the high level of adaptability or tolerance to changes. At the aggregated level, the results from these assessments should indicate the degree of accomplishment of PPM goals.
Process Measurements
Besides the outcome measurements, the effectiveness of PPM should be measured based on the effectiveness of PPM processes. The expectation is that an effective process should lead to the accomplishment of the portfolio outcomes. We learned from many cases that the use of processes is to help promote the predictability of project delivery. So, as part of process measurement, the predictability of project delivery should be assessed. One way to do so is to measure the percentage accomplishment of projects in the portfolio in terms of time, cost, and scope. The results from this assessment should indicate the effectiveness of the process and the potential achievement of the expected benefits from the portfolio.
Process measurements should also be developed to assess the level of transparency in decision making, the consistency in the use of processes, the efficacy of the processes, the availability of information systems support, the visibility of projects to senior management, and the level of integrity, cohesion, and morale of the project community. These items can be measured quantitatively and qualitatively. The efficacy of the processes can be quantitatively measured based on time (e.g., the actual time that a project needs to go through various stages vs. plan, number of iterations at each stage, and number of resource conflicts). Besides the quantitative measurement, a questionnaire with rating scales can be developed to assess these items qualitatively. Even though these measurements are subjective in nature, the results from these measurements should indicate the overall process effectiveness, which can lead to the delivery of successful portfolio outcomes. From the case study, several organizations have attempted to assess the effectiveness of the process. Delta uses a questionnaire to assess such effectiveness. On the other hand, Gamma's CIO, by observation, has recognized that the levels of process consistency and project visibility have increased.
Besides asking the informants to directly assess processes, a questionnaire should be designed to ask questions regarding work climate, especially one that impacts PPM effectiveness (refer to the key factors in Chapter 5). For example, questions can be designed to assess the level of understanding of an organization's strategic direction, the level of communication both vertically and horizontally, the level of commitment both from top management and project community, the level of teamwork and collaboration, and the quality of information. The results from these assessments provide valuable information as the lead indicator of the accomplishment of PPM effectiveness.
In conclusion, we propose that the PPM effectiveness should be measured using outcome measurements and process measurements (see Table 2). The outcome measurements assess PPM effectiveness based on the accomplishment of the portfolio outcomes. While measuring the ultimate outcomes of the portfolio in terms of realized benefits is necessary, the intermediate measurement to assess projected benefits and to evaluate whether or not the portfolio is managed according to its goals are also important. With the expectation that an effective PPM process should lead to the accomplishment of the portfolio outcomes, it is necessary that process measurements be developed to evaluate the efficacy of the processes and to evaluate whether or not the process will generate the desirable outcomes.
