Chapter 3

The Project Management Index

TYPES OF ORGANIZATIONS WHERE THIS METRIC IS APPROPRIATE

This metric is a must for any engineering or consulting organization where all their work is projects. It is probably also important for construction organizations regardless of whether they are building power plants, office buildings, or homes. Any organization that works mostly on projects needs to have this metric so they can easily track all of their projects without having to get separate reports on each one. R&D organizations also mostly do projects and need to have this metric on their dashboards, as do many support departments like information technology and marketing. Okay, what about service, manufacturing, health care, universities, and government organizations? Well, most of them spend some percentage of their time working on projects as well. Hospitals might be converting to electronic medical records or implementing the Baldrige model, which can be viewed as projects. Manufacturing companies are often purchasing and installing new equipment, building new plants, and repurposing factories for other uses. Service organizations like hotels, restaurants, airlines, delivery services, and repair facilities work on big projects as well. In short, this metric is important for any medium to large organization and imperative for those organizations that mostly do projects.

HOW DOES THIS IMPACT PERFORMANCE?

If you are an engineering, consulting, or R&D organization, your performance on projects impacts all other measures on your scorecard, including revenue, profits, customer satisfaction, and employee engagement. Project management organizations like Bechtel, known to be one of the best, need to have real-time balanced measures on how each of their hundreds of projects are being managed. However, even exemplary project management personnel cannot manage the details of 50 to 100 projects. Even trying to monitor the details of 10 projects can be overwhelming. If your organization spends a minimum amount of time and money on projects, your performance on this measure still can have a major impact on many performance measures. For example, I worked with several California State University campuses while they were converting to PeopleSoft. The conversion involved streamlining and improving many of the processes prior to installation of the software. This effort required many months of work and was viewed as a major distraction for many of the staff—and that was just one project. They all knew that things would supposedly be better once the new system was up and running, but they would have to endure a lot of pain to get there. Consequently, the levels of employee engagement in their jobs and the “distraction index” described in Chapter 19 probably turned yellow or red during the course of this project. Another client (Thrivent) was changing and improving all of their operational processes in preparation for a new enterprise resource planning (ERP) system. EDS was in there with a big team doing the project over the course of several years, but there was a big team from Thrivent involved as well, and the project affected the jobs of hundreds of people.

Important projects that are not managed well can have a devastating impact on an organization. Good performance on a project can also have a big positive impact on a number of different measures, or at least the avoidance of any declines in performance. My community health care client AltaMed recently completed construction of a new corporate headquarters building. The company had been renting space on several floors of a nearby office building. The new three-story building was completed early and under budget. The project was very well managed and the move from the rented offices to the new headquarters went off without a hitch. People moved over in several groups as the new facility was ready, and over the course of a few weeks everyone was settled into the new facility with little impact on productivity or other factors. In this case, the project management index on the chief executive officer’s (CEO) dashboard looked bright green because the construction project and the others that made up the index were all going well. This high-level analytic that told him everything was going well prevented the CEO from having to micromanage any of the projects.

COST AND EFFORT TO MEASURE

The cost of constructing a project management index will be either low or high. In organizations that just do projects, they often have project data loaded in project management software that tracks milestones, deliverables, expenditures, and other factors. When I worked with the navy’s shipyards, they had detailed project plans and budgets, and databases were in place that tracked cost, quality, and schedule on each project. Creating the project management index for the commanding officer was just a matter of having him decide whether he wanted all of the projects in the index (he did) or just the larger ones such as aircraft carriers. Creating the project management index for a navy shipyard like the one in Norfolk, Virginia, or in Pearl Harbor, Hawaii, was fairly simple and low-cost. At one government R&D lab I worked with, the cost was not high because the director elected to only include about four or five big projects in the index. One big project is called the “Master Plan,” a multiyear effort to change the campus to get rid of old buildings that are not useful or too expensive to maintain, build new offices and testing facilities to suit the current and future work mix, and change the whole layout of the campus to make it more pedestrian-friendly. There is a detailed project plan for this master plan, and data exists that could be built into the index. The lab works on literally hundreds of R&D projects at any one time and some of them are very small (e.g., an analysis done by one researcher that is completed in a week), whereas others were quite large and involved partnerships with universities and other entities. For the most part, none of these projects were managed using any kind of project management software. Some of the projects were so small that they did not even have a project plan. Consequently, the data on project milestones, expenditures, and deliverables would take a lot of work to track and we would likely get pushback from the researchers, who are used to having a great deal of autonomy. Thankfully, the boss did not want to include all the organization’s projects in his index, only the ones that he personally wanted to track.

Another client had a database on milestones or schedule and on costs and expenditures, but they did not have any data on quality, customer satisfaction, or other factors that they thought were important. Consequently, it ended up being a lot of work figuring out a system for measuring quality and customer satisfaction on a project and incorporating that data into the project management index. The factors for determining whether this is a lot of work and money are:

• Number of projects that are included in the index.
• Existence of project management software and data.
• Number of different metrics included in the index.

HOW DO I MEASURE IT?

Every project management analytic I have ever helped a client construct includes three lagging measures of a project’s success:

PROCESS AND CHURN: TWO ADDITIONAL METRICS TO CONSIDER IN A PROJECT MANAGEMENT ANALYTIC

The problem with measuring only budget, quality, and schedule to evaluate the success and progress of a project is that they are all lagging metrics. In other words, you have to have already gone over budget or missed the deadline or had some defect for data to be recorded. All three are measures of the past. Therefore, it is wise to include at least one leading indicator in the project management analytic. A few examples from client case files illustrate how this can be done.

Measuring Process

The Sacramento District of the Army Corps of Engineers (ACOE) put a lot of thought into the project management analytic it developed for its new scorecard, because the analytic represented most of the work being done. One thing that concerned the ACOE about the district was that project managers often did not plan and manage their projects systematically. Part of the reason for this was that the staff had a great deal of experience and was more comfortable just “winging it.” A dimension the ACOE wanted to include in the analytic was a measure of the degree to which a systematic process was followed. In addition, the ACOE felt that the process submetric would always be given a weight of 30 percent, so that even if the budget, quality, and schedule showed perfect performance, the overall score for the project would be in the yellow zone.

The process submetric was derived from a checklist of activities that should be done on all projects:

• Involving the customer in creating the project plan;
• Holding weekly project review meetings;
• Entering progress data on a weekly basis using the ACOE project management software;
• Inviting key contractors and partners to project review meetings early on; and
• Keeping the project plan up to date as changes occur.

Measuring Churn

Two other clients (Northrop Grumman Newport News Shipbuilding and U.S. Navy Aircraft Carrier Maintenance Team One) came up with another leading indicator that they now include in their project management analytic: churn. Churn is defined as changes to key project parameters (such as scope, deliverables, budget, and schedule). What both organizations had found through research is that poor project planning led to excessive churn, which led to poor performance on project outcomes like cost and schedule. Hence, churn was a good leading metric to include in the analytic.

Some amount of churn is normal and expected, because it is impossible to estimate all the work that will be needed to maintain a vessel as big as an aircraft carrier. However, experienced project managers were able to define reasonable red, yellow, and green ranges for churn, based on past projects. This was also a measure that was easy to track because every time there was a change to a project, the information had to be updated in the project planning records.

Another example that illustrates the churn metric on a smaller scale involves a neighbor of mine who had his house torn down a year ago and is building a new one on the same lot. The move-in date has been changed many times and the scope of the project has changed as well. Some of the changes were a result of alterations requested by my neighbor, but others were caused by the contractor’s poor planning, poor-quality work, or failure to meet building codes. The churn metric on this construction project would clearly be in the red zone. If my neighbor had a process metric, that would probably be red as well.

Benefits of Measuring Process and Churn

By making process or churn part of your project management analytic, you will have a more balanced measure of project performance. To roll each individual project up into the overall performance metric, you need to decide which projects get counted and roll up the boss’s scorecard. One client measures only the top five most important and costly projects each year in his project management analytic. In contrast, the Norfolk Naval Shipyard counts all projects, but it assigns them a weight based on their size (measured in labor-days) and importance to the navy. This way, the major-dollar projects and other important projects do much more to drive the overall project management analytic than do the smaller ones. However, the commanding officer still wants to know if a small project is in trouble, so even though a small one might not move the overall analytic into the yellow or red, it would certainly turn the little warning light on the analytic red. This is the cue for the commanding officer to drill down into project-level data to locate which one is showing poor performance.

MEASURING RISK ON PROJECTS

A sixth dimension that you might consider adding to your project management index is risk. Changes are made all the time as projects progress, and sometimes project personnel wisely choose to do something that might add to the scope (churn) and cause a deadline to be missed (schedule) because the additional work prevents future problems. This concept is important to a group I work with in the Los Angeles Fire Department (LAFD). They have a division called Supply and Maintenance (S&M) that does maintenance and repair work on the fire department’s trucks, cars, ambulances, and fire engines. Manager Mark Clark explained to me that managing risk is an important part of the mechanics’ job. They have to make decisions on replacing parts or doing additional work on a vehicle if it means reducing the risk that the vehicle will fail in the field. Sometimes this might make a customer mad because his truck was not done when promised due to the extra work. However, making these decisions is a big part of the day-to-day work of managing maintenance and repair projects. The LAFD S&M division liked the idea of a project management index that summarized all the work going on in the shop and was not really concerned with churn or process. What the division was concerned with was risk, believing that that factor needed to be measured as part of the index to offset the lagging indicators on quality, cost, and schedule. There is more information on measuring risk in Chapter 5.

VARIATIONS

I’ve seen all sorts of variations in creating indices to track project performance. In fact, just about every client develops their own metric tailored to the types of projects they do and what type of project data they have available. One really great idea one client came up with was to assign a differential weight to each of the submeasures in the project management index, which were budget, quality, and schedule. One client that repairs jets for airlines negotiates on the weights of these factors with the customer on each job. For example, one airline might decide to put a low weight on budget and a high weight on schedule, and is willing to pay the repair company overtime to get its plane back in service quickly. On this project, the repair company might establish weights of 50 percent on schedule and 25 percent each on quality and budget. Another client is having financial trouble and is willing to have the repair company take its time getting the plane fixed, as long as the company does it as cheaply as possible. For this client, the weights might be 60 percent on budget or cost and 20 percent on the remaining factors. The point is, the boss can view one high-level analytic that tells her about overall project performance so she can drive the right priorities from employees who are linked to what is important to the customers. When this was first discussed, the customers typically said all three factors (budget, quality, and schedule) were equal in importance. However, after more discussion they admitted that often one of the three was more important and deserved a higher weight than the others. The weights were not done by client, but by aircraft. A single airline could have one aircraft in the shop where schedule was the number one priority and another aircraft where quality was the most important.

The good thing about this approach is that it forces the customer to define its priories up front, and it tells the project team which of the three factors (budget, quality of deliverables, or schedule) is most important. At the end of each project, customers fill out a report card grading the vendor on how it performed for each factor. The grades are then multiplied by the weights to arrive at a total project score. Each project score is then given a weight in the overall analytic based on the dollar value of the project and the dollar value of the account. This brings up another variation that usually makes sense: assigning differential weights to each of the projects in the index. To keep it simple let’s say the CEO’s dashboard included a project management index with four projects that feed into the index. Project no. 1 is implementation of a new ERP system, a multiyear, multimillion-dollar effort that is given 50 percent weight in the index. Project no. 2 is implementing a new succession planning process for executives, given a weight of only 15 percent; project no. 3 is opening a new warehouse in St. Louis, Missouri, given a weight of 20 percent; and project no. 4 is creating a new marketing campaign for a renewed old product that has always been a solid revenue generator (15 percent). The assignment of a weight is not just based on the dollar cost of the project but how important it is to the organization’s success. Succession planning might get a much higher weight if most of the executives are in their sixties and no replacements have been groomed.

FORMULA AND FREQUENCY

Most organizations do track projects, but rarely do they roll up project data into an index for senior leaders. The following is a description for creating a project management index that allows executives to assess the health of key projects on a regular basis without having to review detailed data and hundreds of charts:

Generally this is a monthly metric, but it could be tracked daily and weekly as well, depending on the number and types of projects. A recent homebuilder client had milestones completed every day, so the CEO monitored key project statistics daily.

TARGETS AND BENCHMARKS

Absolute targets do not make sense for many of these project management measures. Most of these measures have a target where in the middle is the green zone, with two yellow zones and two red zones. For example, when looking at budget, if you underspend your budget by a big margin, this is a red flag, possibly indicating that you padded the original budget or that not enough progress is being made. The same is true with quality to a lesser extent. If you do more than the customer is willing to pay for in terms of quality, that is a problem, just as defects or rework are a problem. If you decide to include churn as one of your PMI metrics, targets for churn also work this way. Having zero churn is probably red, and having too much of it is bad as well. The key to setting good targets for all of the project management index submetrics is to gather enough of your own history to have stable baselines and benchmark others that do similar projects to see what they have for targets.

BENEFITS OF DATA

There are some huge benefits of having one high-level composite metric that tells leaders about their key projects:

• More integrity in the data by having actual numbers to review rather than listening to words describing the status of projects.
• No time spent reviewing projects where everything is going fine.
• Up to 75 percent reduction of time spent in project review meetings.
• Real-time project performance data that can be reviewed at any time without waiting for a meeting.
• Ability to drive the right priorities by weighting submetrics.
• Ability to integrate customer priorities with project priorities.
• Assessment of future factors such as risk, churn, and process so the analytic is a mixture of past (percent, quality, schedule) and future.
• Ability to add or delete projects as old ones get completed and new ones begin.
• Ability to have different submetrics and weights for each project.
• Avoidance of reviewing each individual project by having data stacked in layers from the highest-level analytic to individual project data.
• Forcing of a more disciplined approach to planning and managing important projects because of quantifiable tracking of performance in real time.