APPENDIX G
PITFALLS AND RECOMMENDATIONS
G.1 Introduction
Experienced project managers know there are associated pitfalls when EVM is not implemented properly.
The purpose of this appendix is to identify the most common EVM pitfalls and briefly describe some key recommendations as to how these pitfalls can be avoided. Practical experience demonstrates that if these pitfalls are properly handled, any apparent difficulties and hidden hazards can be overcome and the EVM method will deliver the expected benefits.
The main EVM pitfalls are grouped into the following main categories:
(1.) Lack of organizational support
(2.) Poor project controlling policies and poor quality of management response based on the EVM metrics
(3.) Poor project planning
(4.) Inappropriate cost and budget distribution over time
(5.) Inappropriate assessment of the earned value during work execution
(6.) Data consistency issues
(7.) Inappropriate consideration of risk management
(8.) Inappropriate modeling of subcontracted work
(9.) Inappropriate use of the base EVM formulas in “boundary” conditions
(10.) Inappropriate use of EVM for forecasting cost and schedule
(11.) Overreliance on the IT system/software
(12.) Inappropriate level of disaggregation leading to excessive data
(13.) Not using EVM at the portfolio and program management levels (e.g., for prioritization)
The remaining sections in this appendix address some of the most common pitfalls within each of these categories.
G.2 Lack of Organizational Support
It is crucial that the essential foundations of EVM are known and understood throughout the organization. The concept of a complex set of formulas and advanced metrics must be replaced by the straightforward need for a very essential set of measurements that deliver effective guidance to managerial control.
The EVM metrics must be used primarily to guide and help the organizational resources in achieving the objectives, and even function as the basis for incentives related to performance.
The indications, trends, forecasts and qualitative suggestions that can be derived from the EVM system, must be understood as solid objective ground for managers to make better decisions, and not as decisions themselves or as a prediction of the inevitable future.
Confining the knowledge about the EVM method to only specialized experts within the organization will tend to create fears and rejection by the organizational resources who feel the performance of their work is under constant auditing. Over reliance on the output produced by the EVM system, with little or no space for debate about the causes and interpretation, will lead to negative results.
Above all, EVM should be positioned within the organization as a common language to speak objectively about the performance of projects and to work as a basis for better informed decisions at all levels in the organization.
G.3 Poor Project Controlling Policies and Poor Quality of Management Response Based on EVM Metrics
While the original purpose of EVM is to deliver an objective description of a project's performance, it will only be useful if it leads to better control decisions. Using monitoring information to lead to effective decisions that positively influence the project future is essential.
This requires project control policies based on EVM metrics to guide management responses. These policies are developed by identifying patterns that relate to project status, responses, and their impacts, as well as the recording of lessons learned and, ultimately, by developing a knowledge management system.
Incorrect interpretations and over simplistic decision-making rules based on EVM results, can lead to a scenario of poor project performance and the lack of credibility.
G.4 Poor Project Planning (WBS—Scope, Scheduling, Estimating, Procurement/Contracting)
A well-known benefit of the EVM method is the discipline that it brings to project planning. In fact, without a set of planning requirements, EVM cannot be properly implemented.
These requirements are described in the main chapters of this practice standard and include:
- Proper decomposition of the project WBS to deal with a number of factors like uncertainty, risk, progress measurement, and accountability
- Proper scheduling to ensure a correct distribution of the project work, resources, and budget
over time - Proper estimating to deliver realistic targets of cost and schedule, against which performance can be meaningfully measured
- Proper consideration of subcontracted work with regard to scope definition, scheduling, and budgeting (as also discussed later in this appendix).
When these requirements are not observed and EVM is implemented on top of poor quality project plans, the outcome can be misguided results or difficulty in implementing EVM. Often in these cases, EVM is referred to as “not being applicable to this type of project or incompatible with this type of environment.”
G.5 Inappropriate Cost and Budget Distribution Over-Time
EVM is based on the integration of scope, cost, and time and, therefore, it deals with the distribution of both the baseline budget and the actual cost over the planned and actual durations of the project.
The distribution of the baseline budget over the life of the project (i.e., PV—the planned value) will determine the rate at which the work is planned to be accomplished, and it is also the basis for evaluating the cumulative work accomplished (i.e., EV—the earned value). In turn, the recording of the actual costs incurred in the project due to resource consumption over time will determine the cost of the work performed (i.e., AC—the actual cost).
The comparison of the metrics PV, EV, and AC over time is the basis of performance measurement in EVM. If the project budget and the actual costs are not properly allocated to the correct time periods to which they refer to, the consequence will be incorrect performance and variance measures, which will misguide management and will be of little or no use.
The key principles for a correct allocation of budget and cost over time are as follows:
- Baseline Budget—The baseline budget must be allocated to the time periods where the corresponding resources are planned to be consumed as a consequence of work execution;
- Actual Cost Incurred:
- Actual costs are allocated to the time periods where the scope that consumed the resources was executed (and not when the resources were invoiced or paid for)
- Actual cost are calculated based on the real unitary cost of the resources (and not based on the unitary cost considered for the baseline budget)
- Indirect costs require proper cost accounting procedures be employed (especially if these costs are of considerable magnitude).
There are some typical situations where an incorrect time allocation of the budget and cost may occur.
G.5.1 Excess of Level-of-Effort Work Packages
Level-of-effort (LOE) packages distribute the cost uniformly over the time period of the work package. LOE work packages exhibit a performance “as planned” (cost or schedule). If this method for crediting EV is used extensively, the real project performance will tend to be disguised by an “as-planned” performance tendency. Therefore, the LOE method should only be applied to those work packages that, in essence, represent ongoing work within the project (e.g., technical supervision or management monitoring). LOE must not be used to overcome practical difficulties in implementing the correct method to credit EV.
G.5.2 Discrete Allocation of Large Costs
There are situations where some work packages will consume very expensive resources that are acquired for the project at specific moments in time (a typical example is the acquisition of industrial equipment). If such large costs are allocated discretely to a specific time period (e.g., day, week, or even month), this will generate a sudden steep increase in both planned (PV) or actual work progress (EV), as well as in the actual costs (AC). This has a number of disadvantages either by “diluting” relevant performance variation, or by generating extremely high variations in the EVM performance measures that do not reflect the true performance of the project. Therefore, proper cost accounting techniques should be used to correctly distribute the baseline budget and the actual cost of the resources throughout the project scope.
G.5.3 Spread of PV Over Time Not Consistent with How EV is Credited
The way in which the baseline budget of a work package is spread throughout its planned time period (i.e., the PV) reflects the rate at which the work is expected to be executed. On the other hand, the method chosen in planning to credit EV will affect the way in which the actual work rate will be measured during execution. As an example, the LOE method will report a constant rate of execution, whereas the 25%/75% rule will report a step change in scope accomplishment. Therefore, if the distribution of the baseline budget over time does not take into account the method chosen to credit EV, false schedule variances will be reported (e.g., planning a uniform distribution in PV with a 50%/50% method being used to credit EV).
G.5.4 Difference Between Cash-Flow and Resource Consumption in Estimating AC
The AC metric is intended to measure the actual cost of the work performed. The cost being measured should relate to the resources consumed by the scope accomplished to date. This concept of resource consumption is different than cash flow costs, either invoices or actual payments. Invoicing and payment will almost always differ in time from the moment when the resources are actually consumed. Therefore, if AC is mistaken as financial cash flow, then false cost variances will be reported. For example, an up-front payment of materials acquired may generate an incorrect negative value in cost variance. Costs should only be reported as part of AC when resources are consumed as a consequence of being applied to scope accomplishment.
G.5.5 Estimating Actual Cost (AC) and Revising It Retrospectively
There are real-life situations where the data and information required to determine actual costs is not available at the end of the control period. This can be due to various reasons, such as time delays in data collection, or in accruing indirect costs. These constraints will generate a delay in updating AC, where it could be behind the update of EV and thereby generate incorrect cost variance. In these situations, the value of AC for the time period can be estimated using various methods such as trend analysis, correlation, parametric models, or other approaches. When the data and information required becomes available at a later period, the AC value can be corrected retrospectively. These procedures to estimate and retrospectively update AC should be defined and established in the EVM system during planning to ensure traceability through the proper implementation of a log.
G.6 Inappropriate Assessment of the Earned Value During Work Execution
The earned value measures the work accomplished in the form of the value in the baseline budget (PMB) of the work accomplished to date, whether it is on time, ahead, or behind schedule. Besides the selection of the appropriate method to credit EV, there are situations where an incorrect assessment of earned value may take place, which may lead to incorrect variances and performance measures.
G.6.1 Scope Reduction: Cancellation of a WBS Package After EV Has Been Credited
When scope reduction occurs, there may be situations where the work packages being cancelled already have progress reported and therefore EV credited. In this scenario, a rebaseline will take place where the budget corresponding to the remaining work in the work packages that are being cancelled (BAC – EV) needs to be removed from the total budget of the work packages prior to their closure. If the scope associated with the cancellation remains on the project, but is not distributed to control accounts, then it should reside in undistributed budget until a final decision is made to remove it from the project.
G.6.2 Overestimation of Earned Value
There may be situations where the degree of physical accomplishment in a work package is overestimated. A typical example is the discovery of unexpected rework or errors in data collection and progress measurement. When these situations occur, depending on the causes and on the organizational procedures, a corrective action is required to eliminate or compensate for the incorrect variance. For example, in some environments, procedures do not allow for revising past reported EV information; therefore, reporting a negative value for EV in the next time period will be required (i.e. to reduce the cumulative EV). On the other hand, in other environments, past EV information can be changed and corrected so that an accurate and true representation of how the work actually progressed over time is produced (and for the benefit of the current project, and for a better benchmarking in future projects). Whatever procedure is adopted, it must be defined and established upfront in the EVM system during planning.
G.6.3 Inaccurate Measurement of Work Accomplished to Credit EV
Crediting EV is primarily based on measuring the amount of work accomplished in a work package. If this measurement is subjective and not sufficiently accurate and precise, incorrect performance measures may be produced and stakeholders could lose confidence in the use of EVM.
In addition to selecting a method to credit EV, proper and feasible data collection and metrics generation procedures need to be planned and implemented, to ensure that the measurement of the work accomplished is accurate and delivers sufficient precision. These procedures should also be timely, and the cost of implementing them should correspond with the project budget and importance. The process of measuring the work accomplished should be objectively auditable—this is essential for transparency and to ensure the stakeholders’ confidence in EVM reporting.
G.7 Data Consistency Issues
The EVM method deals with data that integrates scope, cost, resources, and schedule. This multi-dimensional integration of the project data is prone to generate various forms of inconsistency. Some typical examples are as follows:
- Incomplete work packages scheduled in the past. The corresponding activities of the work not yet accomplished need to be scheduled in the future.
- Remaining planned resources and budget in work packages scheduled in the past. This will underestimate the required future resources. Resources and budget not yet consumed needs to be allocated to the work scheduled in the future.
- Completed work packages with the completion date in the future. Completed work needs to be scheduled in the past.
- EV credited to work packages scheduled to be executed in the future. Work is only accomplished in the past.
These and many other situations lead to the generation of incorrect EVM performance and variance measures by portraying an overall incorrect image of the project status. This will undermine the required confidence of stakeholders in the EVM method and will misguide management decisions. Therefore, some form of data quality controls must be put in place and planned as part of the EVM system, during project planning.
G.8 Inappropriate Consideration of Risk Management
Risk management is not optional in project management. Therefore, the risk process must be properly integrated into project planning and controlling, and this must be taken into account in the EVM system. The time and budget under the control of the project manager should incorporate the contingency reserves established by the risk process as part of responding to risks. Even with the most proactive risk management policies, there should always be some level of contingency reserves as a result of active acceptance and contingency plans. Contingency reserves are the project manager's responsibility and therefore proper budgeting and scheduling should be incorporated into the PMB.
There are various forms of considering and incorporating contingency reserves into the PMB. Whatever the method used, it must preserve the validity of the EVM metrics and performance measures. For example, if reserves are simply “diluted,” or hidden, across the work packages, then the EVM performance measures in some cases will overestimate performance (where actual risk lower) and in other cases will underestimate (where actual risk is higher).
Contingency reserves should be allocated to the time period and work packages in which the corresponding risks were identified to possibly occur. When reserve is transferred from a reserve into to a work package to accommodate a contingency plan or a reaction to a risk that was simply accepted, the PMB must be revised to portray the new expected distribution of the work and cost over time.
G.9 Inappropriate Modeling of Subcontracted Work
Subcontracted work is executed by an external party. The planning and monitoring of this type of work from a buyer's perspective is different from the contractor's perspective. For example, the selling price for the contractor is the budgeted cost for the buyer, and the actual cost for the contractor might be visible under the contract for the buyer. Another example is the impact of rework; this could be extra cost for the contractor (AC) and a slowing down of EV for the buyer, or it could be extra cost for the buyer (AC) and a re-baseline (PMB) for the contractor. This will depend on who is responsible for the extra work required and on the type of contract.
It is important to note that, potentially, there can be two separate EVM systems: one that measures the internal performance of the contractor and another that measures the performance of the contracted work under the buyer's perspective as part of its own EVM system. Performance measures will often differ in both systems, especially cost performance. When modeling subcontracted effort in an EVM system, care must be taken not to confuse the two performance systems. The perspective to be considered is always the buyer's perspective.
Furthermore, in order to measure the performance of subcontracted work, the type of contract and the agreements about data collection established with the contractor are important factors to take into account. For example, in a fixed-price contract, unless there is no extra scope, rework, or other extra costs as part of the buyers responsibility, the AC is equal to the credited EV for the subcontracted work package. The measurement of work progress to credit EV is based on the data collected under contractual agreements. However, with a cost-reimbursable contract that has an incentive fee (CPIF), the method for determining the incentive fee needs to be accounted for in the PMB.
G.10 Inappropriate Use of the Base EVM Formulas in “Boundary” Conditions
The EVM method considers a number of metrics that can produce strange results if applied to extreme boundary conditions. In particular this is the case of ratios like SPI (EV/PV) and divisions like IEAC (BAC/CPI). For example, consider the case of SPI in Table G1:
Table G1. SPI Behavior
| SPI = EV/PV | PV = 0 | PV > 0 |
| EV = 0 | SPI = 0/0 = ? No performance |
SPI = 0/PV = 0 A late start |
| EV > 0 | SPI = EV/0 = ? An early start |
SPI = EV/PV > 0 Normal scenario |
The behavior of the SPI indicator in extreme conditions has been the basis of some controversy with regard to its use for the effective measurement of time performance, because it always floats to the value of 1 when the work is completed and/or the baseline completion date is reached. Alternatives have been proposed, for example, the use of the earned schedule concept (discussed in Appendix D), or a modified version of the SPI indicator based on work volume.1
Another example and simpler scenario is where CPI = EV/AC = 0, where EV = 0 (i.e., no work done) and AC>0 (i.e., cost has been incurred). In this case, IEAC = BAC/CPI = BAC / 0 = +∞. In this case, a better forecast would be: IEAC = AC + BAC. The diagnosis of the cause is essential.
Extreme boundary conditions are identified and resolved under the specific circumstances where the project work is being executed. If not properly identified and resolved, this can lead to a lack of trust in the EVM system and even rejection.
G.11 Inappropriate Use of EVM for Forecasting Cost and Schedule
The term “forecast” has been used in the early days of the EVM method, primarily referring to trend analysis of cost and schedule.
For example, for an initial budget of BAC = $10,000, and where the CPI = 0.5 (i.e., $0.5 produced per each $1 spent), then the “at completion forecast” would be: IEAC = BAC / CPI = $10 000/0.5 = $20,000. Or in other words, the project cost will double.
This method of forecasts for both cost and schedule is often rejected as it often presents large variances at completion that may appear both exaggerated and unstable as the project progresses.
In fact, this type of forecast is really a trend that should be correctly read as: “…if the performance of the project is not improved (i.e., through management intervention) and continues like this, the final cost of the project is likely to double.”
The primary purpose is to deliver a warning about the consequences of the project past performance “propagating” into the future, and therefore these forecasts are a stimulus for management intervention. In other words, if the cause of past performance is not addressed and properly handled, the future performance will be like the past or even worse. Therefore, act!
Of course, as the project unfolds and performance changes, so does the forecast as it continually projects a new view of the project's future that is consistent with the most recent performance.
These forecasts based on trends analysis, are often misinterpreted as an attempt of the EVM method to predict the inevitable future—as if the future would not be affected by management responses. This is a misconception, since a project is a social system and not an independent physical system as one observes in the natural sciences. In social systems, forecasts have a strong influence in the future outcome since they trigger reaction.
G.12 Overreliance on the IT System/Software
While an IT system is essential to support integrated project planning and all the data collection required by the EVM method, whatever system is used should surrender to the needs and specific requirements of the project management process based on EVM.
Often overreliance on the IT system and/or software packaged limits the scope for interpreting the EVM metrics, criticizing results, and ultimately making good decisions.
The IT system helps management to develop a better image of the project but it does not deliver the ultimate image. To understand the status of the project, the project manager needs to understand the context of the data. EVM data is an indicator of project progress; however, it must be combined with analysis and critical thinking to understand the meaning and implications of the data.
G.13 Inappropriate Level of Disaggregation Leading to Excessive Data
Decomposing the project work into elementary subpackages helps in improving the accuracy of estimates, in establishing progress measurement metrics, and in assigning responsibilities. All of this is positive for the purpose of implementing the EVM method.
However, excessive decomposition can lead to serious problems in using EVM metrics. As the work is further decomposed into even more elementary work tasks, variation during execution tends to increase in those small work packages ultimately leading to “noise” or random and meaningless variation.
The EVM metrics are primarily cumulative and, therefore, some accumulation of results is important for variation to be meaningful and helpful in devising management responses.
Care should be taken not to decompose the work packages into excessive detail. Besides meaningless information, there is also a significant increase in the management overhead effort required to collect and maintain the EVM data.
G.14 Not Using EVM at the Portfolio and Program Management Levels
The EVM method was developed to measure scope accomplishment and cost and schedule performance. The term “earned value” actually relates to “scope accomplished.”
However, the term “earned value” is often interpreted by managers to mean “realized benefits” or “produced economic value.” In most cases for projects, the budget value of the scope accomplished does not equate to the value of business benefits achieved, nor economic value produced.
For that to be the case, EVM concepts would have to be applied at the program and portfolio level, to measure the performance of programs based on benefits realization and the performance of portfolios based on the creation of organizational values.2
1 Rodrigues, A. (2010). Effective measurement of time performance using earned value management–A proposed modified version for SPI tested across various industries and project types. PM WORLD TODAY, Monthly Column–OCTOBER 2010, Advances in Project Management. Available from http://www.pmforum.org//library/column/2010/PDFs/oct/Column-RODRIGUES.pdf
2 Rodrigues, A. (2012). Earned Value Management for Programmes and Portfolios. Advances in Project Management Series. London:Ashgate. Available from http://www.gowerpublishing.com/default.aspx?page=637&calcTitle=1&title_id=10483&edition_id=13261