CHAPTER 4
Enterprise Metrics
Organizational metrics measure or infer the attributes of the enterprise as a whole. They deal with both the structure and the environment of projects, particularly as influenced by policies and procedures.
Organizational metrics indicate whether the environment is project-friendly, which, in turn, reflects friendliness toward project people and project things. Traditional organizational metrics also deal with the business objectives of projects and the all-important return on investment, which views the success of projects—and the success of the organization—from the pragmatic vantage point of stockholders. Enterprise metrics also show whether an organization is gaining a competitive advantage by achieving its strategic plan.
To acquire and sustain a competitive advantage, organizations must be informed about all aspects of projects and the organization as a whole. Critical information includes the current state, as well as the future direction, of the enterprise and the industry.
As organizations move toward a management-by-projects concept, the project management practice will continue to mature. Until the early 1990s, only a few organizations implemented metrics systems as an integral part of their business processes. Since then, however, project management has been recognized as a critical and competitive tool of world-class companies that are achieving desired business performance (Thamhain 1996).
Thus, a metric can be established to determine whether the effort associated with improvements in project management practice is increasing, decreasing, or staying about the same. This metric would quantify whether the culture of project management is permeating the organization and whether project management processes are effective. (Instruments 4–A, 4–B, and 4–C)
If a formalized maturity evaluation is conducted, the results will further establish a baseline for improvement in the success rate of projects. In essence, metrics should be regarded as enablers of enterprise project management maturity.
Ideally, metrics systems should be viewed as tools by which the enterprise can evaluate overall performance thoroughly and objectively in order to identify continuous improvement opportunities (Neuendorf 2002). However, traditional business metrics do not take into account the full spectrum of the requirements and perspectives of projects. Very few metrics address the formalized manner for transforming business requirements and management perspectives into project objectives. To that end, some organizations have established tools and procedures to determine what worked well and what did not in recent projects.
Ultimately, projects translate the organization’s strategic vision into action toward a positive competitive advantage. Likewise, project failures can thwart achievement of leadership in class. In one case, for example, stock value dropped 1.75% upon the negative news of projects (Ward 2002).
Sometimes, even if the metrics system is aligned with company strategy, reporting of performance information is fragmented and isolated within specialty areas. Unfortunately, this pattern of reporting leads to mixed results on overall performance. For example, the quality department may measure and report quality attributes at a certain time period, the finance department may collect and report the values of cost attributes at a different time period, and operations may report daily on project delivery metrics.
In other cases, the metrics designed to measure the performance of the units within the functional areas might not support the overall intended organizational strategy. Activities that are cross-functional may not be evaluated in an integrated fashion, primarily because links may not be in place between project performance and operational performance. Part of the reason for this disparity could be the absence of a consistent performance monitoring system for the organization’s overall project environment.
The cost of failures resulting from the lack of appropriate guidelines far outweighs the efforts required to establish those guidelines. Therefore, if an organization has not fully matured in the project management discipline, then a significant amount of effort, organizationally and within the project, might have to be allocated to developing the unified procedures to manage the project. As the organization matures, the total expenditures for project management infrastructure will decrease while the proportion of proactive functions will be maximized (see Figures 4–1 and 4–2).
Figure 4-1
Project Management Office Effectiveness
Figure 4-2
Project Management Office Effectiveness and Maturity
Organizations at higher maturity levels encourage their clients to become actively involved in specifying performance metrics and in setting quality goals. A customized metrics system that is established for the enterprise might be characterized as one that is:
• Driven by business goals
• Consistent across the organization
• Adaptable to specific user needs
• Focused on collecting data as directly as possible
• Compliant with operational data collection policies
• Supported by all affected parties, including the client.
PROJECT MANAGEMENT MATURITY
A formalized assessment of the organization’s current project management capabilities yields a detailed description of organizational strengths and weaknesses in project management. Further, the observations made using a metrics system provide the impetus for establishing uniform, enhanced project management practices within the organization.
Continuous improvement procedures provide a quantified baseline for maximizing the benefits from experience on one project into the next. A formalized metrics-based assessment provides the foundation for improvements and guidance for advancement (see Figure 4–3). From this vantage point, portions of the metrics system can be used to determine the current and targeted sophistication of the processes and procedures of the project management activities of the entire organization.
Figure 4-3
Enterprise Project Management Maturity
A suite of metrics can be used to determine the existence of realistic, effective project management policies and procedures. A serious distinction should be made between having unused procedures on the shelf and using existing procedures on a regular basis. The metrics associated with a maturity model can be used to signal how often and how regularly project personnel follow established procedures, which, in turn, will shed light on the effectiveness of prevailing project management policies and procedures.
The literature (Pennypacker and Grant 2003) shows that most organizations, even those that consider themselves project-oriented, tend to rank at the lower end of the scale (see Figure 4–4). Thus, there is room for improvement in most organizations. With greater project management maturity, an organization will experience direct and tangible improvements in delivery efficiency, as well as a better market positioning and higher profits. (Instrument 4–D)
Maturity Ranking Scale
An established maturity ranking scale provides plateaus for purposes of continuous improvement of organizational project management capabilities. The indicators of the maturity model highlight the sophistication of key practices in the organization, particularly those in need of improvement. An added advantage of using a standardized scale is that its normalized capability indicators enable industrywide comparison of project management sophistication. Favorable rankings often provide a point of pride, while less favorable results frequently serve as a source of motivation.
Figure 4-4
Maturity Demographics
Higher maturity levels signify more effective project procedures, better quality deliverables, lower project costs, and higher project team morale. Organizations at higher levels of maturity have achieved the desired balance among cost-schedule-quality, which ultimately translates into improved profits. In such organizations, there is an objective, quantitative basis for measuring quality and analyzing problems.
With these sophisticated tools, organizations achieve the expected results for cost, schedule, functionality, and quality on most, if not all, projects. Other features of mature organizations include clearly defined roles and responsibilities and the expectation of client satisfaction as the norm.
At higher levels of maturity, metrics evolve with organizational process enhancements until the target process is fully mature and until the process performs at a consistently high level of effectiveness and efficiency. Interestingly, in mature organizations changes to processes are typically small and infrequent. Moreover, in these organizations policies and procedures are not permitted to become counter-productive to the manner in which the project is managed.
Finally, mature organizations establish an environment in which success is expected on every project. This environment can only be created by encouraging project professionals to use innovative practice techniques in support of their responsibilities.
By contrast, lower maturity levels are found in organizations that encourage fixing problems in the field instead of doing it right the first time. In these environments, it is entirely possible that project personnel will repeat the same error in multiple projects or even multiple times in the same project. A low maturity ranking also could be symptomatic of unnecessary or redundant procedures, which, in turn, usually indicate an organization with a history of misdirected improvement efforts.
In immature organizations, there is no rational basis for judging product quality since processes are improvised, quality-directed activities are often curtailed or eliminated, and there is no methodical basis for diagnosing and correcting problems. In these organizations, when a project has an imposed date, functionality and quality are often compromised to meet the schedule. Project managers are like firefighters who react to the current emergency.
Maturity Level Descriptions
Metrics can be grouped by the various maturity levels of the model.
Initial Level
At Level 1, metrics focus on the things aspects of the project, not the people aspects. Some forms are available for some specific processes, but typically there is no guidance on how to use them. Because managers must invent their own practices, processes depend on each project manager’s personal orientation, experience, and people skills. At this level, over the life of the project, various team members use miscellaneous project management tools sporadically.
Developed Level
At Level 2, most projects use metrics and follow procedures for some project elements. The organization emphasizes effective project performance, and there is sporadic use of best practices. Relatively meaningful metrics emphasizing things issues are sometimes specified, documented, and repeated.
Evolved Level
At Level 3, best practices, and their accompanying metrics, are integrated into organizational policies, and team members receive training in these best practices. Procedures and metrics reflect indications of widespread consistency. There also is a culture of a common practice, because a general trust exists in the common practice. While it is recognized that the same practice may be conducted differently in different areas to reflect unique circumstances and situations, an organization-wide structure exists for these practices. Finally, project progress data are routinely collected, analyzed, and archived.
Advanced Level
At Level 4, organizational performance is viewed as being interwoven with project performance. Project performance is characterized and analyzed on numerous projects. Corrective actions are prescribed across the organization based on data analysis within specific knowledge areas and pre-established success factors. Performance assessment relies on quantitative data, and project planning data and project performance data are readily available. The organization focuses on continuously improving and aligning personal, workgroup, and organizational capabilities. Equally important, the organization empowers its competent people to conduct continuous improvement in their work processes and to propose organizational changes that support those successful improvements.
Leader Level
At Level 5, the organization regularly evaluates the latest practices with sophisticated metrics, participates in benchmarking forums and learning communities, and uses widespread performance data to forge future improvements. Further, improvement actions are readily identified and quantified. Improvements are either modifications to existing procedures or entirely new procedures. Carefully collected data are used to isolate problems and recommend corrective action in a seamless fashion. Change management is a consistent organizational process.
FOCUS OF ORGANIZATIONAL METRICS
For purposes of identification and elucidation, project management metrics can be grouped into five categories: performance, stability, compliance, capability, and improvement (Florac, Park, and Carleton 1997). Given that these categories are somewhat broad, some overlap exists; consequently, they are not mutually exclusive.
While a relationship exists between maturity levels and these categories, it is not linear, direct, or precise. Metrics in these categories include indices and models that characterize project management maturity from Level 2 through Level 5; Level 1 is not addressed as it signifies a near absence of metrics and/or success.
Figure 4–5 presents a stylized depiction of this relationship. Notably, the metrics categories loosely relate to the three focus areas of things, people, and enterprise. The performance category relates to things, while the compliance and stability categories relate to people. Finally, the capability and improvement categories relate to enterprise issues. The depiction of the relationship allows a better understanding—and a more efficient characterization—of the metrics present at various levels of operational sophistication. (Instrument 4-E)
Performance Metrics
Performance metrics can be regarded as efficiency measures. They focus on the attributes of quality, quantity, cost, and time. These metrics gather specific factual information about project performance, such as the ability to deliver products and services within the client’s requirements for quality, timeliness, and cost. Performance metrics also provide the baselines for comparison as the organization works toward establishing repeatable project management practices. If the values of performance metrics vary erratically and unpredictably over time, then the project management process is not in control.
Figure 4-5
Metrics Category and Focus
Performance measurement is a process of quantifying the effectiveness and efficiency of purposeful action (Neely et al. 1996). Performance metrics allow the determination of whether internal and external requirements are being met. In addition, Kaplan and Norton (1992) state that performance measurement is a tool for implementing the organization’s strategy in order to: translate strategy into concrete objectives, communicate the objectives to the organization’s employees, and focus and guide efforts as objectives are reached.
Stability Metrics
Stability metrics focus on the ability to complete projects according to plan consistently. They provide quantitative diagnosis of the shortcomings and respond to the need to improve the processes used for project management.
Stability of project management performance is central to each organization’s ability to produce products and deliver services according to plan and to improve processes for even better, more competitive products and services. For example, if people in the organization are using the project management methodology, but products and services still are not meeting the client’s requirements, then the methodology must be improved to reduce the variance. (Rad and Levin 2002)
For project management to be stable and predictable, it must be used consistently throughout the organization. To that end, data need to be collected and made available as to how the organization specifies the outcome of projects, the procedures for successful completion, and the resource profile for the project.
Compliance Metrics
Within the context of projects, compliance means that the project management standards of knowledge and practice exist, and that project personnel follow them. Compliance metrics assess project team adherence to the processes and the appropriateness of those processes. They also address the competency and appropriateness of the project team and the sophistication of project management tools. Project management processes that are clearly defined, effectively supported, faithfully executed, continuously reinforced, and diligently maintained are indicative of higher project management maturity and repeatable occasions of project success.
Through an understanding of how well projects are performed throughout the organization, compliance metrics can serve as the basis for improving project management processes. However, project personnel must be aware of, trained in, and given the tools that are needed for the most successful execution of these processes, and the methodology must be executed as defined. For example, even though an organization may have a Project Management Office (PMO), the latest project management software, a suite of approved methodologies, a cadre of project mentors, a project management training program, and a clearly defined career path, there is no guarantee that people in the organization will actually use the available project management concepts, tools, and techniques. (Rad and Levin 2002)
Capability Metrics
Capability metrics allow the project manager to see whether project performance has satisfied client requirements and whether project results meet the intended business needs. The performance variances need to fall within ranges that have been specified for business success. Then, analysis of performance data identifies areas where the process can be improved in order to provide better support for business objectives. Capability metrics should be effective in identifying minute elements of performance so that each project will achieve its intended success factors.
Proven effectiveness of the project management process can easily instill confidence in current and potential clients. However, capability metrics might need improvements to satisfy competitive pressures or to comply with client needs.
Improvement Metrics
Improvement metrics assess how project management can help move the organization to a level of greater profits. They determine if project management is working successfully throughout the organization and if any changes introduced have been effective ones. Improvement metrics typically are considered harder to collect (e.g., the importance of project management tools and techniques on overall profitability of the organization, whether successful implementation of a new product leads to new markets and new facilities). To promote improvements, the project team must understand both the business goals and strategies of the organization and the priorities, risks, and issues associated with them. (Rad and Levin 2002)
PROJECT MANAGEMENT OFFICE
A formalized PMO should be commissioned to perform the functions dealing with development and dissemination of recommended project management guidelines. The PMO should be the cornerstone and focal point of the enterprise’s project management initiatives. Thus, the PMO should be directly and actively involved in the development of project management policies and procedures as a means of improving project delivery throughout the organization (see Figure 4–6).
Figure 4-6
Functions of the Project Management Office
The effectiveness of the PMO should be evaluated by the degree to which the organization’s portfolio of projects is successful in terms of responsiveness, cost, and duration. (Instrument 4–F) Therefore, once a PMO is fully installed, it is reasonable to expect that the success rate of projects will increase. Hopefully, metrics will show that more projects are being completed successfully, and with fewer overruns.
At an operational level, PMO effectiveness can be measured by evaluating how the overall cost of maintaining a PMO decreases with time and/or how the organization achieves a higher maturity level. The reduction in effort will come primarily from a significantly reduced project-oriented effort, that is, augmenting, consulting, or mentoring.
The enterprise-oriented efforts, which are usually nonexistent in immature organizations, will reach their peak of effectiveness once the organization has reached a full maturity level. Notably, the cost of maintaining an operational peak for enterprise-oriented functions in a mature organization is far less than maintaining an operational peak for project-oriented functions in an immature one (see Figure 4–7).
Figure 4-7
Project Management Office Costs and Activities
Looking at the full spectrum of PMO functions, most metrics are interwoven with most of those functions. The full suite of PMO-related metrics includes success of projects, maturity of the organization, competency of personnel, and general employee morale. However, some metrics are PMO-specific in that they determine the effectiveness of the PMO in increasing organizational sophistication in managing projects.
The PMO metrics should also measure the performance of each of its major functions in the project-specific group and in the enterprise-oriented group. These metrics would address the effectiveness of the PMO in carrying out its overall organizational functions, and ultimately, PMO contributions to the profit/loss position of the organization. In that vein, the contributions of all projects to the profit base can be quantified to highlight market deliveries that are earlier than anticipated, hopefully with less cost.
Other indirect metrics for effectiveness of the PMO might include net profit, shareholder value, and increasing asset base. The expectation is for gradual increases in enterprise-oriented functions and significant decreases in project-specific duties, while success rates of projects continue to climb.
It is fair to say that PMO metrics are mostly project metrics; if most of the projects are successful, then, by extension, the PMO is successful. (Instrument 4-G) The usefulness and effectiveness of the PMO could be assessed by indices for the existence, and ease of use, of a standard project management methodology. (Instrument 4-H)
Another index for PMO effectiveness could be a quantification of the timeliness of incorporating changes to the organizational project management methodology as the methodology is being continually improved. The following items could be included in such an assessment:
• Progress against goals for improving project management capabilities
• Enhancements to the project management methodologies and models that are in use
• Benefit of project results to the organization
• Integration of a standard project management information system with the organization’s financial, accounting, and human resource systems
• Incorporation of lessons learned into the project management methodology (Instrument 4-I)
• Improvement in project predictability (Instrument 4-J)
• Comparative performance relative to other organizations that are considered to be best in class
• Impact of improvement proposals on the organization’s project management competencies
• Effectiveness of the PMO in implementing suggested improvements
• Timeliness in transferring technological innovations into normal practices throughout the organization
• Effort associated with implementing new technological innovations
• Continuous increases in project management competencies
• Impact of a team-based performance incentive system on the organization’s profitability.
WORK BREAKDOWN STRUCTURE/RESOURCE BREAKDOWN STRUCTURE TEMPLATES FOR THE ENTERPRISE
Managers have a long history of dividing anticipated project work into smaller and smaller parcels and presenting the resulting schemas graphically. This “breaking down” of the work facilitates a more efficient management of organizational matters in many ways.
Within a project, a Work Breakdown Structure (WBS) provides a framework of common reference for all project elements, for specific tasks within the project, and ultimately for better schedules and better estimates. A WBS facilitates the process of integrating project plans for time, resources, and quality. An effective WBS encourages a systematic planning process, reduces the possibility of omission of key project elements, and simplifies the project by dividing it into manageable units. If the WBS is used as the common skeleton for the schedule, estimate, and quality, it will facilitate communication among the professionals implementing the project.
Each WBS is unique to the project for which it was created; however, if the PMO maintains historical copies of all WBSs on file, they can be used for reference, best practices, and training (see Figure 4–8). Rather than developing a WBS for each project, sometimes it is helpful to develop a general WBS for a family of projects or a common portfolio of projects. Then, when developing the WBS for a new project, only the applicable segments are selected and modified. This practice is appropriate in organizations that conduct projects that are somewhat similar but not identical.
Figure 4-8
Organizational Work Breakdown Structure Modules
The managerial philosophy of breaking down work also applies to resources. For completeness, in-house resources that are needed for the project should be enumerated in a methodical manner, at the earliest opportunity, through the creation of a Resource Breakdown Structure (RBS). The RBS is a logical, useful classification of the resources necessary to accomplish the project objectives. The RBS facilitates resource assignment and project scheduling in similar projects that use roughly the same mixture of resources.
Rather than developing a new RBS for each project, it is sometimes more efficient to develop an overall RBS for a family of projects. As each new project is planned, only those portions of the common RBS that apply to the project are selected and used. A project RBS is different from all other human resource or budgeting classification methods in that it reflects applicability to project management as compared to cost accounting or personnel evaluations. An RBS is essentially a catalog of all the resources that are, or should be, available to the project.
The practice of formalizing the resource pool falls at the interface between general management and project management. The RBS has its analog in the well-known WBS. In the few organizations that currently use variations of the RBS, project managers can plan the project with greater assurance of the reliability of the resource data (i.e., the project manager can depend on this structure to determine the amount of resources at hand, and their estimated costs, in order to capitalize on organizational memory with respect to project resources).
Sometimes, the project manager might modify and use the RBS that was previously prepared by those charged with accounting for the organization’s resources. For best results, the resource content of the RBS should be kept up to date, keeping costs current as well (see Figure 4–9).
Again, major efficiencies will arise from preparing the RBS only once, and then enhancing this single structure so that it becomes accurate, sophisticated, and appropriate. The enterprise RBS is essentially the same for all projects within the same organization, with the distinction that one project may use one portion of the RBS while another project might use a different portion of the RBS. Customization of the RBS simply involves carving out the needed portion for a particular project. Finally, it is exceptionally useful if other administrative units of the organization, such as human resources and accounting, conform to this structure, rather than the other way around.
Figure 4-9
Organizational Resource Breakdown Structure
PROJECT PORTFOLIO MANAGEMENT
Proper execution of projects is pivotal to overall organizational success and financial growth. There have been cases where large sums of money were spent on projects that did not produce a profitable product or service for the project sponsor. Therefore, there must be formalized tools for optimizing investment in the full suite of all projects. This formalized methodology must take into account the diversity of projects so that both the long- and short-term needs of the organization are met.
Even if the portfolio management process is not fully formalized, and therefore not consistently effective, it does help organizations become aware of their project expenditures and the benefits that they get from such expenditures. When there are many projects, and no formal selection process, project selection is subjective, ad hoc, or even political. Under these circumstances, different levels of an organization often subscribe to different and conflicting objectives.
Portfolio management is the process by which an enterprise focuses its limited resources toward new initiatives such as research and development, new products, or operational enhancements. Portfolio management is primarily known for evaluating and prioritizing prospective projects, but its functions also include accelerating, decelerating, or terminating ongoing projects. Naturally, an added level of sophistication would be to regard the portfolio management process as an ongoing activity rather than an evaluation that is conducted at specific points and largely forgotten between those two evaluation points. Formalized portfolio management is most successful in organizations that have reached or surpassed Level 3 of a staged maturity model, in which case all projects are planned and implemented with nearly the same procedures and competence.
The purpose of a portfolio management system is to identify projects that, more than other projects, best serve the business goals of the organization. The organizational entity that conducts portfolio management must monitor the progress of these projects in light of an evolving corporate strategy, individual project performance, and overall enterprise resource demands. Sometimes, the results of the formalized portfolio management system will point to postponing, or even decommissioning, some projects that address only low-priority business objectives.
Accurate, up-to-date data regarding cost, schedule, deliverables, and resources must be available when using the project portfolio management system to evaluate a project midstream in its life cycle. Only then can the portfolio management system provide the organization a current, accurate picture of where project resources are committed. Such knowledge enables quick reaction to external market conditions and redirection of valuable enterprise resources to those projects that achieve the most favorable competitive position.
An ideal portfolio contains a balanced group of projects that is fully aligned with the current organizational strategy. The details of the system must be customized to the circumstances and environment of each organization.
Once the portfolio management system is implemented, the organization might find that there is a significant difference between the desired levels of investment and the actual use of resources on a project-by-project basis. In turn, this knowledge might trigger informed realignment of enterprise resources. The literature shows that if the portfolio management structure is systematic and fully formalized, it benefits the entire organization significantly (Cooper, Scott, and Kleinschmidt 2001).
Ideally, a portfolio management system should be in continuous touch with the strategic direction of the organization, as this direction shapes the selection criteria and continuation conditions for projects. In turn, project attributes influence the portfolio’s project mix, as updated estimates of cost and duration are compared to original values. When cost and duration values differ from the preliminary amounts used in project selection, a conscious decision must be made as to whether or not to continue that project.
The project portfolio management system puts the enterprise in an excellent position to compile data on the full complement of enterprise projects in a consistent fashion. More important, the process of project prioritization is repeatable and unbiased if decisions are based on quantified attributes of the project and quantified needs of the organization. The purpose of a formalized portfolio management system is to make explicit what is implicit in many current systems, so that enhancements can be made for the ultimate benefit of the organization.
Portfolio data include details of project objectives, cost, duration, risks, accomplishments, resource demands, and success factors. The availability of these data, together with a rating methodology, makes project selection and continuation decisions a straightforward process. Further, a portfolio management system provides communication tools for letting project personnel throughout the organization know project priority, so that they do not arbitrarily decide which projects are more important.
If the portfolio management function is treated as an ongoing process, then the organization will always have a dynamically prioritized list of projects. The portfolio management function will identify which projects need to be added to the pipeline, which need to be ejected from the pipeline, and which should continue in the pipeline.
Under portfolio management, a project undergoes initial scrutiny in light of current organizational objectives when it is conceived and conceptually planned (see Figure 4–10). After being selected for implementation, it is evaluated periodically in light of prevailing organizational objectives (see Figure 4–11). The midstream evaluation is significant and necessary because, during the intervening period, both the forecasted values of the cost and duration of the project might have changed, as well as the organization’s objectives and strategies (see Figure 4–12). This monitoring process determines whether a project should continue to receive resources.
The categorical proportion of projects in the pipeline represents the organization’s direction at that point in time, because the categorical mix will change dynamically (see Figure 4–13). When an organization regularly reviews projects in its portfolio, its suite of projects continually aligns with its mission. If an organization’s mission is modified annually, so are the pipeline characteristics (see Figure 4–14).
A formalized portfolio management system affords the organization the capability to make conscious choices in selecting projects for implementation. Naturally, if projects were periodically tested for relevance to the organization’s strategic direction, the same system would be used to revisit the conscious decision of expenditures of funds for that specific objective.
Figure 4-10
Portfolio Management: Sequence of Activities
The straightforward, and slightly oversimplified, solution for a portfolio management system would be to proceed with a project of high return, low cost, and low risk, while postponing projects with low return, high cost, and high risk. In some cases, the enterprise might be willing to invest a small portion of project funds in projects that are highly risky and costly but offer substantial payoff if they are successful.
Figure 4-11
Initial Project Selection
Figure 4-12
Midstream Project Evaluation
Ultimately, if there is a formal procedure in place, the portfolio management methodology can be enhanced to suit the needs of the organization by delivering a large amount of usable products for a minimal expenditure (see Figure 4–15). It is far more logical and relevant to characterize a project not only by its cost, return, and risk, but also by its impact on competitiveness, resource profile, and other more comprehensive organizational issues.
One of the better known portfolio management models is the stage-gate model proposed by Cooper, Scott, and Kleinschmitt (2001). In it, the project is broken down into several review phases called stages; the milestone between two successive phases is called a gate. The key feature of this process is that the validity of the project is revisited at every milestone, hence the name stage-gate. Probably the most valuable facet of this approach is the identification of the milestones at which the validity of the project needs to be affirmed for it to go forward.
Figure 4-13
Generic Project Categories
Figure 4-15
Project Categorization
Naturally, the number and texture of project phases would be different in different industries, maybe even in different projects. Accordingly, the nature of the inter-phase tests conducted at the gates is industry-specific and project-specific.
A detailed, formalized project portfolio management system determines the collection of projects that will have the most positive impact on the entire enterprise. With increasing use of the management-by-projects concept, an ongoing project portfolio management process ensures that the composite group of selected projects is totally supportive of the organization’s financial portfolio.
The portfolio management process, in turn, provides motivation to project managers to enhance project information so that better decisions can be made. For example, the system could use sophisticated analogous or parametric models to estimate cost duration from project size. Using a sophisticated, and possibly complex, portfolio management model and having formal procedures to implement the model signal an organization’s sophistication. In other words, just having a model does not indicate sophistication, nor does a regular review cycle.
The portfolio management system should not be simply a tracking system, or just a facilitating system, such as any of the abbreviated varieties of the Project Management Office. Rather, it should be an integrated decision-making tool. Then, the portfolio management system can use the progress of the project, combined with its budgeted scope-cost-duration attributes in light of current strategic objectives, to determine whether the project should go forward. In some situations, a project that is reasonably well run might be terminated simply because the deliverable no longer commands a top-priority position among other deliverables.
Portfolio Management Models
Given that the portfolio management system handles very large sums of corporate funds, it needs to be comprehensive. Even though variations of the same model might be used to prioritize different groups of projects, a collective portfolio should contain all the projects undertaken by the organization. In addition, the portfolio evaluation system should be sophisticated enough to recognize duplicate projects so that a decision can be made to combine them. Independent of how data on project attributes are created and ranked, best results can be achieved if the project is ranked using a consistent, formalized process as frequently as possible throughout the life of the project.
The current portfolio management models, which tend to be qualitative and judgment based, are not mathematically complex, although many are regarded as effective and satisfactory by their respective enterprises. The WBS-like model described here represents an initial attempt at introducing formality and directness into the process. Using this structure for the characterization and calculation of an indicator for project scoring, the first level will have elements for project and organizational attributes (see Figure 4–16). The relative weight placed on each element will depend on organizational objectives, strategic goals, and corporate environments.
Figure 4–17 provides a first approximation of these values. The total possible number of points assigned to the project is 500. The reason for this particular distribution is to arrive at a scale of 1 to 5 for the viability of the project, somewhat akin to the staged project management maturity ratings assigned to organizations.
As for distribution of funds among project categories, formality and openness to the process are enhanced if the organization regularly documents the proportion of funding among project types within the organization. Clearly, the balance between the various categories of projects remains a subjective issue. However, the proportion of funding allotted will speak volumes about the strategic orientation of the company. With publicized information, it is possible to compare the organization to itself over time, as well as to other companies in the same industry.
Figure 4-16
Project Scoring Model
Figure 4-17
Sample Project Scoring Model
Figure 4–18 shows a typical proportioning of funds. Even though the percentages assigned may seem arbitrary, they implicitly signal the importance the organization places on different recipients of project funds.
Having developed this funding allocation scheme, and having developed a project-scoring model, the task of prioritizing projects becomes logical, straightforward, and formalized. Figure 4–19 depicts funding categories with the projects they currently support. This structure illustrates two realities of project funding. First, at any given point, there are limited amounts of funds available for each project category. Second, within each category, funds are dispensed to projects highest on the list until funds are depleted. Accordingly, all projects below the point of funding would either be cancelled or deferred until the next review cycle. Naturally, the funding proportion can change at every review point, as can the order in which projects in each category are funded.
Figure 4-18
Sample Project Funding Distribution
Figure 4-19
Sample Project Ranking (Grouped by Categories)
When managing a project portfolio through the use of models and indices, one metric must be focused on balancing the different types of projects, such as short term, long term, low risk, high risk, etc. With the use of this categorization index, all similar projects would be funded from a distinct amount of funds that is separate for each category.
The use of a category index infuses a certain amount of formality into the funding process, although sometimes organizations choose to allocate an arbitrary amount of funds to a particular group of projects. For example, a deliberate decision can be made to allocate funds to very high-risk projects that have high returns if they are successful. Further, project diversification is often a primary goal of a portfolio management system. Therefore, the system ensures that each desired category of projects has at least one project in the implementation pool.
Portfolio Management Indices
Each project review model, in any of the funding categories, will comprise several indices that characterize the project to the satisfaction of the organization. The indices of a portfolio management model can be divided into two distinct categories: (1) those that deal with the attributes and goals of the sponsoring organization, and (2) those that deal with the features of the projects that are intended to meet the identified goals of the organization. When such a model is used to evaluate a project, the attractiveness of the project is dependent on how well the project fares in both categories
Indices that characterize the organization include those focusing on business objectives strategy, profitability strategy, market conditions, interest rates, and general economic growth. The organization-related indices can be either quantitative or qualitative (see Figure 4–20).
Figure 4-20
Project Selection: Organization Indices
Quantitative indices are usually finance-based, have a definite formula, and are relatively easy to determine (see Figure 4–21). Qualitative indices are usually rooted in strategy, competitiveness, or marketability. Qualitative organizational indices tend to be more experience-based and subjective. Indices that characterize the project deal with the performance, or predicted performance, of individual projects, specifically cost, schedule, and deliverable (see Figure 4–22).
The project-related indices sometimes describe the variances between planned and actual values of the scope-cost-schedule attributes of the project (see Figure 4–23). Such variances are crucial pieces of information during the midstream evaluation of projects when the decision has to be made whether to continue the project or abandon it.
Figure 4-21
Organizational Indices: Financial
Figure 4-22
Organization Indices: Strategic
Figure 4-23
Project Selection Project Indices
Organizational metrics quantify the attributes of the environment within which projects must operate. In a way, organizational metrics measure the friendliness of the enterprise toward projects. Organizational metrics include, first and foremost, project portfolio models; however, they also include models and indices that quantify an organization’s project management maturity and the effectiveness of its PMO.
INSTRUMENT 4–A
Continuous Improvement in Processes
By acting on the processes used to achieve results in project management, the organization’s performance will continue to improve. However, the organization’s culture must change to support these improved practices.
Use the following checklist to determine the current state of the organization.
Consider the following questions to help determine which improvement suggestions will have the greatest impact on the organization:
What are the specific business goals and objectives that will be addressed?
How will the improvement project impact ongoing projects and other work?
Are there any external factors to consider?
How much effort will be required to formalize the process improvement and to implement it?
How much time is needed?
Will any funding be needed?
What training will be needed in the use of the new process?
Who will conduct the training?
What other concurrent projects must be considered?
Will any existing systems require change as a result of the process improvement?
How will the implementation be managed?
Will a pilot program be used before full-scale implementation throughout the organization?
Are there any specific barriers that must be overcome in the implementation process?
How will the new process be evaluated to assess its effectiveness?
INSTRUMENT 4–B
Project Management Style
This instrument assesses the project management environment of an organization. The premise is that the behavior of the people in an organization indicates the preferences of upper management.
The statements below constitute the anchors (i.e., numbers 1 and 10, respectively) of a 1 to 10 rating scale. Assess organizational attributes within these extremes, assigning a rating between 1 and 10, inclusive.
INSTRUMENT 4–C
Organizational Project Management Attributes Checklist
This instrument helps quantify the organization’s sophistication in managing projects. By repeating the survey on a regular basis, and by comparing those results with the baseline, one can determine the extent to which the organization has matured since the previous assessment.
PROJECT INTEGRATION MANAGEMENT
1. What is your view of project management in this organization?
2. Is a project charter prepared? If yes, who issues it and what are its contents?
3. What is included in your Project Management Plan?
4. Who participated in the development of the Project Management Plan?
5. Who reviews and signs off on the Project Management Plan?
6. Who receives copies of the plan?
7. How is the Project Management Plan used as the project proceeds?
8. How often is the Project Management Plan updated?
9. If the Project Management Plan is updated, who is notified?
10. Who is your project sponsor?
11. Does the organization have a standard project life cycle? If yes, please describe it.
12. Does the organization have a project management methodology? If yes, please describe it. If no, what do you feel should be part of this methodology?
13. Is there a project management information system?
14. Is standard project management software used across the organization on projects?
15. What type of information is collected on project status?
16. What is involved in project reviews?
17. Is earned value used to measure and report project progress?
18. Are lessons learned documented and maintained so they can be used by future projects?
19. Should this organization establish a Project Management Office (PMO)? If yes, what functions should the PMO perform?
20. Is a project closeout plan prepared?
PROJECT SCOPE MANAGEMENT
1. Is a scope statement prepared?
2. Is a Scope Management Plan prepared?
3. Is a Work Breakdown Structure (WBS) prepared? If yes, how often is it updated?
4. Are items in the WBS assigned to particular individuals or organizational units?
5. Is a WBS Dictionary prepared? If yes, how often is it updated?
6. Is a scope baseline prepared?
7. Are inspections conducted?
8. Is a process in place for formal customer signoff on deliverables?
9. How is scope change handled?
10. Is scope creep common?
11. Are Change Control Boards used?
12. Is formal configuration management used?
13. Is configuration management software used?
14. Are lessons learned documented?
15. At the end of the project, is official customer confirmation obtained specifying that all requirements have been met satisfactorily?
PROJECT TIME MANAGEMENT
1. Is an activity list prepared?
2. Is a project schedule prepared? If yes:
a. Were schedule risks identified?
b. Were dependencies identified between activities?
c. Has the critical path been identified? If yes, were activities on the path tracked regularly?
d. Are there a project calendar and a resource calendar?
e. Does the schedule include key events and/or major milestones identified by the customer?
3. Is a Schedule Management Plan prepared?
4. How are schedule changes handled?
5. How often is the schedule revised and reissued?
6. Is earned value used to measure and monitor schedule performance?
7. Do formalized procedures exist for compressing the network to achieve a lower duration?
8. Do formalized guidelines exist for expanding the network in response to a shortage in resources?
9. Is there an organizational standard for the software used as the scheduling tool?
10. Are there organizational policies and tools to handle multi-project resource planning and monitoring?
PROJECT COST MANAGEMENT
1. How are project cost estimates prepared?
2. How is the cost estimate correlated to the organization’s accounting system?
3. How is the project budget prepared?
4. How are changes to the budget handled?
5. Is a Cost Management Plan prepared?
6. Is a cost baseline used?
7. What comprises the cost control system?
8. How often are budget updates required?
9. Is earned value used to measure and forecast project cost performance?
10. What type of cost tracking software is used?
PROJECT QUALITY MANAGEMENT
1. Are quality audits and inspections conducted on a regular basis?
2. How is the overall quality of projects evaluated?
3. Does the organization have a quality policy?
4. Does the project have a quality policy?
5. Is a Quality Management Plan prepared?
6. Are data collected on the cost of quality?
7. How is customer satisfaction assessed?
8. What types of data on quality performance are collected?
9. Is a Process Improvement Plan prepared?
10. Does the organization participate in benchmarking activities?
PROJECT HUMAN RESOURCE MANAGEMENT
1. How long have you been a Project Manager?
2. What are your responsibilities as a Project Manager?
3. What types of project management training have you received?
4. What types of project management training do you feel would be useful to you in your job as a Project Manager?
5. How were you selected to be a Project Manager?
6. How were your team members selected?
7. Are there any incentives to completing projects on time, within budget, and according to scope? If no, what types of incentives would you recommend?
8. Is a Responsibility Assignment Matrix prepared?
9. Is a Staffing Management Plan prepared?
10. Is a project team directory compiled?
11. Is a Resource Breakdown Structure prepared?
12. What types of team-building activities are conducted?
13. Do you provide input into each team member’s performance appraisal?
14. Is it often necessary to escalate conflicts to higher level managers for resolution?
15. Is a team charter prepared?
PROJECT COMMUNICATIONS MANAGEMENT
1. Is a stakeholder analysis conducted?
2. Is a Communications Management Plan prepared?
3. Are issue logs used for tracking stakeholder concerns?
4. Are final project reviews held? If yes, is there a standard process for these reviews?
5. Are lessons learned documented? If yes, how are they used?
6. Are forecasts of project performance prepared?
7. Are time reporting systems established?
8. Are cost reporting systems established?
9. Are regular status review meetings held?
10. Is feedback from stakeholders routinely solicited?
PROJECT RISK MANAGEMENT
1. Is a Risk Management Plan prepared? If yes:
a. Who was involved in preparing the plan?
b. Is this organization risk averse, risk neutral, or risk seeking/proactive?
2. How are risks identified on projects?
3. What risk identification methods are used?
4. Do the identified risks map to the WBS?
5. Are risks organized into categories? If yes, what categories are used?
6. What types of risk analyses are performed?
7. In preparing risk responses:
a. How are risk owners identified?
b. Is there a standard contingency reserve?
c. Is there a risk budget?
d. What metrics are used to measure the effectiveness of the risk response strategy?
8. Are risk reviews held?
9. What types of data are collected to monitor and control risks?
10. How is earned value used in project risk management?
11. How often are workarounds or corrective actions needed because of risks?
12. Are lessons learned documented concerning risk management?
13. Is a risk repository used?
14. Are variance and trend analyses conducted?
15. Are fallback plans prepared?
PROJECT PROCUREMENT MANAGEMENT
1. Is a make-or-buy analysis conducted?
2. Is a Procurement Management Plan prepared?
3. What are your responsibilities in procurement management activities versus those performed by the contracting unit?
4. Are contract administration activities assigned to team members?
5. Is there a contract change control system?
6. What type of information is collected to assess supplier/vendor performance?
7. Is formal notification provided to suppliers when their work has been completed satisfactorily?
8. Does the organization regularly work with the same suppliers on its projects?
9. Does the organization have a list of approved suppliers/vendors?
10. Is a standard checklist followed for contract closeout?
INSTRUMENT 4–D
Maturity Assessment
(abbreviated form)
This instrument can provide a preliminary determination of the organization’s maturity level. If the responses to more than 75% of the statements corresponding to each level are affirmative, then the organization is likely at that maturity level.
If these statements describe your project management environment, the organization is at Level Zero:
• No procedures
• No policies
• No formal planning
• No formal monitoring
• No formal collection of project history
• Projects consistently miss targets for cost
• Projects consistently miss targets for duration
• Projects consistently miss targets for scope
• Projects consistently miss targets for quality
If these statements describe your project management environment, the organization is at the Initial Level:
• Inconsistent procedures and results
• Sporadic team support
• Inconsistent familiarity with other team members’ areas of expertise
• Lack of project management training
• Little interaction among stakeholders
• Lack of confidence in teamwork
• Inappropriate conflict management policies
• Disorganized staffing decisions
• No formalized project purpose
• Limited attention to planning
• Limited use of the Work Breakdown Structure
If these statements describe your project management environment, the organization is at the Developed Level:
• Organizational support for projects
• Formalized project charter
• Clear definition of roles and responsibilities
• Formalized project Scope Management Plan
• Formalized project Scope Statement
• Formalized team formation
• Formalized Project Management Plan
• Formalized project Communications Management Plan
• Well-defined performance criteria
• Some progress monitoring and data clearinghousing
If these statements describe your project management environment, the organization is at the Enhanced Level:
• Projects linked to organizational strategic goals
• Formalized project management methodology used
• Common purpose for project teams
• Standard project templates
• Streamlined project team formation
• A visible, participatory project team culture
• Formalized individual and team performance evaluation
• Formalized project progress monitoring policies
• Routine collection and use of historical data
If these statements describe your project management environment, the organization is at the Advanced Level:
• Organizational recognition of the importance of projects
• Organizational support of project teams
• Project teams involved in policy issues
• Open communication and a visible atmosphere of trust
• Quantitative metrics for project evaluation
• Formalized mentoring program
• Collaborative leadership
• Team effectiveness metrics
• Team and individual rewards and recognition
If these statements describe your project management environment, the organization is at the Leader Level:
• Projects regarded as necessary for organizational transformation and innovation
• Teams considered a strategic resource
• Continuous improvement of templates and processes
• Project teams deployed appropriately
• Risks viewed as opportunities
• Professional responsibility emphasized
• Team environment conducive to professional creativity
• Formalized improvements in project performance
• Experiences provide the foundation for continuous improvement
INSTRUMENT 4–E
Organizational Metrics Categories
EXAMPLES OF PERFORMANCE METRICS
• Completeness of requirements
• Accuracy of the cost estimate
• Extent of rework
• Number of key milestones completed
• Number of key milestones missed
• Use of the Work Breakdown Structure (WBS) to develop project plans
• Use of the team charter to manage conflicts
• Resource utilization versus the plan
• Expected results and actual results in testing
• Effectiveness of risk response strategies in mitigating risks
• Vendor progress in meeting schedule, cost, and performance
• Extent of requests for information outside of regular communications
EXAMPLES OF STABILITY METRICS
• Effectiveness of scope, schedule, and cost-tracking processes
• Value of cost tools and techniques in managing projects
• Value of scheduling tools and techniques in managing projects
• Effectiveness of contract change management system
• Revisions to subsidiary plans of the overall Project Management Plan in
– Procurement management
– Cost management
– Quality management
– Schedule management
– Scope management
EXAMPLES OF COMPLIANCE METRICS
• Product conformance with requirements
• Effort required to use the standard project management information system
• Timeliness of project information
• Customer acceptance of product deliverables
• Extent of tools and templates available to the team
• Extent of changes to the cost baseline
• Number of workarounds required
• Number of conflicts requiring escalation outside the project team
• Applicability of the methodology for the range of projects under way by the organization
EXAMPLES OF CAPABILITY METRICS
• Use of knowledge, skills, and competency profiles
• Participation in project management career path
• Participation in mentoring programs
• Extent of improvement of project predictability
• Extent to which each team member is an active participant on the team
• Success of projects undertaken by the team
• Status of the team’s best practices in project management
• Use of models for schedule, cost, and performance
• Capability and ease of use of the team’s integrated systems
EXAMPLES OF IMPROVEMENT METRICS
• Involvement of individual team members in performance improvement initiatives
• Impact of each improvement proposal in terms of increasing capability in one of the project management competencies
• Effect of technology in terms of performance improvement
• Optimization of the motivations and viewpoints of the client and the project team
• Benchmarking data within the industry and even outside of the industry
Adapted from Parviz F. Rad and Ginger Levin, The Advanced Project Management Office (Boca Raton, FL: CRC Press). © 2002. Reproduced with permission of Routledge/Taylor & Francis Group, LLC.
INSTRUMENT 4–F
Effectiveness of the Project Management Office
This instrument can help determine the extent to which the organization benefits from its existing Project Management Office (PMO). The effectiveness of the PMO is a function of both its sophistication and the organization it supports.
At the broadest level, the average score for all statements provides a score (between 1 and 5) for the effectiveness of the organization’s PMO. The resulting score represents a first approximation of organizational maturity, which also is measured on a scale of 1 to 5 in staged maturity models.
1—Strongly Disagree/Not Available
2—Disagree
3—Neutral/No Opinion
4—Slightly Agree
5—Definitely Agree
INSTRUMENT 4–G
Post-Delivery Project Audit Checklist
Use this instrument to assess the level of sophistication of project management activities. Depending on when the audit is performed, some questions might not be applicable. Record the percentage of questions in each category that were answered in the positive or the affirmative. Comparing scores for each category with the organizational average provides an indication of improvement over the life of the project.
INITIATING ACTIVITIES
Determine whether the project charter was helpful in resolving any conflicts concerning the Project Manager’s authority and responsibilities.
Determine the effectiveness of the statement of work in developing the charter.
Assess whether the organizational, environmental, external assumptions, and constraints identified in the charter were applicable.
Determine whether the Project Manager had the needed authority, as expressed in the charter, given the nature of the project.
Assess the various environmental factors and systems that surrounded and influenced the project as the charter was developed.
Determine whether the project was linked to the organization’s strategic plan.
Assess the various organizational processes and procedures that were used in developing the project charter.
Determine the usefulness of the project management methodology in developing the charter and the preliminary scope statement.
Assess the project’s initial estimate of its financial contribution.
Determine the usefulness of the preliminary scope statement in defining the project; its characteristics, boundaries, associated products, and services; and methods of acceptance and scope control.
PLANNING ACTIVITIES
Determine whether a scope statement was prepared.
Determine whether the Project Management Plan covers the entire project life cycle and follows the project management methodology.
Assess the usefulness of the project management methodology throughout the project planning process.
Determine whether a performance measurement baseline was prepared.
Assess the level of involvement of the project team, clients, and other stakeholders in developing the Project Management Plan.
Determine whether external factors, constraints, or assumptions were identified correctly.
Assess the stability of the requirements and whether they were defined sufficiently at the beginning of the project.
Determine the extent of use of the Work Breakdown Structure (WBS) as a framework for planning the project.
Assess the usefulness of the project’s WBS to serve as a template for future projects.
Determine whether a scope or technical baseline was established.
Assess the usefulness of the Scope Management Plan.
Determine whether an activity list was prepared and whether it can serve as a template for future projects.
Determine whether the project team established too many discretionary dependencies that limited schedule options.
Assess the interface between risk management and schedule preparation.
Determine whether a schedule was prepared.
Determine whether a schedule baseline was prepared.
Assess the usefulness of the Schedule Management Plan.
Assess the accuracy of resources in terms of actual resource requirements.
Assess the accuracy of the cost estimate that was prepared.
Determine whether a cost baseline was established.
Assess the usefulness of the Cost Management Plan.
Determine whether changes were needed to the organization’s quality policy to meet the needs of the project.
Determine whether a quality baseline was established.
Determine whether a Process Improvement Plan was prepared.
Determine whether a Quality Management Plan was prepared.
Assess the usefulness of quality system.
Determine whether the cost of quality was considered.
Determine whether a Resource Assignment Matrix was prepared.
Assess the usefulness of the Resource Breakdown Structure.
Assess the usefulness of the Staffing Management Plan.
Determine whether qualified staff members were available for the project at the required times.
Assess whether any stakeholders exerted negative influence over the project.
Assess the usefulness of the Stakeholder Management Plan.
Assess the usefulness of the Communications Management Plan.
Assess the level of commitment to project risk management.
Assess the usefulness of the Risk Management Plan.
Determine whether the risk classification system that was used can serve as a template for other projects.
Evaluate the usefulness of risk identification approaches for future projects.
Determine whether the organization’s risk repository was used.
Determine whether a make-or-buy analysis was conducted.
Assess the usefulness of the Procurement Management Plan.
Determine whether the project team actively worked with the procurement department to provide input before any solicitations were issued and during the source selection process.
EXECUTING ACTIVITIES
Determine whether client expectations were solicited, included, met, or exceeded.
Determine whether the client was informed of progress, changes, and delays.
Evaluate recommendations from audits in terms of quality and project improvements.
Determine the usefulness of client satisfaction surveys to identify needed process improvements.
Determine if the project team was a high-performing team that could work well together on a future project.
Assess whether the Project Manager had input into team member performance evaluations.
Determine whether a team-based performance evaluation system was used.
Determine the usefulness of any training conducted in terms of improvements of both individual and team skills.
Determine potential improvements in team development.
Determine the usefulness of the team charter.
Evaluate the usefulness of the project’s information distribution system.
Determine whether qualified lists of approved vendors were used for project purchases.
Determine how changes were authorized to the project’s scope.
MONITORING AND CONTROLLING ACTIVITIES
Assess the causes of major scope, time, and cost deviations.
Determine whether earned value management was used.
Assess whether formal client acceptance of deliverables was obtained, based on defined acceptance criteria.
Assess the extent of scope changes during the project.
Determine the extent of any schedule delays.
Assess schedule updates that resulted from scope change.
Determine the extent of any schedule corrective actions.
Assess budget updates that resulted from scope changes.
Evaluate the severity of cost variances.
Determine what was required to bring expected performance in line with the Project Management Plan.
Assess the reasons key corrective actions were selected.
Assess client feedback throughout the project in terms of meeting quality standards.
Ascertain the usefulness of the project review process.
Evaluate the use of earned value analysis and trend analysis to predict future project status and progress.
Evaluate the usefulness of the reports that were prepared to facilitate forecasting.
Ascertain the usefulness of a project risk audit to examine the project risk management process.
Determine whether the risk ranking was used to make decisions concerning the project at project review meetings and to assess trends in terms of the effect of risk on project objectives.
Assess the usefulness of contingency plans in reducing the cost of actions, if risks occurred.
Assess the number of workarounds that occurred, as a measure of the effectiveness of risk responses.
Assess the effectiveness of the contract change control system.
CLOSING ACTIVITIES
Assess whether the client provided formal acceptance of project deliverables.
Assess whether suppliers received formal notification when contracts were completed.
Determine whether records management policies were followed.
Assess whether the employee skills database was updated.
Determine whether lessons learned were collected throughout the project and made part of the organization’s knowledge repository.
Assess whether a closeout checklist was used.
INSTRUMENT 4–H
Assessing the Usefulness of Templates/Processes
The overall level of utility of using templates and processes is the average of the scores for the questions below. This index constitutes a baseline upon which to improve the success of future projects based on team feedback regarding performance on the current project.
1—Not Applicable/Not Done
2—Rarely
3—Sometimes
4—Often
5—Always
INSTRUMENT 4–I
Documenting Lessons Learned
This instrument helps determine the extent to which the organization encourages project team members to collect and use lessons learned. Ultimately, this instrument can be used to develop a baseline for continuous improvements in the project management environment.
At the broadest level, the average score for all questions provides a score (between 1 and 5) for utilizing organizational experience. This is an initial approximation of organizational maturity, which also is measured on a scale of 1 to 5. Summarizing the scores for all the questions in each knowledge area (scope, cost, quality, etc.) provides the next level of detail. Summarizing the scores for each of the three categories of each of the knowledge areas (i.e., process, compliance, and efficiency) provides the final level of sophistication in analyzing these data.
1—Strongly Disagree
2—Disagree
3—Neutral
4—Agree
5—Strongly Agree
INSTRUMENT 4–J
Customer Satisfaction
Use this index as a baseline upon which to improve the success of future projects based on customer feedback. The overall customer satisfaction index is the average of the scores for each question below.
1—Never
2—Rarely
3—Sometimes
4—Often
5—Always
