4.3.1 Objectives Hierarchies for Different Management Levels

Executive-level objectives are the strategic objectives of the enterprise, derived in this case from the NASA Strategic Plan, and are considered to have a time frame of more than 10 years. Two strategic objectives are considered in this example, as shown in Figure 4.1.

Programmatic-level objectives are concerned with design, development, fabrication, fielding, and operation of systems that support the various strategic objectives that relate to NASA's mission. Four programmatic-level objectives are included in this example, as shown in Figure 4.2, differentiated according to the time frames over which they apply.

Center-level objectives fall into two different categories: program/project support and development of the institutional capability. Two of each category are considered in this example, as shown in Figure 4.3. Like programmatic-level objectives, they are differentiated according to the time frames over which they apply.

4.3.2 Integrated Objectives Hierarchies for the Enterprise as a Whole

Figure 4.4 illustrates for the example demonstration how the objectives hierarchies at executive, programmatic, and center levels are integrated into a single objectives hierarchy that maintains the relationships between objectives within each level and introduces the principal interfaces that exist across levels. In general, the programmatic-level objectives support one or more executive-level objectives, whereas the center-level objectives may support both the programmatic-level objectives and one or more executive-level objectives. For example, the objective numbered C (5–10) #6 and titled “Maintain a Sufficient Cadre of Highly Capable Analysts” directly supports both objective P (5–10) #4 titled “Design, Build, Deploy, and Operate the Next Generation Space Telescope,” and objective E (>10) #2 titled “Attract and Advance a Highly Skilled Workforce, Cultivate an Innovative Work Environment, and Provide the Facilities, Tools, and Services Needed to Conduct NASA's Missions.”⁴

4.4.1 Known Risk and Opportunity Scenarios

Based on the information pertinent to JWST and HST presented in Section 4.2, a total of eight individual risks and one individual opportunity are postulated for this demonstration. Two of the individual risks are assigned to strategic objective E(>10) #1, as shown in Figure 4.5, and one of those is also assigned to objective E(>10) #2 because it directly affects both objectives. Also shown are suggested leading indicators that apply to each risk.

In Figure 4.6, another three risks, with associated leading indicators, are assigned, respectively, to programmatic objectives at the top level, the mid-term level, and the short-term level, and in Figure 4.7, two risks, with associated leading indicators, are assigned to mid-term mission support objectives C (1–5) #9 and C (1–5) #10 in the institutional category. Finally, in Figure 4.8, a single opportunity, with associated leading indicators, is identified for strategic objective E (>10) #1, along with three introduced risks, with their own associated leading indicators, that would be of concern if the opportunity were acted on.

4.4.2 Cross-Cutting Risks and Opportunities

Risk and opportunity scenarios identified during the practice of EROM may be cross-cutting in several ways:

• Organizationally cross-cutting scenarios affect multiple organizational units within an enterprise. All the risks and opportunities in Figures 4.5 through 4.8 are cross-cutting in this sense.
• Programmatically cross-cutting scenarios affect multiple programs and/or projects within the enterprise. One of the two risks in Figure 4.5 (the one placed higher in the chart) and both risks in Figure 4.7 are programmatically cross-cutting.
• Strategically cross-cutting scenarios directly affect multiple high-level objectives in the objectives hierarchy. The risk in Figure 4.5 identified in the previous bullet as being programmatically cross-cutting is also strategically cross-cutting.
• Pan-agency cross-cutting scenarios affect more than one agency. Such risks occur when agencies are involved in a cooperative effort.

Furthermore, it may be observed in Figures 4.5 through 4.8 that certain leading indicators may affect multiple risk scenarios and/or multiple objectives. For example, the indicator “complexity of design” affects a risk scenario and two objectives in Figure 4.5, as well as two risks and two objectives in Figure 4.6. This may be thought of as a “cross-cutting” leading indicator.

The EROM methodology is designed to promote consideration of cross-cutting risks, opportunities, and leading indicators by allowing for them to be entered into the accounting wherever is appropriate. Multiple listings of the same scenario or indicator under different entities, programs, or objectives is not a problem so long as they are treated consistently each time they are encountered. The use of taxonomies, as described in Section 3.4.1, can facilitate the identification of cross-cutting risk and opportunity scenarios and cross-cutting leading indicators.

4.4.3 Unknown and Underappreciated Risks

The process of identifying risk and opportunity scenarios is, of course, aimed at known risks and opportunities. In addition to known scenarios, however, the potential for unknown and/or underappreciated (UU) risks must be considered when determining the overall likelihood of success or failure (i.e., the cumulative risk of not being able to satisfy the top objectives).

It may be noted that several of the leading indicators listed in Figure 4.5 through 4.7 pertain either directly or indirectly to sources of UU risk. For example, “complexity of design” and “complexity of integration and testing” are direct exemplars of the first UU factor cited in Section 3.4.5: “amount of complexity, particularly involving the interfaces between different elements of the system.” In addition, the following leading indicator cited in Figure 4.6: “severity of unresolved technical issues for cryocooler development,” is an indirect exemplar of the third UU factor in Section 3.4.5: “use of fundamentally new technology or fundamentally new application of an existing technology.” If the assessment in this example had been performed prior to 2010, the sixth UU factor in Section 3.4.5 might also have been cited as a leading indicator of future risks about former management deficiencies pertaining to subcooler development: “degree of oversight when responsibilities are distributed among various entities” (NASA 2016b; HubbleSite.org 2016).

Beside these indicators, there are also other leading indicators (see Section 3.4.5) that tend to be correlated with the occurrence of UU challenges. Two of the most important are:

• Pressures to meet extremely tight schedules and/or budgets, particularly in combination with a complex set of tasks
• Deficiencies of management such as failure to maintain adequate oversight of distributed suppliers and failure to respect and promote open communication

In addition to the ranking of cumulative known risks and opportunities for each objective, therefore, the principal outcome of the demonstration will also include a ranking of the cumulative level of concern for UU risks based on the leading indicators that pertain to UU risks, along with a list of the key attributes of the UU indicators that drive that cumulative ranking.⁵

4.5.1 Risk and Opportunity Identification Template

Table 4.2 presents the Risk and Opportunity Identification Template, which collects the information about known risks and opportunities presented in Section 4.4. It does this by tabulating the identified individual risk, opportunity, and introduced risk scenarios, the leading indicators for each scenario, and the objective to which each scenario is assigned.

4.5.2 Leading Indicator Evaluation Template

Table 4.3 presents the Leading Indicator Evaluation Template, which is used to assign watch and response trigger values to each leading indicator, record the current value of the indicator, and provide an indication of the trend. As explained earlier, leading indicator trigger values are used to signal when a risk is reaching a risk tolerance boundary or when an opportunity is reaching an opportunity appetite boundary. Reaching the trigger values for risk leading indicators implies that the likelihood of not being able to satisfy an element of the strategic objectives hierarchy is becoming a concern. Reaching the trigger values for opportunity leading indicators implies either that there is a potential for significantly increasing the likelihood of being able to satisfy an element of the strategic objectives hierarchy or that there is an emerging opportunity to achieve new goals and objectives that were formerly considered unreachable or inconceivable. Exceedance of a watch trigger suggests that an action should be considered but is not imminently needed. Exceedance of a response trigger suggests that an action may be imminently needed (e.g., mitigation of the risk or exploitation of the opportunity).

Italic typeface denotes repeated instances of leading indicators

The level of concern for a given leading indicator is determined by where the current value of the leading indicator and/or a projected future value lie with respect to the watch and response trigger values. In Table 4.3, the projected future value one year from the present (referred to earlier as the “trend”) is shown as a range to account for uncertainty. For conceptual purposes, it may be helpful for some people to think of the uncertainty range as being more-or-less a 90 percent confidence interval for the future value of the leading indicator. However, if confidence levels are used in this context, they should be thought of as qualitative degrees of belief and not statistical quantities.

Table 4.4 illustrates how the Leading Indicator Evaluation Template might be completed for short-term programmatic objective P (1) #11, titled “Deliver the cryocooler subsystem.” The entries are based principally on information about the JWST that was previously itemized in Section 4.2—that is, publicly available material. The use of this information is summarized in the columns labeled “Rationale.” In short, the template records the following information:

• The 100 percent reduction during the past year in the schedule margin for development of the cryocooler (leading indicator 1) has caused concern about the cryocooler delivery date and the schedule of the program as a whole.
• The 40 percent reduction during the past year in the cost margin for the program as a whole (leading indicator 2) has caused concern about the amount of resources that can be reallocated to the cryocooler.
• The fact that several significant technical issues have not yet been completely resolved (leading indicator 3) has caused further concern.
• A somewhat negative progress report written by the GAO (leading indicator 4) about management problems concerning the cryocooler development has again caused concern.
• A trending analysis by JPL implying that the schedule margin will not be further degraded (one-year projected value for leading indicator 1) has alleviated some concern, particularly since the analysis indicates that the schedule for integrating the cryocooler with the spacecraft and beginning the integrated testing should remain intact.
• The warning signs discussed by the GAO, however, suggest there is a large amount of uncertainty in JPL's estimate, leading to additional concern.

4.6.1 Objectives Interface and Influence Template

While the principal interfaces between the objectives were displayed in Figure 4.4, a number of secondary interfaces could also be postulated. For this demonstration, three secondary interfaces between the top level of the programmatic and mission support objectives and the executive level strategic objectives are also considered, as shown in Figure 4.9.

Two of the secondary objectives account for the fact that the success of the exploratory programs and projects (objectives P (5–10) #3 and P (5–10) #4) influence the success of attracting a highly skilled workforce and providing the needed facilities (objective E (>10) #2) by defining the technical qualifications that are needed within the workforce, by providing an incentive for qualified technical people to work at NASA, and by providing the driving function for the facilities to be developed. The third recognizes the fact that maintaining a sufficient cadre of highly capable analysts (objective C (5–10) #6) is necessary in order to successfully interpret the data obtained from the telescopes during their operation and to set the direction for additional observations.

The objectives interface and influence template encodes this sort of information in tabular form, as shown in Table 4.5.

4.6.2 Known Risk Roll-Up Template

The aggregate risk of not successfully meeting an objective can be evaluated by rolling up the individual risk scenarios in either of two alternative ways. The first alternative is illustrated in Figure 4.10, with reference to one of the top objectives in Figure 4.4, and consists of the following steps:

1. Identify the objective of interest from the objectives hierarchy (Figure 4.4).
2. Identify the risk scenarios associated with the objective from the Risk and Opportunity Identification Template (Table 4.2).
3. Identify the risk leading indicators associated with each risk scenario from the Risk and Opportunity Identification Template (Table 4.2), and evaluate the leading indicator levels of concern using the Leading Indicator Evaluation Template (Table 4.3).
4. Roll up the leading indicator levels of concern for each risk scenario to obtain a corresponding level of concern for each risk scenario using a transparent and documented roll-up rationale.
5. Roll up the levels of concern for the risk scenarios, obtained from the previous bullet, with the levels of concern obtained for the interfacing objectives, using the Objectives Interface and Influence Template (Table 4.5) to identify the interfacing objectives. (The order of the roll-up is such that the levels of concern for the interfacing objectives will have been determined prior to the present roll-up.)

The rolled-up level of concern at the end of the process is defined as the aggregate, or cumulative, risk of not meeting the objective, and the transparent and documented rationale for the roll-up process defines the justification for the aggregate risk.

The second alternative for rolling up levels of concern to obtain the aggregate risk of an objective is illustrated in Figure 4.11. The difference between it and the first alternative is that it cuts out one of the steps. Specifically, the aggregate risk of the objective is determined by rolling up the levels of concern for the leading indicators directly (together with the levels of concern for the interfacing objectives), without first performing a roll-up from the leading indicators to the risk scenarios. The rationale is that the leading indicators are de facto surrogates for the risk scenarios, and so it is as reasonable to infer the aggregate risk of not meeting an objective from the levels of concern associated with the leading indicators as it is to infer the aggregate risk from the levels of concern of the risk scenarios. Thereafter, it is possible to determine levels of concern for each risk scenario by performing an after-the-fact roll-up from leading indicators to individual risk scenarios in order to assess the importance of each risk scenario.

Using the second alternative, the aggregate risk of not being successful in delivering the cryocooler subsystem in a timely manner (Objective P (1) #11) is reflective of a roll-up of the levels of concern for Leading Indicators 1 through 4 and the aggregate risk of not being able to provide the needed technical support for this task (Objective C (1) #12), as shown in Figure 4.12.

The associated roll-up is performed on the Known Risk Roll-Up Template, Table 4.6. A similar template for opportunity roll-up will be discussed in the next subsection.

Italic typeface denotes repeated instances of leading indicators          Asterisk denotes secondary interfaces

For each objective, starting from the bottom and working up to the top, the Known Risk Roll-Up Template lists all the risk indicators and interfacing objectives that feed into it. As mentioned, these inputs are obtained from prior templates. The following paragraphs discuss the additional information that is contained in the template, starting from the sixth column.

4.6.3 Opportunity Roll-Up Template

The Opportunity Roll-Up Template, Table 4.9, is comparable to the Known Risk Roll-Up Template, Table 4.6, with the exception that the opportunity scenarios generally have accompanying introduced risks. For the example being considered, the opportunity scenario (as entered into the Risk and Opportunity Identification Template, Table 4.2) is that it may be possible for significant new technology advancements (such as a camera with improved resolution) to be delivered and installed on the telescope through retrofitting either by astronauts or robotically, since the JWST spacecraft is designed with a grapple arm. This could be the case even though the JWST is described by NASA as unserviceable, given that the incentive for trying such a rendezvous mission is strong enough to justify the risks. In this case, the introduced risks are that the likelihood of loss of crew (LOC) or loss of mission (LOM), depending on whether the mission is crewed or robotic, may be unacceptably high, and/or that the cost of the retrofit mission may likewise be too high. The leading indicators for the opportunity are the technology readiness level for improved-resolution infrared cameras and the readiness level for the Space Launch System (SLS), including Orion if the mission is crewed. The leading indicator for the introduced risk pertaining to the probability of LOC or LOM is the most current predictive estimate of P(LOC) or P(LOM) obtained from a probabilistic risk assessment (PRA) for an analogous SLS/Orion mission involving lunar orbit. The leading indicator for the introduced risk pertaining to cost is likewise the most current cost estimate for the analogous mission.

Since the only opportunity scenario for this example is introduced at the top level of the objectives hierarchy (i.e., at Objective E(>10) #1), there are no roll-ups for opportunity from lower-level objectives to higher level objectives. Rather, the roll-up in this example concerns only the relevant leading indicators for opportunity and introduced risk at the level of Objective E(>10) #1 (which is entered at the bottom of Table 4.9). Before performing this roll-up, the significance of the opportunity leading indicators and the tolerability of the introduced risk leading indicators are transferred from the Leading Indicator Evaluation Template (Table 4.3) to the column labeled “Leading Indicator Significance” in Table 4.9. The roll-up for the aggregate opportunity in the next-to-last column of Table 4.9 is based on the perceived balance between the opportunity and the introduced risks as informed by the values of the respective leading indicators. The rationale for the perception of balance is recorded in the last column of Table 4.9.

An example opportunity roll-up is shown in Table 4.10. In this example, the likelihood of being able to conduct a service mission to install a significantly improved IR camera is considered high based on anticipated technology developments, and so the opportunity leading indicators are colored blue (significant opportunity). Less happily, the present-day cost of a rendezvous mission is considered intolerable by today's standards, and hence the leading indicator for the introduced risk associated with cost is red (intolerable). However, the cost may be perceived to be more tolerable once the system is operational because of the following rationale:

• Public enthusiasm for the program will likely increase substantially, as it did for Hubble, once the telescope is operational and its scientific value is fully appreciated.
• Issues about the feasibility of performing a rendezvous will likely abate once the SLS/Orion system becomes operational and is shown to be reliable and safe.
• The economic recovery will likely increase the willingness of the country to spend more on space.

Since it is considered likely that the perceived cost of the rendezvous will become more tolerable over time, the aggregate opportunity considering leading indicators for both opportunity and introduced risks is ranked as marginal rather than insignificant.

4.6.4 Composite Indicator Identification and Evaluation Template

In the examples shown so far, all of the leading indicators have been considered to be standalone indicators, even though the templates have included the possibility for introducing composite indicators. Considering them to be independent in setting trigger values is acceptable as long as the rationale for aggregating them accounts for the more complex relationships that may exist between them. Thus, ameliorating factors were introduced in the rationale column in Table 4.7 to justify an aggregated risk that was yellow (marginal) rather than red (intolerable), and in Table 4.10 to justify an aggregated opportunity that was violet (marginal) rather than beige (insignificant), even though one of the indicators in each case was red.

Using a composite indicator recognizes the fact that the trigger values of some of the indicators may depend on the values of other indicators. For example, in the example for the Leading Indicator Evaluation Template in Table 4.4, leading indicators 1, 2, and 3 have just such a codependency that would justify the use of a composite indicator. That is, the amount of schedule margin needed for cryocooler development (Indicator 1) depends on both the amount of cost margin from other tasks that can be diverted to the cryocooler development task (Indicator 2) and the severity of the technical issues that remain to be resolved (Indicator 3). Note that a composite indicator does not necessarily collapse all the individual indicators for a risk or opportunity scenario. For example, Leading Indicator 4, which concerns GAO's evaluation of the cryocooler development progress, remains as a separate indicator for the risk of not delivering the cryocooler subsystem when required.

The following box provides an example of how one might define a composite indicator to recognize the dependencies between leading indicators 1, 2, and 3.

Table 4.12 illustrates how the Known Risk Roll-Up Template example in Table 4.7 would change as a result of using this composite indicator. Note that the Roll-Up result remains the same but there is no longer a red (intolerable) risk indicator feeding into it.

4.6.5 UU Risk Roll-Up Template

In addition to the leading indicators for known risks provided in the Risk and Opportunity Identification Template, Table 4.2, and the Leading Indicator Evaluation Template, Table 4.3, there is an additional set of leading indicators that correlate with the relative importance of UU risks. Some of the more important ones were identified in Sections 3.4.5 and 4.4.3.

Just as the known risks were rolled up via the Known Risk Roll-Up Template, Table 4.6, to obtain insight into how known risks are affecting the likelihood of being able to satisfy the top objectives of the organization, it is possible to roll up the leading indicators for UU risks via a UU Risk Roll-Up Template to obtain insight into how UU risk indicators are affecting the likelihood of successful objectives achievement. For this example, we postulate the following two UU leading indicators to be the most significant sources of UU risk:

• Schedule/budget pressure
• Quality of oversight and communication

Both indicators are measured on a scale of 1 to 5 for each objective, with the lower end indicating lowest concern and the upper end indicating highest concern. The steps in the Roll-Up are identical to Section 4.6.2. The example template is shown in Table 4.13.

Italic typeface denotes repeated instances of leading indicators          Asterisk denotes secondary interfaces

4.7.1 Risk and Opportunity Driver Identification Template

As discussed in Section 3.6.1, a risk driver causes the cumulative risk for one or more top organizational objectives to change color from green to yellow or red or from yellow to red. Risk factors might include any combination of departure events, underlying causes of departure events, leading indicators, unprotected key assumptions, deficiencies in internal controls, or other factors that affect the risk of meeting the objectives. Similarly, an opportunity driver can be any combination of opportunity factors/elements that collectively cause the cumulative opportunity for one or more top organizational objectives to change color. Risk or opportunity drivers, as already defined, constitute a detailed resolution of the principal factors that contribute to the cumulative risk or opportunity. They therefore are suitable for identifying the constituents of risk mitigations, opportunity actions, and internal controls.

As discussed in Section 3.6.2, a risk or opportunity scenario driver is any combination of risk or opportunity scenarios that cause the cumulative risk or opportunity of one or more top objectives to change color. Risk and opportunity scenario drivers provide a higher-level view of the concerns that need to be addressed, and are therefore suitable for summary presentations.

The Risk and Opportunity Driver Identification Templates facilitate the process of identifying both drivers (henceforth called constituent drivers for clarity) and scenario drivers. Table 4.14 illustrates the process schematically for the next-generation space telescope example. A table similar to Table 4.14 is prepared for each strategic/top objective of the organization.

In Table 4.14, two risk scenarios are identified as candidate risk scenario drivers for the objective “Discover how the universe works” The first is “Failure to deliver the cryocooler subsystem on time,” a scenario that most directly affects the lower-level objective of delivering the cryocooler subsystem but also propagates up through the Known Risk Roll-Up Template to the top objective. The second is “Unavailability of expert technical staff for review.” This second risk scenario was not included in the earlier development of risks for this example, but is added here to make a point.

As shown in Table 4.14, neither of the risk scenarios causes the color of the cumulative risk for the top objective to change from yellow to green, but the combination of the two scenarios does.

Therefore, the risk scenario driver consists of the combination of the scenarios. The associated risk driver constituents, or principal factors that contribute to the cumulative risk, are shown in the next-to-last column of Table 4.14. They are the factors that are noted in the Known or UU Risk Roll-Up Template as being of particular concern, such that all of them have to be addressed in order for the cumulative risk to be reduced to a tolerable state.

It may be noted that although there are no UU risks included in this example for identifying risk drivers, there is no reason why they should not be included if the UU Roll-Up Template shows that they are a large source of concern.

The lower part of Table 4.14 concerns the identification of opportunity drivers. In this example there is only one opportunity scenario, and it is a driver because the deletion of the scenario causes the color of the cumulative opportunity for the top objective to change from blue (significant) to beige (insignificant). The associated opportunity driver constituents, shown in the next-to-last column, are the factors that are noted in the Opportunity Roll-Up Template as being of particular concern.

Also indicated in the Risk and Opportunity Driver Identification Template is the spare time available to initiate a response to mitigate the risk drivers or an action to avail the opportunity drivers without exceeding the available time frame to complete the response. The combination of the identified driver scenarios, the driver identified constituents, and the time frame available to begin a response for each constituent can be illustrated in a matrix format, as shown in Figures 4.13 and 4.14. This form of display is particularly useful for high-level presentations.

4.7.2 Risk and Opportunity Scenario Likelihood and Impact Evaluation Template

As discussed in Sections 2.5.3 and 2.5.4, the likelihoods and impacts of individual risk and opportunity scenarios need to be assessed separately by virtue of the latest draft of OMB Circular A-123. In Section 3.6.3, a ranking scheme of high, medium, and low was presented for likelihoods based on the decision maker's risk tolerance and opportunity appetite, and for impacts based on the identification of risk and opportunity scenario drivers. Suggested ranking criteria were depicted in Tables 3.3 and 3.4. The Risk and Opportunity Scenario Likelihood and Impact Evaluation Template in Table 4.15 provides a means for actualizing the approach in Sections 2.5.3, 2.5.4, and 3.6.3.

4.7.3 Risk Mitigation, Opportunity Action, and Internal Control Identification Templates

As discussed in Section 3.6.4, the identification of risk mitigations, opportunity actions, and internal controls is directed at finding viable ways to act on the risk and opportunity drivers that are identified after the risk and opportunity roll-up processes. This identification is performed on the Risk Mitigation and Internal Control Identification Template, demonstrated in Table 4.16 for the next-generation space telescope example, and on the Opportunity Action and Internal Control Identification Template, demonstrated in Table 4.17 for the same example. The former starts from the risk drivers and the latter from the opportunity drivers that were identified in the Risk and Opportunity Driver Identification Template, Table 4.14.

For each risk or opportunity driver, there are one or more proposed mitigations or actions intended to respond to the driver, and for each mitigation or action, there are one or more internal controls intended to provide assurance of success. The internal control is labeled “Assumption” if the intent is to protect an assumption made in defining the mitigation or action, and “Deficiency” if the intent is to address a shortcoming in the present internal controls. These proposed mitigations, actions, and controls are purely hypothetical and intended for illustration purposes only.

4.7.4 High-Level Display Template

The High-Level Display Template, shown in Table 4.18, displays results obtained from the preceding templates in a condensed form. It also includes suggested responses and internal controls that address the risk and opportunity drivers. The entries that are completed in Table 4.18 correspond to those presented in Tables 4.7 and 4.10 and in Tables 4.16 and 4.17.

4.8.1 Scope of the Problem

The demonstration in the preceding sections was of limited scope, involving only 12 objectives, of which two were strategic objectives with time frames greater than 10 years, four were top programmatic and mission support objectives with time frames of 5 to 10 years, four were mid-term performance objectives with time frames of 1 to 5 years, and two were short-term performance objectives with time frames of less than or equal to 1 year. Comparatively, this represents a very small sampling of all the objectives that NASA has listed in its 2014 Strategic Plan and its 2015 Performance and Management Plan, including 15 strategic objectives and hundreds of medium-term and short-term performance objectives. For an undertaking of this size, the volume of information to be collected, encoded, and integrated is challenging, to say the least.

4.8.2 Propagation of Templates

The development and population of templates for a full-scope EROM effort involves collaborative participation of all the organizational units in the enterprise, from the bottom to the top. Each unit should have its own objectives hierarchy, individual risks, individual opportunities, and leading indicators. Each should complete the templates described in this book in a consistent manner, as shown in Figure 4.15, and pass the completed templates on to the next level of authority in the organization. Each higher level in the organization should utilize the templates provided by the organizations subordinate to it to create its own set of templates. Cross-organizational communication (vertically and horizontally) should freely occur during this process so that the templates produced by any one organizational unit are complete and consistent with those produced by other interfacing organizational units.

Obviously, this process requires that all organizational units use the same format in preparing their templates. Using the same format ensures that the information gleaned from the analysis can be passed readily up the chain of authority. It does not mean, however, that the information provided in the templates should be constrained to only risks and opportunities that are already recognized by others. Rather, each organizational unit should be creative in determining the risks and opportunities that influence the likelihood of success in meeting its objectives.

After receiving templates from its subordinate organizational units, the organizational unit at the next level up should determine whether the completed templates it receives are consistent in terms of assumptions made, the interpretation of input information, recognition of important interfaces, and overall conclusions. If there are inconsistencies or misunderstandings in these areas, then the higher-level unit has the obligation of determining why that is the case and how to resolve them.

The highest level of analysis in the EROM analysis scheme is the enterprise-wide level, as illustrated in Figure 4.15. The templates at enterprise-wide level represent a compilation of the templates from all the entities in the organization. Any differences between organizational units in terms of assumptions, interfaces, interpretations, and conclusions should be resolved at the enterprise-wide level. Once completed, the enterprise-wide level templates are passed back down the organizational chain of authority so that the individual organizational units can review them and provide their assent or dissent.

4.8.3 Development of an Integrated EROM Database

The templates prepared by the various units become part of an integrated database. Databases of risks, opportunities, and leading indicators that exist at each unit in the organization are integrated upward first to program directorate, center, and executive levels, and then into an enterprise-wide integrated database. At the extended enterprise level, the integrated database should typically include the information in the risk and opportunity templates, the owner for each risk and opportunity, the organizational entities that are involved, corresponding working groups and management boards, change plans, change history, and status. The development of an enterprise-wide database helps to ensure that cross-cutting risks and opportunities are correctly identified, consistently analyzed, and treated in an integrated manner. It also helps to facilitate the process of rolling up individual risk and opportunity scenarios from lower levels to an aggregate view of the organization's overall likelihood of success in meeting its strategic objectives.

A later section will discuss the challenges of creating and managing a database that integrates risk and opportunity information vertically and horizontally across an extended enterprise consisting of both internal organizational entities and external partners. It will also discuss some best practices for doing so.⁷