CHAPTER 11
Scenario Analysis

Scenario analysis is a challenging element in the operational risk framework. Scenario analysis provides the operational risk framework with a tool to explore the rare but plausible losses that could arise as a result of operational risk. In this chapter, we discuss the various methods used for scenario analysis and explain the important elements of a robust scenario analysis program.

ROLE OF SCENARIO ANALYSIS

Scenario analysis has become an important element in operational risk management and measurement, and the methods used have evolved rapidly over the past few years. Firms use scenario analysis to evaluate their exposure to high-severity events. Unlike RCSA analysis, scenario analysis focuses on “fat-tail” events, or rare catastrophic events. These types of events can put the firm at serious risk. For this reason, scenario analysis was a required element in calculating operational risk capital requirements under Basel II for any firm undertaking the Advanced Measurement Approach (AMA).

Firms that do not have AMA requirements are also pursuing scenario analysis programs, as they provide a valuable insight into the major risks faced and provide the opportunity for an engaging dialogue with the business lines.

The role of scenario analysis in the operational risk framework is illustrated in Figure 11.1.

Scenario analysis is used to derive reasoned assessments of plausible severe losses. The assessments are then used to explore “what-if” cases that may be beyond the current experience of the firm. External data plays a key role in scenario analysis, as it provides insight into what has already occurred in other firms. However, in addition to learning from experiences outside the firm, scenario analysis considers events that might not yet have occurred at any firm.

Schematic illustration of Scenario Analysis in the Operational Risk Framework

FIGURE 11.1 Scenario Analysis in the Operational Risk Framework

A somewhat helpful definition of scenario analysis and its uses can be found in Basel II. However, areas of ambiguity that have proven challenging to the industry are highlighted as follows:

A bank must use scenario analysis of expert opinion in conjunction with external data to evaluate its exposure to high-severity events. This approach draws on the knowledge of experienced business managers and risk management experts to derive reasoned assessments of plausible severe losses. For instance, these expert assessments could be expressed as parameters of an assumed statistical loss distribution. In addition, scenario analysis should be used to assess the impact of deviations from the correlation assumptions embedded in the bank's operational risk measurement framework, in particular, to evaluate potential losses arising from multiple simultaneous operational risk loss events. Over time, such assessments need to be validated and re-assessed through comparison to actual loss experience to ensure their reasonableness.1 [emphasis added]

Finding a process that taps experienced business managers and risk managers and that produces reasoned assessments of plausible losses is challenging indeed. Who is “experienced”? What constitutes a “reasoned” assessment? What do we mean by “plausible”?

In March 2021, the Bank of International Settlements (BIS) summarized the current state of scenario analysis in the financial services sector as follows:

Scenario analysis – Scenario analysis is a method to identify, analyse and measure a range of scenarios, including low probability and high severity events, some of which could result in severe operational risk losses. Scenario analysis typically involves workshop meetings of subject matter experts including senior management, business management and senior operational risk staff and other functional areas such as compliance, human resources and IT risk management, to develop and analyse the drivers and range of consequences of potential events. Inputs to the scenario analysis would typically include relevant internal and external loss data, information from self-assessments, the control monitoring and assurance framework, forward-looking metrics, root-cause analyses and the process framework, where used. The scenario analysis process could be used to develop a range of consequences of potential events, including impact assessments for risk management purposes, supplementing other tools based on historical data or current risk assessments. It could also be integrated with disaster recovery and business continuity plans, for use within testing of operational resilience. Given the subjectivity of the scenario process, a robust governance framework and independent review are important to ensure the integrity and consistency of the process.2

SCENARIO ANALYSIS APPROACHES

There are several different methods that can be used to conduct scenario analysis. Some firms use a workshop approach; some conduct interviews or analyze data in small teams. Some firms conduct many scenario analysis workshops, covering each risk category in each business; some run only a few scenarios at the corporate level. Some firms have standard scenarios for every business line to consider; some prefer that each business line develop their own.

Whatever the approach is, the purpose of the scenario analysis program is to identify those rare, but plausible, large events that should be incorporated into the operational risk framework. In practice, this means that people will be asked extremely difficult questions such as, “How big could such an event be?” or “Could it happen in the next 20 years?”

If the output of scenario analysis is to be used directly in the capital calculation, then it will need to be a particularly robust, repeatable, and well-documented activity. Operational risk capital under an AMA framework is supposed to capture the risk at a 99.9 percent confidence level. In other words, it should be sufficient to cope with a 1-in-1,000-year event.

Conversations with business managers on whether something could happen in 1,000 years has proved unfruitful, and so most firms have developed ways to get close to the very rare by considering the rare. For example, a 1-in-10-year event might be easier to discuss, and several data points might be collected to allow for the data collected to be extrapolated out to the rarer event.

The Basel Committee recognized the challenges banks were facing with this element of the framework and provided some further guidance in their 2011 AMA Guidelines as follows:

Scenario data provides a forward-looking view of potential operational risk exposures. A robust governance framework surrounding the scenario process is essential to ensure the integrity and consistency of the estimates produced. Supervisors will generally observe the following elements in an established scenario framework:

  1. A clearly defined and repeatable process;
  2. Good quality background preparation of the participants in the scenario generation process;
  3. Qualified and experienced facilitators with consistency in the facilitation process;
  4. The appropriate representatives of the business, subject matter experts and the corporate operational risk management function as participants involved in the process;
  5. A structured process for the selection of data used in developing scenario estimates;
  6. High quality documentation which provides clear reasoning and evidence supporting the scenario output;
  7. A robust independent challenge process and oversight by the corporate operational risk management function to ensure the appropriateness of scenario estimates;
  8. A process that is responsive to changes in both the internal and external environment; and
  9. Mechanisms for mitigating biases inherent in scenario processes. Such biases include anchoring, availability and motivational biases.3

We consider each of these aspects as we explore the variety of methods being used today to meet the challenges of scenario analysis.

The use of scenario analysis in capital calculation has proven so difficult that the latest capital calculation guidance from BIS has removed it as an element for the new approaches that are scheduled to be implemented in coming years. However, until those new approaches are implemented, scenario analysis remains a contributor to the AMA approach and continues to be an effective methodology for operational risk management beyond capital calculation.

(a) A Clearly Defined and Repeatable Process

Scenario analysis contents might vary considerably from one set to another, but the process needs to be consistent. To achieve this it is necessary to develop written procedures and standards that will be applied every time a scenario analysis activity is run.

Experience has shown many firms that their auditors and regulators will pore over these documents and will carefully compare them to the process that actually occurred. It is therefore important to ensure that the defined process is not aspirational but is achievable over and over again.

A robust scenario analysis process does not need to be, and should not be, overly complex. Rather, it should meet the criteria outlined earlier, while also providing the maximum benefit and least disruption to the businesses that are involved.

For this reason, much of the scenario analysis process is likely to reside in the corporate operational risk function, in the form of preparation, facilitation, and post-scenario documentation and validation.

(b) Background Preparation

Section b of the AMA Guidelines calls for “good quality background preparation of the participants in the scenario generation process.”4

Interviews

Preparation for scenario analysis is very similar to preparation for RCSA workshops and questionnaires. The facilitator or preparation team interviews the key business managers and support managers for the area under consideration. Background documentation from audits, compliance reviews, and Sarbanes-Oxley assessments is reviewed. Internal and external loss events are analyzed.

Internal Loss Data

The internal loss data of a firm certainly provides a floor for losses, but it does not show what could go wrong; it only shows what has gone wrong. The facilitators of a scenario analysis discussion should be aware of the history of losses, but it should not be shared directly with those participating in the discussion as this introduces a hard-to-overcome anchoring bias, as discussed later.

External Loss Data

One of the most important inputs into the scenario analysis process is external loss data.

For example, if a scenario analysis workshop is being conducted on the risk category Internal Fraud, then the firm might have some internal data, but often very little. However, internal fraud as a category includes unauthorized trading, and the industry has several egregious examples of unauthorized trading that have resulted in losses in the many billions of dollars. Information on these external events can be helpful in developing a what-if scenario for the firm.

In scenario analysis, the questions should not be focused so much on why that event could not happen at this firm (as most businesses will contend), but rather on how such an event could happen at this firm. How many controls would have to fail at once? What sort of positions would the trader have to be able to hold? And so on.

External events provide an excellent opportunity to stimulate discussion on the rare, but plausible, risks in this category.

In addition to the storylines from the news, external data from a consortium such as ORX can provide a helpful benchmarking floor. For example, if your firm is a member of ORX and the ORX data show that in the industry firms of your size have experienced losses over $50 million on average once every five years in this risk category, then is there any reason why your firm is different?

RCSA Results

Another valuable source of background information is the RCSA program. RCSAs will have identified the high risks in each area and can be used to help populate a straw man of possible scenarios for consideration. However, something that is low risk in the RCSA might still qualify as a scenario, as it may be that frequency was the main driver that was keeping the risk low. If something could generate a very large loss, regardless of frequency, then that is an item for consideration in scenario analysis. Therefore, RCSA results need to be carefully reviewed as part of the background preparation.

Scenario analysis should also feed back into the RCSA program, further enriching the risks library that is constantly evolving in the operational risk framework.

Compliance and Audit Findings

Compliance and audit findings can be helpful in challenging claims that a control or a set of controls is working well. These should be carefully reviewed as part of the background preparation and should be on hand for the facilitator to refer to as needed.

Key Metrics and Analysis

Some risk categories may lend themselves to preparatory statistical analysis. For example, when discussing scenarios regarding the risk category Damage to Physical Assets, a scenario might be raised concerning a terrorist attack destroying a building. There are sources of data available on the frequency of attacks globally and in large cities and on the range in impact zone from a single attack. This data can be used alongside the firm's own data on its office locations to develop a model to assist with the estimation of severity and frequency.

The use of such metrics is referred to as factor analysis by some firms and is gaining momentum across the industry. This type of analysis alleviates the difficulties in estimation and seems to be well received by regulators so far. However, according to the AMA guidelines discussed previously, the role of the business expert must still exist and so even this type of analysis requires subjective confirmation from the business and risk managers.

Straw Man Scenario List

Based on research in all these elements, a list of possible scenarios can be brought to the participants for their consideration, or a list of scenarios can be determined for an interview based process.

Participants in scenario analysis activities are better equipped to consider scenarios if they are provided with appropriate background resources.

(c) Qualified and Experienced Facilitators with Consistency in the Facilitation Process

The AMA Guidelines call for “qualified and experienced facilitators with consistency in the facilitation process.”5

If the scenarios are being discussed in a group environment, such as a workshop, then there needs to be a neutral facilitator who not only knows the process completely but is also proficient at managing the conversations to ensure that no one person, or small group, is dominating the discussion and that all ideas are heard.

The skills needed often mean that scenario analysis workshops can be run only one at a time as the facilitation resources are in short supply.

(d) The Appropriate Representatives

The AMA Guidelines call for the involvement of all of “the appropriate representatives of the business, subject matter experts and the corporate operational risk management function as participants involved in the process.”6

The written procedures for scenario analysis should probably include a list of the required quorum. If the firm has a scenario analysis process that requires each business line to complete a scenario analysis workshop for each risk category, then each category may have a different quorum. For example, Employment Practice and Workplace Safety would require a representative from the human resources department.

Most scenarios benefit from attendance by representatives from the legal department, compliance, operations, and technology. Some may also benefit from representation from the finance department. The quorum requirements should be set appropriately.

If the quorum is not met, then it may be necessary to cancel and reschedule the workshop, or it might be possible to loop the missing participants into the review process afterward.

(e) A Structured Process for the Selection of Data

The AMA Guidelines call for “a structured process for the selection of data used in developing scenario estimates.”7

At the heart of scenario analysis activity is the gathering of data to be used to develop the scenario analysis estimates. In a workshop environment, these data include all background preparation data and the estimates that are solicited from the participants during the workshop. While the workshop environment may be a free-flowing conversation, there need to be checkpoints incorporated into the process to ensure that all procedural requirements are being met. For example, a workshop might be designed to gather a worst-case dollar amount for each scenario. If so, there needs to be a defined process by which the worst-case estimates are gathered from the participants in the room and their final consensus reached.

In an interview-based approach, the same challenges exist in ensuring that the way responses are gathered is carefully structured so that the process can be clearly documented and is repeatable.

To meet this requirement, firms have adopted questionnaires and templates that assist the facilitators in keeping the process in line and ensuring that the data are clearly gathered and documented.

Once the data have been gathered, through background preparation and through expert discussion and debate, they can then be used to draw conclusions on the possible severity and frequency for each scenario.

Some firms collect data at the risk category level rather than at the scenario level. For example, there may be five scenarios that have been identified in the Clients, Products, and Business Practices risk category. Some firms gather severity and frequency information for all five, and some firms gather severity and frequency for the group of five (e.g., how many of these scenarios could happen in the next 10 years in total?).

Conclusions drawn and decisions made need to be clearly documented as discussed next.

(f) High-Quality Documentation Which Provides Clear Reasoning and Evidence Supporting the Scenario Output

The AMA Guidelines require “high-quality documentation which provides clear reasoning and evidence supporting the scenario output.”8

In the early days of operational risk scenario analysis, there was a reluctance to document the discussions. Sensitive issues are often raised, and there may be disagreements during the discussions before consensus is reached. The idea of documenting all those details left most firms feeling uncomfortable and their legal departments feeling anxious.

However, in the past few years, the regulatory pressure to ensure that all conclusions are supported by documented reasoning and evidence has led to a more highly documented process despite these concerns.

While the whole conversation does not need to be recorded, there does need to be a well-documented summary at the end of the process that outlines the thought processes, the data and evidence that was weighed and considered, and the reason that consensus was reached on certain conclusions such as severity and frequency.

It is hard for a facilitator to both facilitate the process and document what happens. For this reason, in workshop scenario analysis activities there is often a second neutral participant, perhaps from the corporate operational risk function, whose sole role is to document the proceedings. This is not a court reporter–type activity, but requires a deep understanding of the process and procedures to ensure that all important aspects are captured in the documentation.

It is difficult to go back afterward to look for consensus on something that was missed, and a robust documentation template can assist with ensuring that all important data points and rationales have been captured.

(g) Independent Challenge and Oversight

The AMA Guidelines call for a “robust independent challenge process and oversight by the corporate operational risk management function to ensure the appropriateness of scenario estimates.”9

In a workshop, if the facilitator is provided by the corporate operational risk function, then they can take on the dual role of challenge also. If a third-party facilitator is used, then the corporate operational risk function can be a participant in the workshop and challenge as a member of the quorum.

In all types of scenario analysis, the corporate operational risk function can meet this challenge and oversight requirement by being actively involved in all preparation work, in the scenario analysis activities, and in the review of the documentation.

It is also helpful to establish a formal challenge and review process after the activity. This can consist of a simple e-mail documentation review by all participants or by a follow-up meeting to walk through the final documented conclusions.

(h) A Process That Is Responsive to Changes

The AMA Guidelines require “a process that is responsive to changes in both the internal and external environment.”10

A scenario analysis activity should capture the current state of the business and control environments and should be designed to ensure that any changes in those environments will trigger a new activity as appropriate.

Many firms revisited their Internal Fraud scenario analysis after the 2012 UBS unauthorized trading event, and external events are helpful triggers for such reassessments. It is also important to revisit scenario analysis when a major business change occurs, such as an acquisition or divestiture. Similarly, a major control change such as a technology infrastructure rollout may trigger a new scenario analysis in impacted business and risk categories.

Regardless of triggers, scenario analysis should be conducted on a timely basis to ensure that it remains up-to-date as regards the current internal and external environment. For this reason, many firms will require their scenario analysis to be updated once a year even if no trigger has arisen.

However, the resource challenge can prove overwhelming and less frequent updates might be practically necessary.

(i) Mechanisms for Mitigating Biases

The AMA Guidelines draw attention to the need for “mechanisms for mitigating biases inherent in scenario processes. Such biases include anchoring, availability and motivational biases.”11

In all methods there are biases that enter the process and that require careful consideration. While an expert may be knowledgeable on the subject matter of the scenario under discussion, they might not have the statistical background to understand the implications of certain estimates and decisions regarding impact and frequency of events. They are also likely to be untrained in the biases that can arise in such exercises and how to compensate for them.

Therefore, it is important to ensure that scenario analysis workshops and interviews are facilitated by someone who does have that experience or, at the very least, has an appreciation for the dangers of statistical and behavioral bias in the process.

Where possible, the process should avoid the introduction of biases when providing background or supporting data.

The Australian Prudential Regulation Authority produced a working paper in 2007 that addressed the inherent biases that occur in scenario analysis for operational risk and identified two classes of bias: judgmental and motivational.12 This paper has stood the test of time and still provides strong guidance on how to address bias in scenario analysis today.

Judgmental bias occurs during the estimation process as the experts are swayed by the background data and the form of the questions. An example of judgmental bias is availability bias, which occurs when estimates are influenced by the availability of data. For example, past operational risk event data may be supplied to scenario analysis participants in the form of internal and external loss event data.

This data can influence the perception of likely size and frequency of events, and indeed the type of events that can occur. If an expert has recently experienced a particular event, they are more likely to deduce that that event can occur with a higher frequency and with a similar impact. For example, someone who has recently been in a car accident is likely to estimate the frequency of car accidents as higher than someone who has not.

Similarly, if the firm or the industry has recently experienced a large event, the scenario analysis participants are more likely to estimate that that event could occur again soon, and at the same impact level.

Another example of judgmental bias is anchoring. Anchoring occurs where participants are offered an initial estimate from which to base their estimate. For example, internal and external data may anchor the estimates so that likely impacts beyond that size are considered unlikely, and frequencies that differ from the past are discounted as less plausible.

Scenario analysis should provide an opportunity to look forward and consider what could occur in the future, and not only what has already occurred in the past. Therefore, judgmental bias can seriously undermine the process if not carefully considered. For this reason, it may be best not to provide internal loss data and to only use it as a floor. The facilitator can have access to this data and refer to it if the scenario participants are estimating close to, or below, that floor.

The careful use of internal and external data, and facilitation by the operational risk department, can help to overcome these biases. By addressing these biases up front, the participants can be assisted in resisting them and keeping their estimation processes less constrained to the judgmental influences.

Motivational bias occurs where the estimates of the participants are influenced not by the data presented, but by the personal interest of the participants themselves. More crudely, this can be referred to as “gaming the system.” Senior management may be particularly susceptible to this bias, as they may perceive an estimate that suggests a potentially high impact as reflecting poorly on their department's risk management practices.

In addition, if scenario analysis is used as an input into a capital calculation for operational risk capital, then participants will be aware that high estimates may result in high capital and so may resist estimating the fat-tail events effectively.

Overcoming motivational bias is more challenging than overcoming judgmental bias. One way to avoid gaming of the scenario analysis estimates is to ensure that allocation of capital is driven not only by scenario analysis but also by RCSA, KRI, and loss data results. Alternatively, scenario analysis can be done at the top of the house, rather than at the business unit level, and then allocated down to business lines using a combination of operational risk information.

The facilitator of the scenario analysis workshop might also set parameters for the estimates that preclude underestimating. For example, they might set minimum limits at past event levels if they are in fact larger or more frequent than the estimates.

SCENARIO ANALYSIS OUTPUT

Different methods produce different outputs, but the goal of scenario analysis is to produce reasoned assessments of plausible severe losses, and so outputs need to support that goal.

Some scenario analysis methods produce an average loss estimate, a worst-case loss estimate, and frequency estimates for each of these values. Some produce just the worst-case estimate and a single frequency estimate. Others produce a range of loss estimates, with frequency estimates for each loss. Still others produce the latter range plus a maximum loss estimate.

One example of possible scenario analysis output is illustrated in Table 11.1. In this table, the firm has taken the approach of collecting the number of events that might occur in a category rather than the number of times a single scenario might occur. They are collecting a range of frequencies for each risk category in a selection of severity ranges.

For example, in the Clients, Products, and Business Practices category they have decided upon all the scenarios that apply and are now estimating how many of those scenarios could occur in total.

In the $1 million to $5 million bucket (A), they have agreed that it is plausible that they could experience five events in this category. Hence, they have entered a frequency of five. However, in the greater than $100 million bucket (B), they have agreed that such a large event could occur only once every 10 years. Hence, they have entered a frequency of 0.1.

The total frequency (C) represents how many events could occur in this category in a single year and is simply the sum of the buckets.

The final column (D) contains a maximum loss amount that has been agreed in the scenario analysis workshops.

Some categories do not have any entries (E), as the group has determined that in fact no event could occur at that size. Of course, such an estimation process as represented in Table 11.1 would have to have be supported by robust procedures, supporting evidence, and a well-documented rationale.

The output drives how the scenario analysis information is then used for risk management or for capital calculation purposes, and the model that is applied to calculating capital for the firm. This capital model may have many other elements, and capital calculation methods are considered further in Chapter 12.

While designed to produce fat-tail estimates, scenario analysis is often also responsible for the identification of significant mitigation activities that should be undertaken in order to lessen the risks identified.

This can mean that some overlap occurs between the RCSA program and scenario analysis, particularly if the workshop RCSA method is being used. Indeed, some firms have combined the two elements of the operational risk framework, and at the end of an RCSA workshop they will ask the participants to consider the same risks in an environment where all controls fail. In this way participants can extrapolate from known and relatively well-controlled risks to extreme but plausible fat-tail events.

TABLE 11.1 Sample Scenario Analysis Output

Frequency/Severity BucketsTotal Annual FrequencyMax Single Loss
Risk Category$1 to $5m$5 to $10m$10 to $20m$20 to $50m$50 to $100m> $100m
Clients, Products, and Business Practices5.0(A)3.01.00.50.20.1(B)9.8(C)$600m(D)
Execution, Delivery, and Process Management10.05.02.00.50.20.117.8$150m
External Fraud1.00.50.20.1––1.8$45m
Internal Fraud1.00.50.10.10.10.11.9$1,000m
Damage to Physical Assets3.01.01.00.50.20.15.8$100m
Employee Practices and Workplace Safety5.03.02.01.00.5–22.5$75m
Business Disruption and Systems Failures6.04.02.01.0– (E)–13$40M

Most operational risks that have a high impact occur as a result of multiple control failings, and the RCSA process can help with the thought processes behind imagining such events. The risk is identified in an RCSA. The controls are scored for effectiveness, and the residual risk assessed. Then the same risk is considered in a situation where all controls fail in order to envisage the fat-tail event.

KEY POINTS

  • Firms use scenario analysis to evaluate their exposure to high-severity events by deriving reasoned assessments of plausible severe losses.
  • There are several different methods for scenario analysis, including workshops and interviews. A robust scenario analysis process includes:
    • A clearly defined and repeatable process
    • Good-quality background preparation
    • Qualified and experienced facilitators
    • The appropriate quorum of participants
    • A structured process for the selection of data
    • High-quality documentation
    • A robust independent challenge process
    • A process that is responsive to change
    • Bias minimization
  • The output from scenario analysis can be used as an input into capital calculations and to inform the firm of potentially catastrophic operational risk losses.

REVIEW QUESTIONS

  1. The Basel II definition of scenario analysis requires which of the following elements as part of the process?
    1. Knowledge of experienced business managers
    2. Knowledge of experienced risk management experts
    3. Knowledge of external independent advisers
    4. Reasoned assessments of plausible severe losses
    1. I, II, and III
    2. I and II only
    3. I, II, and IV only
    4. All of the above
  2. During a scenario analysis workshop, a senior manager becomes concerned that an honest but high estimate of plausible losses will reflect badly on her management skills. How might this significantly impact the results? Select the best answer.
    1. The results may reflect a motivational bias.
    2. The results may reflect a judgmental bias.
    3. The results will be unaffected.
    4. The results will reflect the true opinion of the senior manager.

NOTES

  1. 1 Bank of International Settlements, Basel Committee on Banking Supervision, “International Convergence of Capital Measurement and Capital Standards: A Revised Framework,” Comprehensive Version, 2006, section 675.
  2. 2 Bank of International Settlements, Basel Committee on Banking Supervision, “Revision to the Principles for Sound Management of Operational Risk,” March 2021, section 35(f), https://www.bis.org/bcbs/publ/d515.htm.
  3. 3 Basel Committee on Banking Supervision, “Operational Risk—Supervisory Guidelines for the Advanced Measurement Approaches,” June 2011, www.bis.org/publ/bcbs196.pdf, section 254.
  4. 4 Ibid., (b).
  5. 5 Ibid., (c).
  6. 6 Ibid., (d).
  7. 7 Ibid., (e).
  8. 8 Ibid., (f).
  9. 9 Ibid., (g).
  10. 10 Ibid., (h).
  11. 11 Ibid., (i).
  12. 12 Emily Watchorn, “Applying a Structured Approach to Operational Risk Scenario Analysis in Australia,” APRA Working Paper, September 2007.
..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset