INTRODUCTION

ERM is an organization-wide effort that requires significant time and resources in order to develop the requisite talent, policies, processes, and systems. The key question for board members, corporate executives, and regulators is this:

How do we know if the ERM program is working effectively?

The purpose of this chapter is to answer that question. The key lies in establishing an objective performance feedback loop for ERM. The feedback loop is essential for starting a new ERM program or enhancing an existing one. Based on my work in ERM, I strongly believe that this is a critical missing link to which many companies do not pay sufficient attention.

In the last few chapters, we have discussed ways in which companies can measure risk, evaluate performance, and track where they stand in relation to strategic objectives. While risk policies articulate processes and requirements for ERM, the board and management still need information and feedback in order to ensure that ERM programs not only remain on track, but continue to evolve and improve. The solution to these issues lies in the assurance processes established by the organization, which include monitoring and reporting to the board, independent assessments, and objective feedback loops.

This chapter will discuss how feedback loops permit effective evaluation of risk management performance,¹ provide critical risk information to boards and senior management, and offer actionable data to capture error and improve processes. We'll look at how feedback loops work in general and how they can be used to measure and improve performance in ERM.

WHAT IS A FEEDBACK LOOP?

At its most basic, a feedback loop is a system that uses the outputs from one action as inputs to the next, eventually creating a continuous loop of inputs and outputs. A performance feedback loop, therefore, is a critical concept that supports self-correction and continuous improvement by adjusting a process according to the variances between actual and desired performance. Such feedback loops can be used in the context of business for measuring effectiveness of certain efforts against goals, then refining processes based on the resulting feedback.

The goal of a feedback loop in a risk context is to determine if risk management is working effectively, and if not, to provide a route toward improvement. The feedback loop consists of three main steps: establishing business and risk objectives, carrying out the ERM program, and reviewing the program's results. First, an organization must determine its business and risk objectives, which will dictate the structure of its ERM program. When a predetermined feedback period ends, the organization assesses the results to determine which parts of the program were successful and which need improvement. This analysis in turn informs the objectives for a revised or augmented ERM program, and thus the cycle begins again. Figure 19.1 provides an illustration of an ERM performance feedback loop. If the business and risk results are not consistent with the objectives, something has to change: people, incentives, processes or systems—or all of them.

EXAMPLES OF FEEDBACK LOOPS

Feedback loops are not unique to business organizations. In fact, they form the core of the scientific method and empirical inquiry. Successful research requires the ability to gather and synthesize data to refine, reformulate, or reject a hypothesis in the development of general scientific theories.

Even the human body uses natural feedback loops to maintain homeostasis. After an increase in activity, the brain sends signals to the heart to help stabilize internal temperature. Another feedback loop incorporating the senses of sight and touch and muscle movement informs us of where we are in the physical world and how to maneuver our way through it. Feedback keeps us upright and allows us to manipulate objects.

Perhaps most relevant to our discussion are the feedback loops that guide monetary policy: The Federal Reserve uses them to identify policies intended to keep the economic measures of unemployment, inflation, and GDP growth within acceptable, sustainable parameters. The Fed's primary lever is setting interest rates, but it can take more dramatic forms such as the quantitative easing policies implemented in response to the 2008 financial crisis.

Feedback loops have also become common in computer programming, manufacturing, and other fields, particularly in the use of iterative development structures such as scrum and lean manufacturing. In these processes, large projects are broken down into smaller practical units that can be tested and corrected at each stage of the operation. More recently, the use of feedback loops has gained traction in hedge fund management, health-care interventions, and the effective altruism movement.

Bridgewater is one of the largest and most successful hedge funds in the world. The founder, Ray Dalio, argues for the use of a performance feedback loop to monitor and shape organizational effectiveness.² Akin to the basic feedback loop described in the previous section, Dalio's model has three main stages: goals, the “machine,” and outcomes. Dalio likens the organization to a machine fueled by objectives, or goals, which in turn produces certain results, or outcomes. The machine has two major components: culture and people. If the outcomes do not match up with the goals, this indicates the machine is not functioning properly, and by that same logic the culture or the people are not working as they should. The last stage of the process is to determine which part of the organization is defective, and to suggest improvements. In order to ensure that a feedback loop is effective, it must cycle through numerous iterations. This establishes a large sample size and ensures that outcomes are accurate and not a result of human error. By conducting the performance feedback loop model this way, the “machine” will continue to develop and follow “a steep upward trajectory.”

We often face far more difficult questions than whether our employees and culture comprise a well-oiled machine. How do you value life? And by extension, how do you compare life-saving interventions? One metric is the quality-adjusted life year, or QALY, which was developed to compare competing health programs. The QALY is calculated using two variables: time and quality. Time refers to the additional number of years a particular program could extend an individual's life. Quality is how an individual would rate the quality of his or her health, as a percentage of “full health.” Multiplying time and quality returns a value for QALY. For example, if a program were to extend the life of a person at 70 percent health by 10 years, its QALY would be equal to 7. By testing different healthcare outcomes and comparing which programs maximize QALYs most cost-effectively, we can identify efficient interventions in a field plagued by finite resources and the constant need for triage.

William MacAskill, the cofounder of effective altruism, a new, evidence-based approach to charitable giving and social impact, expands this use of objective feedback loops to determine the effectiveness of altruistic pursuits in general. In his book, Doing Good Better, he provides an example of a program executed without a proper feedback loop.³ In the 1990s, Trevor Field, a middle-aged South African man, came across a business opportunity he could not pass up: a water pump designed as a merry-go-round allowing water to be pumped out of a storage tank while children played. This invention seemed ingenious. Instead of requiring women to walk miles and miles to find a water source, they could have water available on demand nearby, and their children would be doing most of the work. In 1995, Field installed his first water pump, dubbed the PlayPump, and received sponsorships and donations from various organizations including a prestigious award from the World Bank.

However, what Field overlooked was the last stage in the feedback loop, analyzing the results of the program. Yes, his objectives had been fulfilled: He had built water pumps in the hopes of improving and increasing water access. But were these good intentions translating into the desired outcomes? Definitely not, as it turns out. Children were exhausting themselves pushing the merry-go-round, which required constant force. Some even vomited or suffered broken limbs. Thus, their mothers were forced to take on this tiring and demeaning task. Also, the old water pump system was easier to handle and provided five times the amount of water provided by the PlayPump.

MacAskill also describes a successfully implemented feedback loop. An MIT development economist, Michael Kremer, became involved with a Dutch charity program in Kenya that was trying to improve school attendance and test scores. The program provided schools with new textbooks, more teachers, and free uniforms. Before expanding the program to several additional schools, Kremer suggested testing it using a randomized controlled trial: The researchers compared seven schools that had been given the additional resources to seven schools without them. In the end, the program had no discernible effect on student performance and attendance.

Kremer decided to test other programs one by one using the same method, thus creating a feedback loop: At the conclusion of each trial, he measured outcomes against the program's goals, and adjusting the program accordingly. He eventually came up with an idea that worked: deworming. By giving children a simple, inexpensive pill to remove parasites from their intestines, their health improved and thereby increased school attendance and performance. Following up with the students ten years later, Kremer found that on average they worked more hours, and their incomes were 20 percent higher than those of their peers who had not been dewormed.

The effective altruism movement has even leveraged this approach to evaluate and improve the charity industry at large. Its flagship organization, GiveWell, uses data from double-blind, randomized control trials and other metrics to rate and rank the cost-effectiveness of different charities. This completes a badly needed feedback loop in a sector that is rarely held accountable for concrete results. It should come as no surprise that the co-founders of GiveWell both came from Bridgewater!

As the above examples illustrate, a performance feedback loop is a highly worthwhile tool for evaluating and improving any process. They also underscore an important lesson that applies to most disciplines: Ex ante judgments and intuitions only go so far. In order to reach one's goals in a timely and efficient manner, it is essential to incorporate ex post information into the decision-making apparatus. Risk management is no exception!

ERM PERFORMANCE FEEDBACK LOOP

In order to establish a performance feedback loop for ERM, companies must first define its objective in measurable terms. As I've mentioned previously, a prime objective of ERM is minimizing unexpected earnings volatility. It is important to note that the goal is not to minimize absolute levels of risks or earnings volatility, but just that from unknown sources. Once we define the objective, we can create the feedback loop.

Based on this defined objective for ERM, Figure 19.2 provides an example of using earnings volatility analysis as the basis of a feedback loop. At the beginning of the reporting period, the company performs ex ante earnings-at-risk analysis and identifies five key earnings drivers—business plan execution, interest rates, oil price, key initiatives, and expense control—that may result in a $1 loss per share, compared to an expected $3 earnings per share.

At the end of the reporting period, the company performs ex post earnings attribution analysis and determines the actual earnings drivers. The combination of these analyses provides an objective feedback loop on risk management performance. In this example, three of the actual earnings drivers were identified in the beginning of the period. But $0.40 of the variance resulted from unforeseen factors (e.g., operational risk loss).

Over time, the organization strives to minimize the earnings impact of unforeseen factors. Consider two extremes: (1) with no risk management the entire $2.00 would come from unforeseen factors since the company is completely in the dark, and (2) with perfect risk management the contribution from unforeseen factors would be zero since the company would have perfect foresight. Of course, no ERM program is perfect but the feedback loop enables management to make continuous improvements to drive unexpected earnings variance to a minimum. Management can also address three key questions with respect to the ERM program:

1. Did we identify the key risk factors? If the RCSA process was effective, we would have identified the key risks that can impact business performance. If there is a material risk or loss that was unforeseen, we need to review and improve our RCSA process.
2. Were our EPS sensitivity analyses accurate? Even with effective risk identification and assessment, our risk analytics and quantification models need to accurately measure their earnings sensitivities. If actual earnings variance (negative or positive) resulted from model risk, then we need to examine the data, assumptions, and formulas used in the risk models.
3. Did risk management impact our risk/return positively? Risk management is also about creating opportunities and adding value. Did the risk team work effectively with corporate and business management to enhance our risk profile, such as risk-based pricing or resource allocation?

Using feedback loops, a company can measure the efficiency of its ERM program both qualitatively and quantitatively. Qualitative methods include achievement of key milestones as well as records of policy violations, and root-cause analyses of material losses or other unexpected events. Quantitative methods might include tabulations of data such as ERM Scorecards (explained in greater detail below) that measure performance relative to expectations and calculate the gap between actual and expected results. These permit a more focused effort to improve underlying processes and minimize unexpected earnings variance. In addition to earnings, the same feedback loop can be created for other performance metrics, such as cash flows and enterprise value. Not only do these feedback loops measure responses to key risk issues, but when taken cumulatively, they can gauge the effectiveness of the ERM program as a whole.

MEASURING SUCCESS WITH THE ERM SCORECARD

We have already discussed the most important outcome of feedback loops: continuous improvement of ERM. Now we can delve into a secondary outcome: measuring success. Just as we rely on negative feedback (unexpected earnings variance) to recognize what needs improvement in the risk management process, we also need positive feedback to recognize what is going well. Both help to gauge the effectiveness of an ERM program.