CHAPTER 10

Correlation Causes Questions: Environmental Consistency Confidence in Wholesale Financial Institutions

Michael Mainelli

Z/Yen Group Limited

Introduction

Wholesale financial institutions have tried a number of approaches for managing and modeling operational risk, with limited success. Z/Yen Group Limited, for example, has developed an approach called environmental consistency confidence (ECC) meaning, basically, if you can predict incidents and losses with some degree of confidence, then your modeling is useful. It is often said, “correlation doesn’t demonstrate causation.” That is true, but “correlation should cause questions.” The core of environmental consistency confidence is using modern statistical models to manage financial institutions through the examination of correlations between activity and outcomes. This chapter sets out how environmental consistency confidence works, how it differs slightly from other approaches, its basis in system dynamics, its fundamental concept of predictive key risk indicators for losses and incidents (PKRI⇔LI) and two early trials showing promising results.

Wholesale financial institutions have tried applying system dynamics and modeling techniques from at least the 1970s with minimal returns. Investment banks have modeled trading floors in order to see how to optimize trade flows; large payments processors have tried modeling their multipath networks in order to optimize processing time and security. Further, Basel II initiatives led numerous wholesale institutions to document their operations in order to show their control of operational risk. Nevertheless, a few decades on, it is clear that the application of formal system dynamics modeling tools is rare in finance, at least in the minds of systems modeling experts, when compared with some other industries.

Wholesale financial institutions may have smaller transaction volumes than retail institutions, but even a modest investment bank can process 250,000 equity trades a week (Z/Yen global investment banking benchmarks). The very largest investment banks might handle 40 million equity trades a year (Z/Yen global investment banking benchmarks) and large amounts of foreign exchange, money markets, and other instruments. With large numbers of transactions, numerous paths, and variable activity levels this should be a fruitful environment for the application of system dynamics, yet most wholesale operations managers do not believe that system dynamic techniques bear fruit. A number of factors contribute to this lack of apparent benefit. First, wholesale trading finance is a fast-changing environment with little time for analytical reflection and a need for quick payback on investment in operations. Secondly, there has been only a modest amount of emphasis on the back-office processing operations. Instead, most of the emphasis has been on supporting the front-office trading floor. Thirdly, wholesale institutions tend to respond positively to regulatory initiatives setting out operations standards, but otherwise do what everyone else is doing. Thus, despite a few trials of system dynamics approaches, as almost no one has a big success story in an environment where rapid, perhaps overly rapid, decisions are taken, almost no one will undertake a systems modeling project.

Operational Risk

According to §644 of the “International Convergence of Capital Measurement and Capital Standards”1 from the Bank for International Settlements, known as Basel II,2 operational risk is defined as the “risk of loss resulting from inadequate or failed internal processes, people and systems, or from external events.” Since operational risk became a regulatory discussion topic in the early 1990s, a number of approaches have been tried to both measure it and manage it, and all have been found wanting. Arguably, the evolution of current thinking about operational risk has already had three stages:

• OpVAR (operational value-at-risk). This was an early approach that attempted to treat operational risk in the same manner as market and credit risk. The basic idea was to build a large, stochastic model of the various operational risks and use Monte Carlo simulations to calculate a “value-at-risk” that would allow a financial institution to set aside an appropriate amount of capital. This approach requires probability distributions of operational risk, in the same way that banks analyze market movements or credit defaults. A few industry initiatives attempted to collect large datasets of operational risk losses, for example, defalcation by employees, but found that the data were heterogeneous and difficult to extract because of its sensitivity—who wants to admit publicly that they have been defrauded? In our opinion, OpVAR still has a place as a useful analytical check, but not as a primary means of measuring and managing operational risk.
• Process modeling. Many financial institutions documented their operations in order to analyze their operational risks. Many of the tools used to document the operations were also the same tools used to input models to system dynamics simulation software. While this also led many institutions to experiment with system dynamics techniques, they then encountered problems of validating the models and chaos theory effects, that is, extreme sensitivity to initial conditions. Further, this approach failed to provide a useful measure for banks to calculate an appropriate amount of capital to set aside to cover operational risk.
• Risk dashboards or “radar.” Some financial institutions explored the application of compliance tools that required operational managers to prove that they had followed procedures that minimized operational risk. While this heuristic approach is culturally suited to banks (it has the bureaucratic “tick-bashing” and form-filling with which they are so familiar), it also fails to provide an overall measure of operational risk. Further, there was little consideration of the human systems within which this approach was being applied so, for instance, people just repeatedly answered questions with the desired answer, for example, “is your computer room secure?”—“yes,” thus negating any benefit. Finally, this approach results in a lot of RAG (red-amber-green) type reports that cannot be readily summarized numerically and are incapable of contrasting different risks other than by their frequency or place in the taxonomy. So, with little account taken of the severity, five open computer room door incidents may be rated more important than a single total power ­outage.

Elements of these three approaches are still used, and useful, but on their own they do not provide measurement and management of operational risk. There are some other approaches worth noting. Though these have not been as popular, they may have more long-lasting benefits:

Culture change. As operational risk is primarily risk generated by ­people internally (people fail to follow processes, deliberately sabotage or make poor decisions), a culture that promotes reduced operational risk should provide significant benefits.3 This approach, however, does not provide a measure of operational risk for capital purposes.

Cost-per-transaction variance. This approach attempts to contrast operational risk across products by fully allocating costs to each transaction, thus generating a more typical distribution curve for risk.4 This approach does find system dynamic modeling useful, to help allocate pooled costs based on activity levels (activity-based costing), and does appear to work in practice and across the industry. However, this approach has not been widely adopted, possibly because the full-blown version requires extensive, and expensive, systems modeling, and possibly because the regulators have been slow to see that it does help them provide comparable metrics; though those metrics are not traditional.

Another approach worth evaluating, which leads to a slightly different view of how system dynamics is applied to the organization, is the use of key risk indicators.

What Is a Key Risk Indicator (KRI)?

Key Risk Indicators: risk indicators are statistics and/or metrics, often financial, which can provide insight into a bank’s risk position. These indicators should be reviewed on a periodic basis (often monthly or quarterly) to alert banks to changes that may be indicative of risk concerns. Such indicators may include, for example, the number of failed trades, staff turnover rates, and the frequency and/or severity of errors and omissions.5

KRIs are measurable metrics (sic) or indicators that track different aspects of operational risk.6

A working definition is “regular measurement based on data which indicates the operational risk profile of a particular activity or activities.” KRIs can be environmental, operational or financial. For instance, environmental indicators (that might turn out to be KRIs) could be such things as trading volumes and volatilities on major commodities or foreign exchange markets. Operational indicators (that might also be KRIs) could be general activity levels in the business, number of deals, mix of deals, number of amendments, staff turnover, overtime or IT downtime. Financial indicators (that might also be KRIs) could be things such as deal volatility, dealing profit, activity-based costing variances or value of amendments.

The key link is to apply a more scientific approach to managing risk. Firms must test the usefulness of operational risk data collection by using losses or incidents to discover what the indicators should have been. In other words, what drives operational risk? We describe this approach as predictive key risk indicators to/from loss/incidents prediction (PKRI⇔LI).

The important point to note is that people can suggest many possible risk indicators (RIs), but they are not KRIs unless they are shown to have predictive capability for estimating losses and incidents. A KRI must contribute to the predictability of losses and incidents in order to be validated as a KRI. If an RI does not predict losses or incidents, it remains an interesting hypothesis, someone’s unvalidated opinion. Experience does help to identify the true drivers of operational risk and should help focus attention and control actions, but the PKRI⇔LI approach supports and validates (or invalidates) expert judgment of true drivers of operational risk losses. The intention of this approach is not to replace expert ­judgment, but to support that judgment in a more systematic way in an ever-changing environment.

Why Are KRIs Important?

KRIs are important for at least four reasons:

• KRIs measure probable operational risk arising over a time period, as opposed to tracking operational risk, and thus make an appropriate management tool for operational risk;
• KRIs help to form an input for economic capital calculations by helping to produce estimates of future operational risk losses and thus helping to set a base level of capital for ­operational risk;
• KRIs are increasingly examined by rating agencies, for example, Moody’s or Standard & Poor’s, and financial analysts; and
• KRIs are increasingly important to regulators.

Without capturing incidents and loss data, there is nothing to predict. Sound incident-data capture is a prerequisite for anything but the most basic capital allocation under Basel II. It is worth quoting at length from Basel II7 as this shows what regulators expect both for operational risk and for key risk indicators.

Exhibit 10.1 How should KRI’s work?

Source: Z/Yen Group Limited.

In addition to using loss data, whether actual or scenario-based, a bank’s firm-wide risk-assessment methodology must capture key ­business environment and internal control factors that can change its operational risk profiles. These factors will make a bank’s risk assessments more ­forward-looking, more directly reflect the quality of the bank’s control and operating environments, help align capital assessments with risk management objectives, and recognize both improvements and deterioration in operational risk profiles in a more immediate fashion. To qualify for regulatory capital purposes, the use of these factors in a bank’s risk measurement framework must meet the following standards:

• The choice of each factor needs to be justified as a meaningful driver of risk, based on experience and involving the expert judgment of the affected business areas. Whenever possible, the factors should be translatable into quantitative measures that lend themselves to verification.
• The sensitivity of a bank’s risk estimates to changes in the ­factors and the relative weighting of the various factors need to be well reasoned. In addition to capturing changes in risk due to improvements in risk controls, the framework must also capture potential increases in risk due to greater ­complexity of activities or increased business volume.
• The framework and each instance of its application, including the supporting rationale for any adjustments to empirical estimates, must be documented and subject to independent review within the bank and by supervisors.
• Over time, the process and the outcomes need to be validated through comparison to actual internal loss experience, relevant external data, and appropriate adjustments made.

There are a number of KRI initiatives in the financial services industry to share best practice on KRIs and loss/incident reporting or collection. A leading initiative is the Risk Management

Association’s KRI Banking Study, KRIeX,8 in which some 50 banks defined 1,809 KRIs, though the relevance of these has not been tested using PKRI⇔LI prediction (to be fair, there has been some talk of doing something at an unspecified point in the future). Some examples of the Risk Management Association’s KRIs are the percentage of transactions requiring manual input, percentage of unsettled transactions after due dates and theft per 1,000 ATMs. A breakdown of the KRIs by category and number is set out in the following Exhibit 10.2.

Exhibit 10.2 KRIeX RIs by category

Source: Risk Management Association (author’s chart).

It is implausible to ask any organization to track 1,809 key risk indicators. To be fair, only 74 indicators are common and apply to virtually all risk points. Further, only (sic) 533 are “high-risk points.” While some activities are “done to be seen” by regulators, KRIeX remains a valuable resource. This is an exhaustive approach at an early stage that does help by providing a starting set of RIs, but the KRIs for different institutions must evolve from individual institutional experience, rather than being imposed over-heavily from a template. Exhibit 10.3 sets out the characteristics of a KRI as seen by the Risk Management Association.

Exhibit 10.3 KRI characteristics

Effectiveness	Comparability	Ease of Use
Indicators should…. Apply to at least one risk point, one specific risk category and one business function. Be measurable at specific points in time. Reflect objective measurement rather than subjective judgement. Track at least one aspect of the loss profile or event history, such as frequency, average severity, cumulative loss or near-miss rates. Provide useful management information.	Indicators should…. Be quantified as an amount, a percentage or a ratio. Be reasonably precise and define quantity. Have values that are comparable over time. Be comparable internally across businesses. Be reported with primary values and be meaningful without interpretation to some more subjective measure. Be auditable. Be identified as comparable across organisations (if in fact they are).	Indicators should…. Be available reliably on a timely basis. Be cost-effective to collect. Be readily understood and communicated.

Source: Risk Management Association.

In a sense, the choice is between what is currently done informally (no significant business lacks RIs) and what could be done better through more formality, statistics and science to make them KRIs. For each KRI, there needs to be definition and specification. The Risk Management Association’s template specification structure (see Exhibit 10.4) gives a flavor of what this means.

Exhibit 10.4 KRI template

Definition	Specification	Guidance
KRI number KRI name Description Rationale/Comments Nature Type Typography Ratings	Specification version Term definitions Value kind Dimensions Limitations on scope Buckets Bucket variants Definition thresholds Measurement rules Underlying KRIs Calculation method Benchmark rules Aggregation method Scaling denominator Scaling rules	Usage Collection frequency Reporting frequency Frequency of change Collection level Variants Directional information Extraneous influences Control indicator Source

Source: Risk Management Association.

PKRI⇔LI Issues

One could readily conclude that a fairly static KRI cannot be “key.” For example, a KRI such as the number of lawsuits received by a particular function might change very little for long periods. In this case, one might wish to examine “lawsuits in period,” “estimated settlement values” or other more sensitive measures than just a very slow-changing “outstanding lawsuits.” However, what matters is whether or not the KRI contributes to the capability of predicting operational losses/incidents, not its variability.

There is overlap between KRIs and key performance indicators (KPIs). It would be easy to say that KRIs are forward-looking and KPIs are backward-looking, but far too simplistic. There are clearly overlaps. For instance, high trading volumes and high volatility on one day might be good performance indicators predicting a high likelihood of good future financial performance turnout for that day, but also indicative of emerging operational risks from that day.

KRIs that increase in some ranges and decrease in others can cause confusion as KRIs are not necessarily linear. For example, staff overtime might be an example of a KRI with a bell-shaped curve. No overtime may indicate some level of risk as people aren’t paying attention or do tasks too infrequently; modest levels of overtime may indicate less risk as staff are now doing a lot of familiar tasks; and high rates of overtime may indicate increased risk again through stress. KRIs help to set ranges of acceptable activity levels. There can be step changes in operational risk associated with a KRI. For instance, a handful of outstanding orders at the close of day may be normal, but risk might increase markedly when there are over a dozen outstanding orders. KRIs should vary as risk changes, but they do not have to vary linearly.

What about all the stuff that is taken for granted? For example, electricity and water supplies may seem to be an important consideration when looking at KRIs for developing world locations, yet do not really feature in criteria in the developed world. In the major financial centers, many things are assumed, for instance, an absence of natural threats such as hurricanes or flooding, yet London used to have a significant flood risk, and may again as the Thames Barrier comes to the end of its projected usefulness. Geological issues such as earthquake-prone faults or health issues such as malaria do not seem to feature. Nor does terrorism risk seem strong in people’s perceptions of what really matters. There are also numerous personal issues that do not feature—work permits, opening bank accounts, arranging for utilities, schools, personal safety—any of which could scupper a trading floor.

Somewhat naturally, people tend to care about those things of which they are most conscious. Any of a number of issues could have us looking back several years from now and grimly nodding about how trading ceased to function when “people wanted to avoid concentrating terrorism risk” or “infectious diseases just became too dangerous to have people so highly concentrated.” The PKRI⇔LI approach is an approach for regular management, not extreme events.

It is a combination of factors that makes a set of KRIs successful, not just a single factor. Jared Diamond derives an Anna Karenina Principle from the opening line of Tolstoy’s novel: “Happy families are all alike; every unhappy family is unhappy in its own way.”9 Diamond believes the principle describes situations where a number of activities must be done correctly in order to achieve success, while failure can come from a single, poorly performed activity. This is certainly the case for KRIs—the evolving set of KRIs is important, not a single one at a point in time, nor too many all the time.

Environmental Consistency Modeling Using Support Vector Machines

Two examples of environmental consistency confidence projects using PKRI⇔LI are explained a bit later (a European investment bank and a global commodities firm), but it is worth looking at the support vector machine approach that underlay the modeling. Both projects used classification and prediction tools based on support vector machine mathematics to undertake predictive analysis of the data. Support vector machines (SVMs) are algorithms that develop classification and regression rules from data. SVMs result from classification algorithms first proposed by Vladimir Vapnik in the 1960s, arising from his work in Statistical Learning Theory.10 SVMs are based on some wonderfully direct mathematical ideas about data classification and provide a clear direction for machine learning implementations. While some of the ideas behind SVMs date back to the 1960s, computer implementations of SVMs did not arise until the 1990s with the introduction of a computer-based approach at COLT-92.11

SVMs are now used as core components in many applications where computers classify instances of data (e.g., to which defined set does this group of variables belong?), perform regression estimation and identify anomalies (novelty detection). SVMs have been successfully applied in time-series analysis, reconstructing chaotic systems and principal component analysis. SVM applications are diverse, including credit scoring (good or bad credit), disease classification, handwriting recognition, image classification, bioinformatics, and database marketing, to name a few.

SVMs are said to be independent of the dimensionality of ­feature space as the main idea behind their classification technique is to ­separate the classes in many data dimensions with surfaces ­(hyperplanes) that ­maximize the margins between them, applying the structural risk ­minimization principle. The data points needed to describe the ­classification algorithmically are primarily those closest to the hyperplane boundaries, the “support vectors.” Thus, only a small number of points are required in many complex feature spaces. SVMs can work well with small datasets, though the structure of the training and test data is an important determinant of the effectiveness of the SVM in any specific application.

SVMs compete forcefully with neural networks as well as other machine-learning and data-mining algorithms as tools for solving ­pattern-recognition problems. Where SVMs do not perform well, it is arguable that the algorithmic rules behind the support vector algorithm do not so much reflect incapabilities of the learning machine (as in the case of an overfitted artificial neural network) but rather regularities of the data. In short, current opinion holds that if the data in the domain are predictive, SVMs are highly likely to be capable of producing a predictive algorithm. Importantly, SVMs are robust tools (understandable implementations, simple algorithmic validation, better classification rates, overfitting avoidance, fewer false positives, and faster performance) in practical applications. “The SVM does not fall into the class of “just another algorithm” as it is based on firm statistical and mathematical foundations concerning generalization and optimization theory.”12 However, comparative tests with other techniques indicate that while they are highly likely to be capable of predicting, in applications SVMs may not be the best approach for any specific dataset. “In short, our results ­confirm the potential of SVMs to yield good results, especially for classification, but their overall superiority cannot be attested.”13

Environmental Consistency Confidence

The conjunction of SVMs with traditional system dynamics may seem unorthodox, but it gives organizations the capability of regularly applying scientific management. Our hypothesis is that certain KRIs predict future losses and incidents, so let us test that using modern statistical tools. If our environmental factors are consistent with the outcomes, then we can be confident we are tracking the right things. From the fact that we are tracking the right factors, we should then develop projects to eliminate or mitigate the causes. If we fail to predict, we are not tracking the right things and need to explore further, and fairly rapidly as it indicates that things may be “out of control.”

If we look at the wider system of wholesale financial institutions, we see similar high-level systems that can be predicted, not just operational risk. The following exhibit sets out a simple model of finance as one where risks are selected through positioning and marketing and then priced by attempting to ascertain the difference in value to customers and the cost of capital.

At each point in this abstract model of finance, we can use a KRI system, for example:

• Marketing: can we predict sales?
• Pricing: can we predict profitability?
• Underwriting/trading: can we predict incidents and losses?

Exhibit 10.5 Financial services risk/reward cycle

Source: Author’s own.

A KRI system, as with any system, has basic components:

• Governance. Working from the overall objectives of the ­business, set out a definition of the operational risk framework, the calculation of economic capital and a basic set of essential KRIs.
• Input. Gaining stakeholder commitment, assembling resources, and appointing a team that then work to establish the potential KRIs.
• Process. Supporting the operational risk managers through data collection, statistical validation, statistical testing, correlations, multivariate prediction, cross-project discussion, training, template materials, and methodologies.
• Output. Evaluating KRIs, focused on a “customer” point of view (how does this help me manage my business better?), so that people learn from both successes and failures.
• Monitoring. Providing management information up to governors, over to customers, down to project managers and across project managers so that they are coordinated. Monitoring also uses feedback from KRI outcomes to feed forward into new KRI ideas and re-plan the shape of the KRI portfolio. An integral part of monitoring is evaluating KRIs at a technical level—do they predict? PKRI⇔LI prediction is one direction, and LI⇔PKRI is another.

The KRI system is a classic feedback and feed-forward cybernetic ­system. KRIs help managers to manage by reducing the amount of ­measures they need in both feed-forward and feedback. So the crucial distinction is between RIs and KRIs using PKRI⇔LI, as KRIs help to combat information overload:

What information consumes is rather obvious: it consumes the attention of its recipients. Hence a wealth of information creates a poverty of attention, and a need to allocate that attention efficiently among the overabundance of information sources that might consume it.14

By giving managers a clear focus on the operational risk drivers, they can commission further work to reduce them. The KRI system can be represented diagrammatically as shown in Exhibit 10.6.

Exhibit 10.6 KRIs as a system

Source: Z/Yen Group Limited.

Thus, in many ways, the PKRI⇔LI approach is a classic system dynamics approach, but the use of the SVM to link inputs (KRIs) with outputs (incidents and losses) focuses on establishing predictive relationships rather than presuming that the dynamic modeling paradigm is intrinsically important to either how those relationships are validated or how they are interpreted.

The PKRI⇔LI approach is a dynamic process, not a project to develop a static set of KRIs. This means that a team, possibly aligned with other “scientific” management approaches such as 6Σ, needs to be constantly cycling through an iterative refinement process over a time period. This leads to the development of cyclical methodologies. For example, Z/Yen’s Z/EALOUS methodology is one such, and diagrammatically illustrated overleaf.

What Is Current Practice? Two Early Examples

Scientific management of wholesale financial operations is increasing. Investment banks have increased their operational benchmarking markedly since the late 1990s. Managers in many investment banks (e.g., Bank of America, JPMorgan Chase, Citigroup and Merrill Lynch, among others, have publicly announced their pursuit of 6Σ at conferences and on websites) follow the DMAIC or DMADV 6Σ approaches (originally from GE) when they have losses/incidents that they want to eliminate by eliminating root causes.

Exhibit 10.7 Environmental consistency confidence in wholesale financial institutions

Source: Z/Yen Group Limited.

6Σ is clearly related to a dynamic system view of the organization, a cycle of tested feed-forward and feedback. This has led to greater interest in using predictive analytics in operational systems management. Several leading investment banks, using 6Σ programs and statistical prediction techniques (predicting trades likely to need manual intervention), have managed to reduce trade failure rates for vanilla products from 8 ­percent to well below 4 percent within three years. As the costs per trade for trades requiring manual intervention can be up to 250 times more expensive than trades with straight-through processing transactions, this is a very important cost-reduction mechanism, as well as resulting in a consequent large reduction in operational risk.

Predictive analytics also feature where investment banks are moving toward automated filtering and detection of anomalies (dynamic anomaly and pattern response, DAPR).15 Cruz notes that a number of banks are using DAPR approaches not just in compliance, but also as operational risk filters that collect “every cancellation or alteration made to a transaction or any differences between the attributes of a transaction in one system compared with another system. . . . Also, abnormal inputs (e.g., a lower volatility in a derivative) can be flagged and investigated. The filter will calculate the operational risk loss event and several other impacts on the organisation.” He continues, “the development of filters that capture operational problems and calculate the operational loss is one of the most expensive parts of the entire data collection process, but the outcome can be decisive in making an operational risk project successful.”16

Exhibit 10.8 DMAIC—Existing product/process/service

Stage	Objectives
Define	Define the project goals and customer (internal and external) deliverables
Measure	Measure the process to determine current performance
Analyze	Analyze and determine the root cause(s) of the defects
Improve	Improve the process by eliminating defects
Control	Control future process performance

Source: www.isixsigma.com

Exhibit 10.9 DMADV—New product/process/service

Stage	Objectives
Define	Define the project goals and customer (internal and external) deliverables
Measure	Measure and determine customer needs and specifications
Analyze	Analyze the process options to meet the customer needs
Design	Design (detailed) the process to meet the customer needs
Verify	Verify the design performance and ability to meet the customer needs

Source: www.isixsigma.com

Exhibit 10.10 DAPR support vector machine example: contrasting a subset of actual versus predicted trade price bands

Source: Z/Yen Group Limited.

Exhibit 10.11 Sample investment bank KRI data

Source: Z/Yen Group Limited.

PKRI⇔LI aligns with this interest in using predictive analytics to improve operational management. While interest in the PKRI⇔LI approach may be rising, particularly among investment banks, there has been a paucity of data available for these purposes. As operational risk units are growing and developing data collection and measurement ­systems, PKRI⇔LI projects are growing in number.

European Investment Bank

One European investment bank used three years’ data to predict losses/incidents from data such as deal problems, IT downtime, and staff turnover over a six-month period. It achieved reasonable predictive success, an R2 approaching 0.9 at times, though more frequently 0.6 (i.e., 60 percent of losses can be predicted). A high-level snippet gives a flavor of the data.

Note that some of the items in this snippet, for example, HR joiners/leavers or IT disruption at the system level, can in practice be very hard to obtain. It was also noteworthy that, as a data-driven approach, PKRI⇔LI projects are only as good as the data put into them—­“garbage in, garbage out.” In some areas, the data may not be at all predictive. More rigor needs to be used as the data become more important. Data quality can vary over time in hard-to-spot ways and interact with wider systems, ­particularly the people in the systems. For instance, in this trial of PKRI⇔LI, the IT department was upset at IT downtime being considered a “key risk indicator” and unilaterally changed the KRI to “unplanned” IT downtime, skewing the predicted losses. This change was spotted when using the DAPR system to run the reverse LI⇔PKRI prediction as a quality control. Another example of Goodhart’s Law, “when a measure becomes a target, it ceases to be a good measure” (as restated by Professor Marilyn Strathern).

Global Commodities Firm

A large global commodities firm active not only in a number of commodity markets but also foreign exchange and fixed income piloted the PKRI⇔LI approach in one large trading unit. While the predictive success was not especially great in the pilot, an R2 approaching 0.5, the approach was seen to have merit and they decided to roll the PKRI⇔LI methodology out globally across several business units. It was telling that the PKRI⇔LI approach helped them to realize the importance of good data collection and use, and to identify areas where their data specification, collection, validation and integration could be markedly improved. It is also important to note that the SVM approach did not add much value in the early stages; many of the predictive relationships were straightforward, for example, large numbers of deal amendments can lead to later problems.

The PKRI⇔LI approach has become part of a more scientific approach (hypothesis formulation and testing) to the management of operational risk.

Modern (organization) theory has moved toward the open-system approach. The distinctive qualities of modern organization theory are its conceptual-analytical base, its reliance on empirical research data, and, above all, its synthesizing, integrating nature. These qualities are framed in a philosophy which accepts the premise that the only meaningful way to study organization is as a system.17

Conclusion

At its root, environmental consistency confidence means building a statistical correlation model to predict outcomes and using the predictive capacity both to build confidence that things are under control, and to improve. The correlations should raise good questions. KRIs are a continuous, evolving system, not static, hence the focus on the cyclical PKRI⇔LI approach. Today’s KRI should be tomorrow’s has-been as managers succeed in making it less of an indicator of losses or incidents by improving the business. Likewise, managers have to consider emerging KRIs and validate them. Wholesale financial institutions can impress regulators with PKRI⇔LI, perhaps reducing regulatory overhead, but far more important is to use KRIs to improve their businesses and reduce operational risk.

Bibliography

Beer, S. 1966 (1994 ed). Decision and Control: The Meaning of Operational Research and Management Cybernetics. Hoboken, NJ: John Wiley & Sons.

Mainelli, M. June 2005. “Competitive Compliance: Manage and Automate, or Die.” In Journal of Risk Finance, The Michael Mainelli Column, Vol. 6, No. 3, pages 280–84, Emerald Group Publishing Limited.

Mainelli, M., I. Harris, and A. Helmore-Simpson. June 2003. “The Auditor’s Cross Subsidy’ (Statistical Modelling of Audit Prices).” Strategic Planning Society E-Newsletter, Article 1. Also published as “Anti-dumping Measures & Inflation Accounting: Calculating the Non-Audit Subsidy.” www.mondaq.com (June 19, 2003).

Michie, D., D.J. Speigelhalter, and C.C. Taylor. 1994. Machine Learning, Neural and Statistical Classification. Hemel, UK: Ellis Horwood, out of print. See http://amsta.leeds.ac.uk/~charles/statlog/

Thanks

I would like to thank Matthew Leitch, Justin Wilson, Ian Harris, Jürgen Strohhecker, Jürgen Sehnert and Christopher Hall for helping to develop some of the thinking behind this article, though not to claim they agree with all of it.

---