Are Your Risk Models Flawed?

July 2009 Issue

By Diana Del Bel Belluz, M.A.Sc., P.Eng.

Have you found yourself having to defend risk management lately? People at the water cooler, in the executive suite and in the boardroom are asking ‘Why should we invest our time and money into implementing risk management in our company when it’s failed so spectacularly elsewhere?’

Much of the disenchantment with risk management stems from the corporate sector’s inability to foresee and prevent the financial meltdown that has resulted in the current global economic recession. It has made people question the value of sophisticated risk assessment models and highly paid risk executives, and the massive investments made by publicly traded companies to comply with Sarbanes-Oxley regulations.

The disappointment in risk management also extends to public institutions over their handling of the recent swine flu outbreak. Some argue the response was too little, too late, e.g, the H1N1 virus spread for many weeks before it was detected and identified as a potential pandemic. Others say the response was too much, too soon, e.g., public health agencies declaring it a pandemic when most infections only produce mild symptoms, and stirring up fear when the impact is less severe than that of typical seasonal influenza.

Will recent failures of the risk management discipline lead to its demise? Let’s not to throw the baby out with the bathwater! Instead of abandoning the discipline, as a risk management leader you need to understand the risk modeling weaknesses that lead to the recent failures so that you can guard against those flaws in your organization.

Here are 3 risk management lessons that can be drawn from recent failures and 9 tips to help you avoid making the same risk management mistakes.

Lesson #1: Blind faith in models is dangerous. It would appear that analysts and executives working in the financial sector forgot the famous quote by George E.P. Box: ”All models are wrong, but some are useful.”

The financial risk analysts created models that showed they could increase profits by getting into sub-prime mortgages. Analysts and executives didn’t question the risk models that showed by spreading the risk amongst many investors, it could be lowered to almost zero! They blindly believed wild risk estimates that were based on faulty assumptions.

TIP #1: Ensure that assumptions that underlie your risk model are reasonable. Ask yourself: Does our risk model produce an adequate approximation of real world cause and effect? For example, is it responsive to factors that could realistically affect the risk level?

In building a model, it can be perilous to ignore factors that are difficult to describe or quantify. In the sub-prime mortgage case, the risk models were built on the assumption that housing prices would always go up. The models didn’t account for the possibility of prices falling and therefore didn’t provide a realistic account of the risk.

TIP #2: Constantly test to ensure that the model assumptions hold true as factors in the business environment change over time. Ask yourself: Do any factors deviate from how the model designer expected them to behave? For example, globalization has made us rethink our assumptions about being able to geographically contain the effects of either an influenza virus or an economic downturn.

TIP #3: Risk assessment models are typically built for a particular risk scenario. If you want to use your model to estimate the risk of different scenarios, you need to make sure the assumptions still hold true for any new context in which you apply the model.

LESSON #2: Quantitative models can inspire over-confidence in our ability to measure risk. When something is expressed as a number, as humans we ascribe accuracy to it, particularly if the model used to generate that number appears scientific. For example ‘a 95% chance’ sounds more accurate than ‘a high chance’.

TIP #4: Rate the quality of your model inputs and outputs. Ask yourself: Is the qualilty of our risk estimates produce disclosed to and understood by the managers who rely on them? Don’t be fooled by numbers. As with any model, ‘garbage in equals garbage out’.

TIP #5: Use the best information available in your qualitative and quantitative risk models. Where possible, use objective facts based on actual experience with the risk you are modeling.

TIP #6: Where facts are not available, develop guidelines to ensure that expert opinion and judgment is applied consistently with as little room for subjective bias as possible.

LESSON #3: Risk models are better suited to anticipating the future than predicting it.

We misuse risk models when we expect them to provide a single, exact prediction of the future. In his book The Prediction Trap and how to avoid it, Randy Park observes that “if you are predicting the future you are invested in your guess … and you will look for evidence to confirm your prediction.” This prediction bias explains why the brightest financial analysts didn’t see their models were wrong.

TIP #7: Express all risk estimates as a range of possible values rather than as a single point value. It is critical to recognize and communicate that risk models produce estimates of risk, NOT definitive measurements of risk.

TIP# 8: Let go of the notion that the purpose of a risk model is to predict what is most likely to happen. Instead, use the risk modeling process to create stories about the range of outcomes that could occur. By considering multiple views of how things might unfold, you’ll be in a much better position to plan your response options. For example, do your risk response plans consider alternate scenarios of how the H1N1 virus might re-emerge during the coming flu season?

TIP# 9: In anticipating the future, seek out information that contradicts your current worldview. You will dramatically reduce the chance of being blindsided. Ask yourself: Whose views does the model NOT consider?

*

Risk models are powerful management tools, but only when properly designed and applied. Do you know how to find the flaws in your risk models that could undermine your risk management implementation efforts?

*

To explore how you can fortify your risk models, contact Diana Del Bel Belluz at Risk Wise: (416) 214.7598 This e-mail address is being protected from spambots. You need JavaScript enabled to view it

Follow the links to:

• Read this month's Bonus Resource -  How NOT to re-invent the "risk management" wheel.
• Download a printable version of the entire JULY 2009 issue of the Risk Management Made Simple Advisory.
  
• View the Article Index to access back issues of the Risk Management Made Simple Advisory.