jump to navigation

Model Validation March 7, 2007

Posted by riskopedia in Basel II, EAD, LGD, Model Validation, PD, Retail Risk.

After all these years of Basel II, and all the talks around various methodologies. Finally there is a regconition of Model validation in the press. Sometime this year, there will be a launch of “The Journal of Risk Model Validation”, you can go to this link http://www.journalofriskmodelvalidation.com/ to find out more.

At the moment, most of the financial institutions used external validators such as various consultancy firms (e.g. Experian Scorex, FICO, Moody’s, S&P etc.) and the big 4 (KPMG, EY, PWC, Deloitte). Seemed like they don’t really have a lot of investment in internal validations. Main reason usually the company sees the “internal” thing is costly and not benefit driven. However, I think the institutions under-estimated the “internal” part and over-estimated the “external” part, for the following:

  1. In most cases, most external validators act in their own interest. Because they got budget and time constraint, they will raise a lot of common issues (learn from other similar models) without really getting into the “details” of the current model, which this is the most time consuming part. With all the modellers out there, you would probably understand that it takes time to understand “other” people work even you might have seen the same type of model before. Little differences in each step make up a big difference. And how often you would think external validators will question the modellers down to death to find out all the details.
  2. Because company “paid” them to do the work, external validators usually try not to be toooo harsh because they want to build up a good relationship for more business going forward.
  3. Normally when the valdiation is completed, it got passed back to the company and thats pretty much the end of story. If 6 months down the track that you or someone has raised issue/s or question/s on the model, you don’t get much back from the external.
  4. Because the validation work has been done externally, the knowledge is pretty much kept between the validators and the modellers. No one else within the company would of kept or transfer that knowledge.
  5. Different companies have different way of doing the models, and thats intellectual properties. Passing extensive number of models externally means giving that knowledge away. Put it this way, the company is “paying” others to “learn” their way of doing things.

 With internal validation team, they can act as “first line of defence”, when they can freely raised issues, ask questions, dig into details. Not only that, because they are employed by the company, how good & details the validation reports become their performance review, which also forced the internal validators to do a good job on it. Lastly, because they work within the same organisation, the knowledge can be kept and traced back easily.

Anyhow, I would be very interested to know how many financial institutions do invest in some sort of model validation and understand the values of them.



1. Drew de Kock - July 13, 2007

No replies yet…strange. I’ll be the first then.

I can’t agree more. Transfer of knowledge between consultants and employees of the company is always going to be difficult (you have highlighted some of the reasons). An Internal validation team, depending on their experience (in general) and number of years with the company, can add much more value.

At our company we have established a completely seperate model build and model development team, in order to address the independence requirement stipulated by Basel II.

I want to raise a seperate issue: initial model validation. We are struggling to determine whose responsibility this actually is (development team vs validation team). A few things to consider here are the impact that initial validation has on the “operating model”, i.e. from model development/enhancement to implementation to ongoing validation. Antoher issue would be independence.

I’d love to have a few more views on validation in general, and on initial validation specifically.

2. BHC - February 14, 2008

I think a close analogy would be internal auditors vs. external auditors. It’s very difficult to obtain true independence and also you would have the time constraints.

What’s important is that the validator knows what issue is material to the business and what is immaterial. But most of these internal validators are usually quant-background, it’s not an easy thing to bridge the gap.

3. Guan Seng - April 8, 2008


I actually head an internal but independent risk models validation team at Stanchart. What you’ve stated about external vendors is absolutely spot on and exactly the same reasons, why the FSA wants all validation work to be performed in-house. Eventually, all the banks will realize that validation is an important Pillar as part of Pillar 2 requirements. thanks.

4. WEO - April 14, 2009

Hi All,
I’m responsible of the Credit Risk Validation Group at a Canadian Bank. Our group is independent from the Modeling Group.

I do agree with all what was said regarding the choice of having internal group instead of hiring external firms. As stated, it is better to keep the knowledge internally and build a robust team able to challenge the modelers instead of dealing with external consultants that usually will not get deep into details like the internal independent team does usually.

I believe the Validation is a very important part in Risk Management in the banks. Also as it was noticed, validation is a difficult matter that requests a deep knowledge of all the modeling methodologies, regulatory requirements….Also very importantly, you should have good PR skills in order to deal with difficult situations that will be raised sometimes when dealing with the modeling teams. It can be hard sometimes! Finally, another important point is to be able to assess the importance of the findings especially for the qualitative points!


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: