Defining model performance

OK then, let's ask a question to start talking about performance: when you estimate a model, how do you say if it is a good model? As you have probably already heard, the American statistician George Box used to say: All models are wrong, but some are useful.

This is, besides a nice quote, also a great truth: there is no perfect model, all models are some kind of an abstraction from reality, like maps are an abstraction from the real Earth. Nevertheless, if those maps are accurate enough, they are invaluable friends in the hands of travelers. This could seem to you nothing more than a suggestive analogy, but it's actually a useful way to intend models since it captures two of their most relevant aspects:

  • Models need to have a good level of abstraction from the real phenomenon they are trying to model
  • Models have to be accurate enough to be useful

I don't need to say to you that the main topics here are to define what a good level of abstraction is and define what it means for a model to be accurate enough. This gives room for two concepts that I am going to discuss with you now:

  • The trade-off between fitting and interpretability, which is closely related to the definition of the good level of abstractions
  • The concept of materiality when making predictions with models, which helps to define if a model is accurate enough
..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset