kasaai / explain-ml-pricing Goto Github PK

While we're targeting Variance, it would be helpful for the broader community to parse the paper if we qualify that exams/ASOPs are for CAS/US. Even better if we can cite some international societies' efforts.

From the ratemaking section

3. Perform analysis on the data, employing desired method or methods to estimate
needed rate relativites
4. Select final rate relativities based on rate indications
5. Present rates to the reglator, including explanation of the steps followed to derive
the rates
6. Answer questions from regulators regarding the method employed
The focus of this paper is on steps 5 and 6

I don't think we can get at relativities with ML models, so maybe we need to revise this bit to say 4-5 will be different in the world with ML models. It may also make sense to point out somewhere that we'll be using ML as a drop-in replacement for GLM rather than for feature engineering only.

This is an open-ended question that I mentioned on our call. Is it realistic to assume that there could be a single standardized set of questions that, properly answered, could reasonably qualify any model?

For example, suppose the question were simply:
"Demonstrate that the rates produced by the model are not inadequate, excessive, or unfairly discriminatory."

That would technically address any concern insofar as a successful answer would mean the model could be approved, but realistically speaking, chances are that it would not produce good regulatory outcomes since modelers wouldn't really know how to answer that.

What I mean to say is that I guess after reading the paper over, I prefer to think of the "question and answer" framework in terms of a set of idealized questions and that a perfect set of questions may not exist that cover every model. My sense has been that no matter how detailed your questions and how varied, you'll probably see some model that leaves you with additional questions from time to time.

That is, I see the question-and-answer thing more as a metaphor for how the actuary should conceptualize the requirement to communicate to intended users of a model than as a suggestion to actually come up with a list of specific questions.

IF you agree - and I suppose that could be a big if - then we may want to consider rewording some of the language around the "standardized set of questions" a bit accordingly.

since rugs are no more

Reminder to point this out when talking about interpretability/transparency... Linear models are "interpretable", but if you have a high dimensional GLM with a bunch of interactions or variable transformations, you can quickly lose that.

Because many machine learning models are deterministic, they may not admit of standard metrics for model comparison (e.g., it’s not straightforward to calculate an AIC over a neural network).

We'll want to reword this e.g. "does not assume an underlying stochastic process" since deterministic has a different meaning in ML

When run the R codes:
fi <- ingredients::feature_importance(
explainer_nn,
loss_function = function(observed, predicted, weights) {
sqrt(
sum(((observed - predicted) ^ 2 * weights) /sum(weights))
)
},
weights = testing_data$exposure,
variables = predictors,
)
the result is “Error in loss_function(observed, predict_function(x, sampled_data)) : argument "weights" is missing, with no default”.
The problem should be “loss_function”,but I don’t know why.

one of the social criteria for rating variables is controllability of premiums by policyholders, we may want to call this out somewhere when discussing questions to ask #8

Some candidates

1. How important is each variable to the model?
2. Given a policy, how do the different characteristics (age, location, make, etc.) contribute to its predicted loss cost?
3. How does the predicted loss cost change if we change a variable a bit?

Brief overview of the ratemaking workflow

Is it accurate to say GLM is state of the art for risk classification today? Are there sources we can/should cite or does it go without saying?

literature review for ML interpretability would be included here, also defining interpretability for the purpose of the paper

———. 2018. “Regulatory Review of Predictive Models 10/25/18 Exposure Draft.” Casualty Actuarial; Statistical (C) Task Force.
https://www.naic.org/documents/cmte_c_catf_
exposure_predictive_model_white_paper.pdf.