Model risk management

Model risk management

How Kaimei's analyses give validators and model owners evidence for conceptual soundness and ongoing monitoring, under model risk guidance such as SR 26-2 and SS1/23.

#What validators look for

US supervisory guidance describes model validation in three parts: evaluating conceptual soundness, outcomes analysis, and ongoing monitoring. SR 11-7, issued by the Federal Reserve and the OCC in 2011, set out that framing, and the revised interagency guidance that replaced it in April 2026 (SR 26-2, OCC Bulletin 2026-13 and FDIC FIL-15-2026) keeps all three. It names interpretability measures, alongside benchmarking against other models, as a practical way to assess conceptual soundness, and it leaves generative and agentic AI outside its scope. The Bank of England's PRA sets out similar principles in SS1/23, in effect since May 2024, for firms with internal model approval.

Kaimei does not replace independent validation, which is a matter of people and process. It produces evidence that validators and model owners can examine and keep.

#Evidence Kaimei produces

Validation activityWhat Kaimei provides
Conceptual soundness
Does the model behave as theory and policy expect?
Global importance; behaviour curves with their direction and consistency; learned rules in plain English; interactions between inputs.
Sensitivity and stabilityHow often decisions flip under small input noise, and which inputs are most fragile.
Ongoing monitoringDrift for every input and for the output, scheduled analyses, alerts, and traffic statistics.
Outcomes analysisAccuracy on labelled reference data, reported with the importance analysis, and realised outcomes attached to logged predictions, backfilled history included: AUC, accuracy, calibration and the confusion matrix per day, week or month, with a performance analysis that flags degradation.
BenchmarkingInsights for two versions or two models side by side, and explanations of the same case under each. Challenger benchmarks fitted by Kaimei are coming soon.
Audit trailEvery analysis is stored with its options, seed and model version, and every result can be retrieved through the API.

#Reading the insights

Each analysis writes insights in plain English. These are the ones the credit model produced:

InsightWhat it answersExample from the credit model
Global importanceWhich inputs the model relies on most, across everyone.Credit score is the most influential input, followed by debt-to-income and utilisation. Together they carry 59% of the measured importance.
Learned rulesHow each important input moves the prediction, in a sentence.Higher credit score decreases the probability of decline, most sharply between 543 and 610.
Behaviour curvesThe average prediction as one input changes, and whether it moves in one direction.All six of the most important inputs move the prediction in one direction only.
InteractionsPairs of inputs whose combined effect is more than the sum of their parts.The strongest pair is income and loan amount, and 6% of their joint effect is non-additive.
RobustnessHow often small noise in the inputs flips a decision.0.9% of decisions flip under noise of 5% of a standard deviation.
DriftWhether recent traffic still looks like the reference data. Population stability under 0.10 is stable and under 0.25 moderate; above that is significant.Moderate drift, with a stability index of 0.15; loan amount, debt-to-income and income have shifted.
Output behaviourHow predictions are distributed, and how many sit close to the threshold.The model predicts decline 44% of the time, and 15% of cases sit within 0.10 of the threshold.

#Limits to keep in mind, and what's coming

  • Kaimei studies the model on the population you give it, so its conclusions are only as representative as that data.
  • Fairness testing, challenger benchmarks and validation reports exported as a document are coming soon. Until then, run fairness tests and challengers with your own tools, and export Kaimei's structure and insights as JSON.
  • Analyses use sampling; set a seed when you need to reproduce a result exactly.