Model risk management
How Kaimei's analyses give validators and model owners evidence for conceptual soundness and ongoing monitoring, under model risk guidance such as SR 26-2 and SS1/23.
#What validators look for
US supervisory guidance describes model validation in three parts: evaluating conceptual soundness, outcomes analysis, and ongoing monitoring. SR 11-7, issued by the Federal Reserve and the OCC in 2011, set out that framing, and the revised interagency guidance that replaced it in April 2026 (SR 26-2, OCC Bulletin 2026-13 and FDIC FIL-15-2026) keeps all three. It names interpretability measures, alongside benchmarking against other models, as a practical way to assess conceptual soundness, and it leaves generative and agentic AI outside its scope. The Bank of England's PRA sets out similar principles in SS1/23, in effect since May 2024, for firms with internal model approval.
Kaimei does not replace independent validation, which is a matter of people and process. It produces evidence that validators and model owners can examine and keep.
#Evidence Kaimei produces
| Validation activity | What Kaimei provides |
|---|---|
| Conceptual soundness Does the model behave as theory and policy expect? | Global importance; behaviour curves with their direction and consistency; learned rules in plain English; interactions between inputs. |
| Sensitivity and stability | How often decisions flip under small input noise, and which inputs are most fragile. |
| Ongoing monitoring | Drift for every input and for the output, scheduled analyses, alerts, and traffic statistics. |
| Outcomes analysis | Accuracy on labelled reference data, reported with the importance analysis, and realised outcomes attached to logged predictions, backfilled history included: AUC, accuracy, calibration and the confusion matrix per day, week or month, with a performance analysis that flags degradation. |
| Benchmarking | Insights for two versions or two models side by side, and explanations of the same case under each. Challenger benchmarks fitted by Kaimei are coming soon. |
| Audit trail | Every analysis is stored with its options, seed and model version, and every result can be retrieved through the API. |
#Reading the insights
Each analysis writes insights in plain English. These are the ones the credit model produced:
| Insight | What it answers | Example from the credit model |
|---|---|---|
| Global importance | Which inputs the model relies on most, across everyone. | Credit score is the most influential input, followed by debt-to-income and utilisation. Together they carry 59% of the measured importance. |
| Learned rules | How each important input moves the prediction, in a sentence. | Higher credit score decreases the probability of decline, most sharply between 543 and 610. |
| Behaviour curves | The average prediction as one input changes, and whether it moves in one direction. | All six of the most important inputs move the prediction in one direction only. |
| Interactions | Pairs of inputs whose combined effect is more than the sum of their parts. | The strongest pair is income and loan amount, and 6% of their joint effect is non-additive. |
| Robustness | How often small noise in the inputs flips a decision. | 0.9% of decisions flip under noise of 5% of a standard deviation. |
| Drift | Whether recent traffic still looks like the reference data. Population stability under 0.10 is stable and under 0.25 moderate; above that is significant. | Moderate drift, with a stability index of 0.15; loan amount, debt-to-income and income have shifted. |
| Output behaviour | How predictions are distributed, and how many sit close to the threshold. | The model predicts decline 44% of the time, and 15% of cases sit within 0.10 of the threshold. |
#Limits to keep in mind, and what's coming
- Kaimei studies the model on the population you give it, so its conclusions are only as representative as that data.
- Fairness testing, challenger benchmarks and validation reports exported as a document are coming soon. Until then, run fairness tests and challengers with your own tools, and export Kaimei's structure and insights as JSON.
- Analyses use sampling; set a seed when you need to reproduce a result exactly.