Monitor production traffic

Monitor production traffic

Log production traffic, catch input drift on a schedule, measure the model against real outcomes and explain the predictions that look wrong.

Log what the model does in production, then let analyses compare that traffic with the reference population. The pattern is three calls.

  1. Log predictions as they happen with POST /v1/predictions/batch (up to 1,000 per call) or one at a time. Pass your own prediction_id so a complaint can be traced to the exact record, and timestamp for backfills.
  2. Run a drift analysis on a schedule: type = "drift" compares the latest window logged inputs (default 500) with the reference rows, feature by feature, using the population stability index. PSI under 0.10 is stable, under 0.25 moderate, otherwise significant. The insight's severity follows (low, medium, high), and output_drift reports the shift in the predictions themselves.
  3. Explain the outliers: POST /v1/explanations with the logged input and its prediction_id links the explanation to the record. The project overview and GET /v1/organization/overview surface medium and high insights as alerts.

A shadow model is the same pattern with two models: log every request to both and compare their insights before switching traffic. Drift needs at least 20 reference rows and 20 logged predictions; until then the insight says insufficient_data.

Coming soon Kaimei will run these checks on a schedule for you, send alerts to email, Slack, Microsoft Teams, PagerDuty or a webhook, check data quality as predictions arrive, and watch for attribution drift. Until then, call the analysis from your own scheduler. See everything on the way.

# in the serving path, a batch at a time
client.predictions.create_batch(model="credit-risk-v3", items=[
    {"prediction_id": req.id, "input": req.features, "output": {"label": label, "probability": p},
     "latency_ms": ms, "metadata": {"region": "eu-west"}}
    for req, label, p, ms in served
])

# nightly
analysis = client.analyses.wait(client.analyses.create(model="credit-risk-v3", type="drift").id)
drift = client.insights.list(model="credit-risk-v3", type="drift").data[0]
print(drift.severity, drift.summary)
# medium Recent traffic has moderate drift (PSI 0.15). Shifted inputs: `loan_amount`, `debt_to_income`, `income`.
for row in drift.payload["features"][:3]:
    print(row["feature"], row["psi"], row["status"], row["reference_mean"], "->", row["current_mean"])

# a suspicious prediction, explained
pred = client.predictions.retrieve("pred_8f2a19")
client.explanations.create(model=pred.model_id, input=pred.input, prediction_id=pred.id, method="shap")
curl https://api.kaimei.ai/v1/predictions/batch \
  -H "Authorization: Bearer $KAIMEI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "credit-risk-v3", "items": [
        {"prediction_id": "pred_8f2a1a", "input": {"income": 54000, "credit_score": 681, "utilization": 0.41}, "output": 0.63},
        {"prediction_id": "pred_8f2a1b", "input": {"income": 72000, "credit_score": 790, "utilization": 0.08},
         "output": {"label": "approve", "probability": 0.91}}
      ]}'

curl https://api.kaimei.ai/v1/analyses -H "Authorization: Bearer $KAIMEI_API_KEY" \
  -H "Content-Type: application/json" -d '{"model": "credit-risk-v3", "type": "drift"}'

curl "https://api.kaimei.ai/v1/insights?model=credit-risk-v3&type=drift" -H "Authorization: Bearer $KAIMEI_API_KEY"
200 OKthe drift insight
{
  "object": "insight",
  "id": "ins_a0a2b5c4a5ddb552758a",
  "model_id": "mdl_390bd6d9568f626851ef",
  "analysis_id": "anl_b8d673126dcf3cdbe79a",
  "type": "drift",
  "title": "Input drift",
  "summary": "Recent traffic has moderate drift (PSI 0.15). Shifted inputs: `loan_amount`, `debt_to_income`, `income`.",
  "payload": {
    "features": [
      {
        "feature": "loan_amount",
        "psi": 0.148039,
        "status": "moderate",
        "reference_mean": 23589.0,
        "current_mean": 27721.72
      },
      …
    ],
    "overall_psi": 0.148039,
    "status": "moderate",
    "reference_rows": 600,
    "current_rows": 198,
    "drifted_features": [
      "loan_amount", "debt_to_income", "income"
    ],
    "output_drift": {"psi": 0.071375, "status": "stable"}
  },
  "score": 0.148039,
  "severity": "medium",
  "created_at": "2026-10-31T22:40:25Z"
}

#Measure performance against outcomes

Drift says the inputs moved; it can't say whether the model got worse. For that Kaimei needs what really happened: the loan that defaulted, the claim that was fraud. Outcomes usually arrive late, so they are attached to the predictions already logged.

  1. Log predictions with an id you can find again, such as your application or claim id, as prediction_id.
  2. Attach outcomes as they come in with POST /v1/predictions/actuals: a class name or position, true / false for a binary model, or a number for a regression. A nightly job that sends the day's resolved cases is enough. GET /v1/predictions?has_actual=false lists those still waiting.
  3. Read performance over time with GET /v1/models/{model}/performance, or on the model's Performance tab in the dashboard: AUC, accuracy, calibration and the confusion matrix (MAE and RMSE for a regression), per day, week or month.
  4. Check for degradation on a schedule: the performance analysis compares the newest window outcomes with the ones before them and raises a high insight when the drop is larger than chance explains. Full analyses include it.

Each comparison needs at least 30 outcomes a side; until then the insight says insufficient_data. The check works for models Kaimei can't run itself, so a model registered with inference: none is monitored the same way.

Why it matters For a lender, outcome monitoring is part of model risk management (the outcomes analysis and ongoing monitoring that US model risk guidance asks for, now in SR 26-2); under the EU AI Act, providers of high-risk systems run post-market monitoring, which covers the accuracy they declared. The performance series and insights are that evidence.

# at decision time
client.predictions.create(model="credit-risk-v3", prediction_id=application.id,
                          input=application.features, output={"label": label, "probability": p})

# nightly, as loans resolve: {application id: outcome}
client.predictions.log_actuals({loan.application_id: "decline" if loan.defaulted else "approve"
                                for loan in resolved_today}, model="credit-risk-v3")

report = client.models.performance("credit-risk-v3", window="week", days=180)
print(report.coverage, report.overall["auc"], [w["auc"] for w in report.series][-4:])

client.analyses.wait(client.analyses.create(model="credit-risk-v3", type="performance").id)
check = client.insights.list(model="credit-risk-v3", type="performance").data[0]
print(check.severity, check.summary)
curl https://api.kaimei.ai/v1/predictions/actuals \
  -H "Authorization: Bearer $KAIMEI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "credit-risk-v3", "items": [{"prediction_id": "app_10482", "actual": "decline"}]}'

curl "https://api.kaimei.ai/v1/models/credit-risk-v3/performance?window=week&days=180" -H "Authorization: Bearer $KAIMEI_API_KEY"

curl https://api.kaimei.ai/v1/analyses -H "Authorization: Bearer $KAIMEI_API_KEY" \
  -H "Content-Type: application/json" -d '{"model": "credit-risk-v3", "type": "performance"}'
200 OKthe performance insight
{
  "object": "insight",
  "id": "ins_42222720abfb279ace59",
  "model_id": "mdl_390bd6d9568f626851ef",
  "analysis_id": "anl_c3ae08e5cfb37102a924",
  "type": "performance",
  "title": "Performance against outcomes",
  "summary": "AUC on the latest 500 outcomes is 0.978, in line with 0.991 on the 785 outcomes before them.",
  "payload": {
    "status": "stable",
    "metric": "auc",
    "higher_is_better": true,
    "comparison": {
      "metric": "auc",
      "higher_is_better": true,
      "baseline": 0.990539,
      "current": 0.977935,
      "change": -0.012604,
      "relative_change": -0.012724,
      "z": -1.667,
      "significant": false,
      "status": "stable",
      "severity": "low"
    },
    "baseline_source": "earlier_traffic",
    "current": {
      "n": 500,
      "auc": 0.977935,
      "accuracy": 0.914,
      …: …
    },
    "baseline": {"n": 785, "auc": 0.990539, …: …},
    "series": …
  },
  "score": 0.977935,
  "severity": "low",
  "created_at": "2026-11-08T17:50:36Z"
}