Platform · Monitoring

Watch every model, and know why when it moves.

Log the predictions your stack already makes, or let Kaimei host the model and predict for you. Either way, every logged prediction can be explained after the fact, and drift arrives with the input that caused it.

Monitoring · grid load forecasting

A pipeline bug shows up as drift.

Two weeks of logged forecasts with a shadow model, then a weather feed that starts sending temperature in Fahrenheit. Nothing errors. Follow the timeline to see Kaimei catch it in one batch, and an explanation name it.

  1. 01Days 1 to 14

    Every prediction, logged

    A neural network that forecasts a city's hourly electricity load is in production. The service that runs it logs each input and forecast to Kaimei; 536 hours of 2017 that the model never trained on stand in for two weeks of traffic. The band behind them is where the temperatures of its 3,340 reference hours sit.

  2. 02Days 1 to 14

    A shadow alongside

    A gradient-boosted model scores the same hours in shadow: logged, never acted on. The network's forecasts average 70.3 MW, the shadow's 70.2 MW. They are more than 5 MW apart on 6 of the 536 hours, which is where switching models would change what an operator sees. Scored against the actual load, the network is out by 1.9 MW on average and the shadow by 1.6 MW.

  3. 03Day 14

    A clean check

    A drift analysis compares the latest logged inputs, up to 500, with the reference rows, one input at a time, using the population stability index. Below 0.1 is stable. Every input is: max PSI 0.05.

    Drift analysis · load-forecast-nnday 14

    Every input stable

    Recent inputs match the reference population (max PSI 0.05).

  4. 04Days 15 and 16

    A new feed goes live

    A new weather feed sends temperature in Fahrenheit, not Celsius: the first hour through it was 16.5 °C and is logged at 61.74. Nothing errors. All 300 of its hours arrive hotter than any hour in the reference data, and every forecast comes out higher: 96.5 MW on average as logged, against 68.6 MW with the temperature in Celsius.

  5. 05Day 16

    Drift on one input

    The same check, two days later: 3,340 reference rows against the latest 500 logged forecasts. Temperature's PSI is 1.548, far past the 0.25 line for a significant shift. Every other input is stable. The forecasts themselves have shifted too: PSI 0.791.

    Drift analysis · load-forecast-nnday 16

    3,340 reference rows vs the latest 500 logged predictions

    1.548Temperature
    Humidity0.034
    Hour0.031
    Load −24h0.023
    Solar0.015
    Load −168h0.011
    Weekend0.008
    Wind0.003
    0.791Predictions

    Every other input 0.034 or less: stable

    stable < 0.1moderatesignificant > 0.25

  6. 06Day 16

    Flagged at high severity

    Kaimei records an Input drift insight at high severity, and the project page lists it under Needs attention. It says which input moved: temperature. Drift needs no outcomes, so the flag does not wait for the actual load to come in.

    Needs attention

    Input drift · City load · neural networkRecent traffic has significant drift (PSI 1.55)… High

    Insightins_622c0b67d6b1d8196ee5

    Input drift

    DriftHighscore 1.548

    Recent traffic has significant drift (PSI 1.55). Shifted inputs: temperature.

    3,340 reference vs 500 recent rowspredictions PSI 0.791 · significant

  7. 07Day 16

    Named by an explanation

    Explain the first forecast logged from the new feed: a Sunday in January at 17:00, forecast 88.2 MW. The reading of 61.74 is the largest push by far, +24.3 MW; no other input moves the forecast by more than 3.7 MW. The explanation is linked to the logged prediction by its id.

    Explanation · SHAP · load-forecast-nnprediction_id linked

    Sunday 29 January 2017, 17:00 · temperature 61.74

    Temperature = 61.74+24.3
    Load −24h = 73.3 MW+3.7
    Wind = 4.9 m/s−2.9
    Weekend = yes−2.6
    Load −168h = 72.1 MW+2.3
    Hour = 17:00−1.2
    Humidity = 54%−1.1
    Solar = 102 W/m²0.0

    average 65.8 MWforecast 88.2 MW

  8. 08Day 16

    The fix

    With the temperature in Celsius, the same hour's forecast falls from 88.2 MW to 69.5 MW; the actual load was 66.4 MW. Fix the feed and re-score the 300 hours: they pass the same check, max PSI 0.05. When the actual loads arrive, Kaimei's performance check sizes the damage: mean absolute error 28.3 MW on those hours, up from 1.9 MW.

    Same hour, temperature in Celsiusload-forecast-nn
    As logged61.74Forecast 88.2 MW
    In Celsius16.5 °CForecast 69.5 MW

    Tick on each bar: the actual load that hour, 66.4 MW

    1. Fix the feed
    2. Re-score the 300 forecasts it affected
    3. Keep the drift check on a schedule
    client.analyses.create(model="load-forecast-nn", type="drift")

Data: Power Consumption of Tetouan City (Salam & El Hibaoui, 2018), UCI Machine Learning Repository. The hours, both models and their forecasts are real, and the drift results, the insight, the explanation and the performance check are what the Kaimei API returned for this run. Staged: the 836 hours are the ones held out of training, a random fifth of January to June 2017, replayed out of order as sixteen days of traffic; the Fahrenheit conversion of the last 300 is ours; and when each check runs and how the bars grow are illustrative. The network has the same design as the one on the Energy page and is fitted on the other four fifths of those months, so the replayed hours are new to it.

See the energy example

In practice

Four pieces, one call each.

Drift needs at least 20 reference rows and 20 logged predictions; until then the insight says so.

Logged predictionsOne call per prediction, or batches. Inputs, outputs and latency, per model version.
Shadow modelsRun a candidate on live traffic beside the champion and compare before you switch.
Drift per inputPSI against the reference rows. Stable below 0.1, significant above 0.25, flagged within a batch.
VersionsShip v2 under the same model, keep v1's history, and compare their insights side by side.

Coming soon

Coming soonScheduled checksDrift and performance analyses run by Kaimei on the schedule you set. Call them from your own scheduler today.
Coming soonAlerts where your team worksMedium and high findings sent to email, Slack, Microsoft Teams, PagerDuty or a webhook.
Coming soonData quality checksMissing values, out-of-range values, unseen categories and unit changes caught as predictions are logged.
Coming soonAttribution driftAn alert when what the model relies on shifts, not only its inputs.
Coming soonFairness monitoringOutcome rates and accuracy compared across the groups you define, release by release.
Coming soonRetention schedulesLogged predictions and explanations deleted automatically once they pass the period you set.

Upload a model this afternoon.

The free plan covers 10 models, 500 explanations and 20 analyses a month. Your own model is one call away.