Platform · Monitoring
Watch every model, and know why when it moves.
Log the predictions your stack already makes, or let Kaimei host the model and predict for you. Either way, every logged prediction can be explained after the fact, and drift arrives with the input that caused it.
Monitoring · grid load forecasting
A pipeline bug shows up as drift.
Two weeks of logged forecasts with a shadow model, then a weather feed that starts sending temperature in Fahrenheit. Nothing errors. Follow the timeline to see Kaimei catch it in one batch, and an explanation name it.
-
01Days 1 to 14
Every prediction, logged
A neural network that forecasts a city's hourly electricity load is in production. The service that runs it logs each input and forecast to Kaimei; 536 hours of 2017 that the model never trained on stand in for two weeks of traffic. The band behind them is where the temperatures of its 3,340 reference hours sit.
-
02Days 1 to 14
A shadow alongside
A gradient-boosted model scores the same hours in shadow: logged, never acted on. The network's forecasts average 70.3 MW, the shadow's 70.2 MW. They are more than 5 MW apart on 6 of the 536 hours, which is where switching models would change what an operator sees. Scored against the actual load, the network is out by 1.9 MW on average and the shadow by 1.6 MW.
-
03Day 14
A clean check
A drift analysis compares the latest logged inputs, up to 500, with the reference rows, one input at a time, using the population stability index. Below 0.1 is stable. Every input is: max PSI 0.05.
Drift analysisday 14Every input stable
Recent inputs match the reference population (max PSI 0.05).
-
04Days 15 and 16
A new feed goes live
A new weather feed sends temperature in Fahrenheit, not Celsius: the first hour through it was 16.5 °C and is logged at 61.74. Nothing errors. All 300 of its hours arrive hotter than any hour in the reference data, and every forecast comes out higher: 96.5 MW on average as logged, against 68.6 MW with the temperature in Celsius.
-
05Day 16
Drift on one input
The same check, two days later: 3,340 reference rows against the latest 500 logged forecasts. Temperature's PSI is 1.548, far past the 0.25 line for a significant shift. Every other input is stable. The forecasts themselves have shifted too: PSI 0.791.
Drift analysisday 163,340 reference rows vs the latest 500 logged predictions
1.548TemperatureHumidity0.034Hour0.031Load −24h0.023Solar0.015Load −168h0.011Weekend0.008Wind0.0030.791PredictionsEvery other input 0.034 or less: stable
stable < 0.1moderatesignificant > 0.25
-
06Day 16
Flagged at high severity
Kaimei records an Input drift insight at high severity, and the project page lists it under Needs attention. It says which input moved: temperature. Drift needs no outcomes, so the flag does not wait for the actual load to come in.
Needs attention
Input drift · City load · neural networkRecent traffic has significant drift (PSI 1.55)… HighInsightins_622c0b67d6b1d8196ee5
Input drift
DriftHighscore 1.548
Recent traffic has significant drift (PSI 1.55). Shifted inputs: temperature.
3,340 reference vs 500 recent rowspredictions PSI 0.791 · significant
-
07Day 16
Named by an explanation
Explain the first forecast logged from the new feed: a Sunday in January at 17:00, forecast 88.2 MW. The reading of 61.74 is the largest push by far, +24.3 MW; no other input moves the forecast by more than 3.7 MW. The explanation is linked to the logged prediction by its id.
Explanation · SHAPprediction_id linkedSunday 29 January 2017, 17:00 · temperature 61.74
Temperature = 61.74+24.3Load −24h = 73.3 MW+3.7Wind = 4.9 m/s−2.9Weekend = yes−2.6Load −168h = 72.1 MW+2.3Hour = 17:00−1.2Humidity = 54%−1.1Solar = 102 W/m²0.0average 65.8 MWforecast 88.2 MW
-
08Day 16
The fix
With the temperature in Celsius, the same hour's forecast falls from 88.2 MW to 69.5 MW; the actual load was 66.4 MW. Fix the feed and re-score the 300 hours: they pass the same check, max PSI 0.05. When the actual loads arrive, Kaimei's performance check sizes the damage: mean absolute error 28.3 MW on those hours, up from 1.9 MW.
Same hour, temperature in Celsiusload-forecast-nnAs logged61.74Forecast 88.2 MWIn Celsius16.5 °CForecast 69.5 MWTick on each bar: the actual load that hour, 66.4 MW
- Fix the feed
- Re-score the 300 forecasts it affected
- Keep the drift check on a schedule
client.analyses.create(model="load-forecast-nn", type="drift")
Data: Power Consumption of Tetouan City (Salam & El Hibaoui, 2018), UCI Machine Learning Repository. The hours, both models and their forecasts are real, and the drift results, the insight, the explanation and the performance check are what the Kaimei API returned for this run. Staged: the 836 hours are the ones held out of training, a random fifth of January to June 2017, replayed out of order as sixteen days of traffic; the Fahrenheit conversion of the last 300 is ours; and when each check runs and how the bars grow are illustrative. The network has the same design as the one on the Energy page and is fitted on the other four fifths of those months, so the replayed hours are new to it.
See the energy exampleIn practice
Four pieces, one call each.
Drift needs at least 20 reference rows and 20 logged predictions; until then the insight says so.
Coming soon
Upload a model this afternoon.
The free plan covers 10 models, 500 explanations and 20 analyses a month. Your own model is one call away.