Designs a production drift-monitoring plan for ML systems - statistical tests per feature type, PSI and KS alert thresholds, monitoring cadence, and evidence-based retraining triggers - including a runnable PSI calculator. Use when someone asks "how do I know if my model is drifting", "set up drift monitoring for this model", "should we retrain", or is investigating unexplained model performance degradation. Do NOT use for judging whether a model is good enough to ship in the first place - use model-evaluation-report instead; for tracking training runs use experiment-tracking; for documenting the model for consumers use model-card-writer.
Click to play with sound.
---
name: Data Drift Monitor
description: Designs a production drift-monitoring plan for ML systems - statistical tests per feature type, PSI and KS alert thresholds, monitoring cadence, and evidence-based retraining triggers - including a runnable PSI calculator. Use when someone asks "how do I know if my model is drifting", "set up drift monitoring for this model", "should we retrain", or is investigating unexplained model performance degradation. Do NOT use for judging whether a model is good enough to ship in the first place - use model-evaluation-report instead; for tracking training runs use experiment-tracking; for documenting the model for consumers use model-card-writer.
---
# Data Drift Monitor
Models trained on historical data degrade when the world changes. Drift monitoring is the early-warning system that distinguishes a model aging gracefully from one silently failing - and without thresholds tied to actions, it degenerates into dashboards nobody reads and alerts everybody mutes. This skill produces a monitoring plan where every alert has a named owner and a defined next step.
## Operating procedure
Work in order: drift types (Step 2) determine which tests apply (Step 3), and thresholds (Step 5) are meaningless until cadence and coverage (Step 4) define what is measured.
### Step 1: Gather inputs
1. The model's top features by importance (at minimum the top 10) and their types (continuous vs categorical).
2. Prediction volume and frequency - this sets monitoring cadence.
3. Ground-truth label lag: how long after a prediction the true label arrives. If unknown, label the estimate a guess.
4. The primary business metric the model serves and its current 30-day baseline.
5. Retraining cost in engineering time and compute, so triggers can be calibrated against it.
### Step 2: Classify what can drift
Not all drift requires the same response.
… install to load the full skill