A model is validated, approved, and deployed. Accuracy looks good on the held-out set. The project is declared complete and the team moves on.
Eighteen months later someone notices the predictions have not been useful for a while. Not obviously wrong — just consistently a bit off, in a direction nobody was watching. The model degraded slowly enough that no single day looked like a failure.
Models do not break. They decay. Decay is much harder to notice than breakage.
Two kinds of drift
Covariate shift is when the input distribution changes. Your customer mix moves upmarket, a new region comes online, a product launch changes traffic composition. The relationship the model learned still holds; it is just being asked about a population it saw little of in training.
Concept drift is more fundamental — the relationship itself changes. Behaviour that predicted churn last year no longer does, because the market moved or a competitor changed the landscape. The model is answering a question that no longer has the same answer.
Both degrade performance. They need different responses, which is why distinguishing them is worth the effort. Covariate shift often resolves with retraining on recent data. Concept drift may mean the feature set itself needs rethinking.
Monitor inputs, outputs, and outcomes
Production monitoring typically compares a reference window — usually training or validation data — against a current production window, calculating measures like Population Stability Index or Kullback-Leibler divergence per feature.
Three layers are worth instrumenting. Input drift catches changes in feature distributions, and it is the earliest available signal. Prediction drift catches shifts in output distribution — if your churn model suddenly flags twice as many accounts, something changed. Outcome monitoring compares predictions against actual labels once they arrive, and it is the only measure of what genuinely matters.
Outcome monitoring is the most valuable and the most delayed. If churn resolves over ninety days, you learn about a degradation a quarter after it began. That lag is exactly why the leading indicators matter.
Set thresholds before you need them
The worst time to decide what counts as unacceptable drift is during an incident, when there is pressure to conclude that the current number is probably fine.
Agree in advance: at what PSI do we investigate, at what level do we retrain, and what performance drop triggers a rollback. Write it down while nobody is under pressure. It converts a judgement call into a procedure.
Retraining is a pipeline, not a project
If retraining requires a data scientist to reconstruct a notebook from months ago, it will not happen at the cadence it should. The retraining path needs to be automated infrastructure: pull recent data, validate it, train, evaluate against the incumbent, and promote only on improvement.
That last step matters. Automated retraining without an evaluation gate is a mechanism for automatically deploying worse models. The gate is what makes the automation safe.
The cost of not knowing
Poor data quality already costs organisations around $12.9 million a year on average. A degraded model sits inside that number in a particularly awkward way, because it produces confident outputs that people have been told to trust.
A model that is obviously broken gets ignored, which limits the damage. A model that is quietly 15% worse than it was keeps being used exactly as before. That is the expensive case.
What to build alongside every model
Feature distribution monitoring with agreed thresholds. Prediction distribution tracking. Outcome measurement wherever labels eventually arrive. A documented retraining trigger and an automated path to execute it. A rollback procedure someone has actually tested.
None of this is research work. It is the operational layer that separates a deployed model from a maintained one, and it is the part that most often gets deferred until the first quiet failure makes the case for it.