Tying the question to a decision
The first step is a decision, not a model: which decision, how often, made by whom, and how a prediction would change it. A prediction that changes no decision stays on the shelf however accurate it is.
- The decision owner and cadence are written first
- The threshold at which prediction becomes action is set
- The current decision method becomes the baseline
- Cost of being wrong is calculated in both directions
Data preparation and leakage
Most of a model's quality is decided during data preparation. The most common error is letting information unavailable at prediction time reach the training data; that leakage produces a model perfect in the lab and useless in production.
- Every feature is verified to exist at prediction time
- Time-based splitting replaces random splitting
- Missing-data patterns are examined as their own signal
- Data source and version are recorded
Validation and explainability
An enterprise decision invites the question of why the model said what it said. Feature contributions are reported, edge cases are reviewed, and the areas where the model is unreliable are marked openly.
- Feature contributions are presented in the decision owner's terms
- Segments where the model is weak are documented
- Risk of discriminatory outcomes is tested at data level
- The validation set represents the production distribution
Production and post-deployment monitoring
A model begins ageing the moment it goes live. As the input distribution shifts, accuracy declines quietly, so monitoring covers data and prediction distributions rather than only system health.
- Input drift is monitored against a threshold
- Prediction distribution is compared with actual outcomes
- Retraining conditions are written in advance
- Model version and its data are retained together