Production ML & MLOps
Training-Serving Skew
Training-serving skew happens when the data or transformations used during training differ from what the model receives in production. It is one of the most common reasons strong offline metrics collapse after launch.
- Feature computation must match across environments
- Prediction-time availability is the real constraint
- Default values are behavior
- Vocabulary and schema drift create silent bugs
- Skew should be tested and monitored directly
| Skew type | Example | Impact |
|---|---|---|
| Time leakage | Feature includes events after prediction time | Offline score too optimistic |
| Preprocessing mismatch | Training lowercases text, serving does not | Different tokens/features |
| Missing-value mismatch | Training drops nulls, serving fills zero | Unexpected production behavior |
| Category drift | New plan type unseen in training | Bad encodings or fallbacks |
| Unit change | Milliseconds become seconds upstream | Predictions become nonsense |