Classical Machine Learning
Model Selection & Validation
Model selection chooses the model family, features, hyperparameters, and threshold using validation evidence. Validation is where you compare ideas without lying to yourself.
- Validation is part of the design loop
- Cross-validation reduces split luck
- Hyperparameter tuning can overfit validation
- Threshold tuning is model selection too
- Baselines prevent complexity theater
- The final test set should be boring
| Tool | Use when | Cost |
|---|---|---|
| Holdout validation | Enough data, quick iteration | One split can be lucky |
| K-fold cross-validation | Limited data / stable estimate | More training runs |
| Time-series validation | Historical prediction | Less randomization |
| Nested validation | Heavy tuning / honest comparison | Much more compute |
| Final test set | One-time audit | Must be protected |