AI/ML Foundations

Training, Validation & Test Sets

A model learns from the training split, is tuned against the validation split, and is judged on a final held-out test split. The point is not to reward memorization, but to estimate future performance.

Training data fits model parameters
Validation data guides choices
Test data is the final audit
Splits must match production reality
Cross-validation helps when data is limited

Split discipline

The test set validates the chosen process; it should not help choose it.

Common splitting strategies
Strategy	Use when	Avoids
Random row split	IID examples with no grouping/time issues	Basic train/test contamination
Time-based split	Forecasting, event streams, historical prediction	Training on the future
Group/user split	Multiple rows per user/account/document	Same entity in train and test
Stratified split	Classification with class imbalance	Missing rare classes in a split