Deep Learning
Regularization for Deep Networks
Regularization reduces overfitting by making memorization harder or less attractive. In deep learning, the main tools are weight decay, dropout, data augmentation, early stopping, and capacity control.
- Weight decay discourages large weights
- Dropout prevents brittle co-adaptation
- Data augmentation encodes invariances
- Early stopping uses validation behavior
- Smaller models regularize directly
- Regularization follows diagnosis
| Technique | What it does | Best when |
|---|---|---|
| Weight decay | Penalizes large weights | Model overfits smoothly |
| Dropout | Randomly removes activations during training | Dense networks over-rely on paths |
| Data augmentation | Adds label-preserving variations | Input has known invariances |
| Early stopping | Stops before memorization dominates | Validation curve turns upward |
| Smaller model | Reduces capacity | Model is too flexible for data |