Classical Machine Learning

Decision Trees, Random Forests & Gradient Boosting

Tree models learn decision rules by splitting feature space. A single tree is readable but unstable; random forests reduce variance by averaging many trees; gradient boosting builds trees sequentially to fix the current ensemble's errors.

A decision tree is nested if/else logic learned from data
Single trees overfit easily
Random forests average many noisy trees
Gradient boosting learns corrections
Tree ensembles handle nonlinearity and interactions well
Interpretability decreases as performance rises

Tree-family trade-offs
Model	Main strength	Main weakness	Good default?
Decision tree	Readable rules	High variance / overfits	For explanation, not usually accuracy
Random forest	Robust with little tuning	Larger and less interpretable	Yes
Gradient boosting	Excellent tabular accuracy	Tuning-sensitive	Yes, with validation

From one tree to ensembles

Forests fight variance by averaging; boosting fights bias/residual error by sequencing.

Tree ensemble baseline

from sklearn.ensemble import RandomForestClassifier, HistGradientBoostingClassifier
from sklearn.model_selection import cross_validate

models = {
    "forest": RandomForestClassifier(n_estimators=300, min_samples_leaf=5),
    "boosting": HistGradientBoostingClassifier(max_leaf_nodes=31, learning_rate=0.05),
}

for name, model in models.items():
    result = cross_validate(model, X, y, cv=5, scoring=["f1", "roc_auc"])
    print(name, result["test_f1"].mean(), result["test_roc_auc"].mean())

Compare ensembles against a simple baseline and each other; don't pick by vibes.