AI/ML Foundations

Loss Functions & Optimization

A loss function turns prediction mistakes into a number, and optimization changes model parameters to reduce that number. Training is the loop between model output, loss, gradient, and update.

Loss is what the model actually optimizes
Regression losses encode different tolerance for outliers
Classification losses reward probability assigned to the correct class
Gradient descent follows local slope
Learning rate controls step size
Objective and reported metric can differ

Training loop mental model

parameters = initialize()
for batch in training_data:
    predictions = model(batch.inputs, parameters)
    loss = loss_fn(predictions, batch.targets)
    gradients = derivative(loss, parameters)
    parameters = parameters - learning_rate * gradients

Frameworks automate the derivative, not the choice of objective.

Loss choice shapes behavior
Problem	Common loss	What it encourages
Regression	Mean squared error	Avoid large numeric misses
Robust regression	Mean absolute / Huber	Reduce outlier dominance
Binary classification	Binary cross-entropy	Calibrated probability for positive class
Multi-class classification	Softmax cross-entropy	Probability mass on correct class
Ranking / retrieval	Pairwise or contrastive losses	Relevant items closer/higher than irrelevant ones