Deep Learning

Backpropagation & Optimizers

Backpropagation computes how each parameter contributed to loss; optimizers use those gradients to update parameters. This is the engine that makes deep learning trainable.

Backpropagation is the chain rule applied to a computation graph
Gradients point uphill, so optimizers step downhill
Mini-batches trade accuracy for efficiency
Momentum smooths noisy updates
Adam is a strong practical default
Learning-rate schedules matter

Forward and backward pass

Training is a repeated feedback loop, not a one-time fit.

Training step sketch

pred = model(batch_x)
loss = loss_fn(pred, batch_y)
loss.backward()        # compute gradients
optimizer.step()       # update parameters
optimizer.zero_grad()  # clear old gradients

The API is small; the debugging surface is not.

Optimizer intuition
Optimizer	Intuition	Common use
SGD	Step opposite gradient	Simple baseline, often strong with schedules
Momentum	SGD with velocity	Smoother progress
Adam	Adaptive per-parameter steps	Default for many deep-learning tasks
AdamW	Adam with decoupled weight decay	Common for Transformers