Deep Learning

Neural Network Basics

A neural network is a stack of parameterized transformations. Each layer converts one representation into another, and training adjusts the parameters so the final representation solves the task.

A neuron computes weighted input plus nonlinearity
Hidden layers learn intermediate representations
Depth and width increase capacity
Neural nets are differentiable programs
Architecture encodes assumptions
The training data distribution is part of the model

Feed-forward network

Training updates the layer parameters so output loss decreases.

Minimal layer stack

x = input
x = linear_1(x)
x = relu(x)
x = linear_2(x)
x = relu(x)
y = output_layer(x)
loss = loss_fn(y, target)

Real models add normalization, residuals, attention, convolutions, dropout, and batching, but the core pattern remains transformations plus loss.

Core terms
Term	Meaning
Parameter	Learned value, such as a weight or bias
Activation	Nonlinear function between layers
Layer	Parameterized transformation
Forward pass	Compute prediction from input
Loss	Numeric measure of mistake
Backward pass	Compute gradients for parameters