Backpropagation

To minimize loss, we need to know how each weight affects loss. Compute via the chain rule applied recursively.

Forward pass: compute output + loss. Backward pass: compute gradients of loss w.r.t. each weight.

For a single neuron y = σ(z), z = w·x + b:

• dL/dy = (loss derivative w.r.t. output)
• dy/dz = σ'(z) = σ(z) * (1 - σ(z))
• dz/dw = x
• dz/db = 1
• dz/dx = w

By chain rule:

• dL/dw = dL/dy * dy/dz * dz/dw = (dL/dy) * σ'(z) * x
• dL/db = (dL/dy) * σ'(z)
• dL/dx = (dL/dy) * σ'(z) * w

For a layer (vectorized):

Backprop is mechanical: every operation has a forward and a backward rule.

Auto-differentiation:

• Build computation graph during forward pass.
• Walk in reverse, accumulating gradients.
• PyTorch / TensorFlow do this automatically.

Manual backprop:

• Easy for simple nets.
• Tedious + error-prone for complex.
• Educational: understand what frameworks do.

Famous gradient pitfalls:

• Vanishing: gradient → 0 as it flows back through deep nets.
  • Cause: sigmoid/tanh saturate, gradient ≤ 0.25.
  • Fix: ReLU (gradient = 1 in active region), batch norm, skip connections.
• Exploding: gradient grows unbounded.
  • Cause: large weights, recurrent loops.
  • Fix: gradient clipping, smaller learning rate.

Recompute vs cache:

• Cache forward activations: faster backprop, more memory.
• Recompute on backward: slower, less memory.
• Modern training uses gradient checkpointing: cache some, recompute others.