Backpropagation

Training needs to know, for every weight, which way to move and how hard. Backpropagation is the bookkeeping that answers that with one forward and one backward sweep. After this lesson your program computes the exact gradient of a loss with respect to every weight and the bias of a neuron.

The question

The loss L depends on a weight w_i only through a chain of small steps. For one sigmoid neuron scored by squared error:

z = w1*x1 + ... + wn*xn + b        (pre-activation)
a = sigmoid(z)                     (output)
L = (a - y)^2                      (y is the target)

Each step has a simple local derivative:

dz/dw_i = x_i        dz/db = 1
da/dz   = a * (1 - a)
dL/da   = 2 * (a - y)

The chain rule says the effect of w_i on L is the product of the effects along the way:

dL/dw_i = dL/da * da/dz * dz/dw_i
dL/db   = dL/da * da/dz * 1

Compute delta = dL/da * da/dz once. Then every weight gradient is delta * x_i and the bias gradient is delta.

A worked trace

Take x = [2, -1], w = [0.5, 0.25], b = 0.1, y = 1.

forward   z = 0.5*2 + 0.25*(-1) + 0.1 = 0.85
          a = sigmoid(0.85) = 0.7006
backward  dL/da = 2*(0.7006 - 1)   = -0.5989
          da/dz = 0.7006*(1-0.7006) =  0.2098
          delta = -0.5989 * 0.2098  = -0.1256
          dL/dw1 = delta * 2    = -0.2513
          dL/dw2 = delta * (-1) =  0.1256
          dL/db  = delta        = -0.1256

Read the signs: w1 has a negative gradient, so raising it lowers the loss (the neuron's output is too low and x1 is positive). w2 has a positive gradient because its input is negative.

Check it by nudging

You can test any gradient without trusting your algebra. Raise w1 from 0.5 to 0.501 and rerun the forward pass: the loss moves from 0.089660 to 0.089409. The change over the nudge is -0.2510 per unit, which matches the analytic -0.2513. If the two disagree by more than a few parts in a thousand, the backward code is wrong. This is called a gradient check, and every autograd library is tested with it.

Why the forward values must be kept

The backward pass uses a and the inputs x from the forward pass. Real networks cache them layer by layer instead of recomputing. That memory cost is why deep training needs so much RAM.

Vanishing gradients, in numbers

a * (1 - a) is at most 0.25 (when a = 0.5) and falls toward 0 as the neuron saturates. A neuron with z = 30 has a within 1e-13 of 1, so even a badly wrong answer produces a gradient around 1e-13: it has stopped learning. Chain ten sigmoid layers and the gradient reaching the first one shrinks by up to 0.25 raised to the 10th power, roughly one millionth.

Common mistakes

• Dropping the 2 from dL/da. Every gradient comes out half the size.
• Forgetting the x_i factor, so all weights of one neuron share the same gradient.
• Multiplying the bias gradient by an input. The bias input is the constant 1.
• Applying the sigmoid derivative to z instead of using the already-computed a * (1 - a).

Your exercise

Each line gives one neuron and its target as x;w;b;y. The starter parses it and has the sigmoid ready. Implement grads, returning the gradient for each weight followed by the gradient for the bias.