Loss Functions

A loss function turns "how wrong is the network?" into a single number that training can push down. After this lesson your program scores a batch of predictions three ways, and you will see why the choice of loss changes what the network learns.

One number per batch

Training needs a scalar to minimise. Given predictions p[i] and targets t[i] for n examples, each loss compares them element by element, then averages. Dividing by n keeps the value comparable between batch sizes.

A worked trace

Predictions p = [0.8, 0.3, 0.6], targets t = [1, 0, 0]:

errors     p - t          -0.2     0.3     0.6
MSE        squares        0.04     0.09    0.36    mean = 0.49/3 = 0.1633
MAE        absolutes      0.2      0.3     0.6     mean = 1.1/3  = 0.3667
BCE        -ln(0.8)=0.2231, -ln(1-0.3)=0.3567, -ln(1-0.6)=0.9163   mean = 1.4961/3 = 0.4987

For a target of 1 only the t*ln(p) half of BCE is alive; for a target of 0 only (1-t)*ln(1-p) is. Each example pays the log of the probability the model gave to the right answer.

Why classifiers use cross-entropy, not MSE

Suppose the target is 0 and the model says p = 0.99, confidently wrong. MSE charges 0.99^2 = 0.98. BCE charges -ln(0.01) = 4.61, and as p approaches 1 that grows without bound. A confident mistake produces a huge loss and a huge gradient, which is exactly the push a classifier needs. MSE on a sigmoid output, by contrast, gives a vanishing gradient when the neuron is saturated and wrong, so learning stalls.

MAE grows linearly, so one outlier moves it less than MSE, but its gradient has constant size and does not shrink near the optimum.

The log(0) trap

ln(p) is minus infinity at p = 0. A network that outputs exactly 0 or 1 while being wrong would give an infinite loss and then nan gradients. The standard defence is to clip probabilities into [eps, 1 - eps] before taking logs. Use eps = 1e-12. With clipping, a certain-and-wrong prediction costs -ln(1e-12) = 27.6310, large but finite.

Common mistakes

• Summing instead of averaging, so the loss grows with the batch size.
• Forgetting to square (MSE) or to take the absolute value (MAE), which lets positive and negative errors cancel.
• Swapping the two BCE halves, so the loss rewards the wrong class.
• Not clipping, which prints inf or crashes on a hard 0 or 1 prediction.
• Returning a negative loss. BCE is a negated sum of logs of numbers at most 1.

Your exercise

The input has a PRED line, a TRUE line and an optional LOSS line naming mse, mae or bce (default mse). The starter parses them and prints your number with four decimals. Implement the three functions; for bce clip every prediction to [1e-12, 1 - 1e-12] before the logs.