Training Loop

Everything so far was one piece: a forward pass, a loss, a gradient, an update. A training loop is those pieces run in a circle until the numbers are good. After this lesson your program trains a classifier from scratch and prints the learning curve a real framework would print.

One epoch, five steps

An epoch is one pass over the training data. In each epoch a full-batch loop does the same five things:

1. Forward: run every example through the model.
2. Loss: average the per-example loss.
3. Metric: count how many predictions were right.
4. Backward: average the gradients over the examples.
5. Update: move each parameter against its gradient, scaled by the learning rate.

The loss and metric reported for an epoch come from the forward pass before that epoch's update. That is also what PyTorch-style loops log, and it means epoch 1 reports the untrained model.

The model: logistic regression

One sigmoid neuron a = sigmoid(w . x + b) with binary cross-entropy loss is a complete classifier. The gradient simplifies beautifully. For one example the derivative of the loss with respect to z is just a - y, the prediction error. Then dL/dw_j = (a - y) * x_j and dL/db = a - y, each averaged over the examples. Start from w = 0 and b = 0. A prediction counts as class 1 when a >= 0.5.

A worked trace

Two examples with one feature: x = 1 with label 1, and x = -1 with label 0. Use lr = 1.

epoch 1  w = 0, b = 0       a = 0.5 and 0.5      loss = 0.6931
         both predicted class 1 (0.5 >= 0.5)      accuracy = 0.5
         errors a - y = -0.5 and +0.5
         dw = (-0.5*1 + 0.5*(-1)) / 2 = -0.5      db = 0
         update: w = 0 + 0.5 = 0.5, b = 0
epoch 2  w = 0.5            a = 0.6225 and 0.3775 loss = 0.4741   accuracy = 1.0
         dw = -0.3775 (average)                    w becomes 0.8775
epoch 3  w = 0.8775                                loss = 0.3477   accuracy = 1.0

Loss falls every epoch; accuracy jumps from 0.5 to 1.0 in one step and then stays put. That is why you watch both. Accuracy is a coarse, step-shaped number; loss still tells you the model is getting more confident.

Reading the curves

• Loss falling smoothly: healthy.
• Loss jumping up or going nan: learning rate too high.
• Loss barely moving: learning rate too low, or the model cannot represent the pattern. A single neuron cannot learn XOR, so its accuracy there never reaches 1.0 no matter how long you train.
• Train loss falling while loss on held-out data rises: overfitting. This is why data is split into train (fits the weights), validation (picks hyperparameters, stops training early) and test (touched once at the end).

Common mistakes

• Updating the weights inside the per-example loop. That is stochastic descent; this exercise averages first.
• Dividing the gradient by the wrong count, or not at all.
• Logging the loss after the update, which shifts the whole curve by one epoch.
• Resetting the weights at the start of each epoch.
• Treating a prediction of exactly 0.5 as class 0. The first epoch depends on it.

Your exercise

The input gives the learning rate, the number of epochs and the labelled examples. The starter parses them, calls your train, and prints one line per epoch plus the final parameters. Implement train: it returns the loss and accuracy of every epoch and the final weights and bias. Use full-batch updates and clip probabilities into [1e-12, 1 - 1e-12] inside the logarithms.