Loss Functions

How "wrong" is the network? Quantify with a loss function.

For regression (continuous output):

• MSE (Mean Squared Error): mean((y_pred - y_true)²).
• MAE (Mean Absolute Error): mean(|y_pred - y_true|).

For binary classification:

• Binary cross-entropy: -mean(y_true * log(y_pred) + (1-y_true) * log(1-y_pred)).

For multi-class:

• Categorical cross-entropy: -mean(sum(y_true * log(y_pred))).

For MNIST: 10-class classification.

• y_true: one-hot vector (e.g., [0,0,1,0,0,0,0,0,0,0] for digit 2).
• y_pred: softmax output (probabilities).
• Loss: cross-entropy.

Why cross-entropy + softmax?

• Combined gradient is clean: d_loss/d_logits = y_pred - y_true.
• Avoids saturation issues.

Loss landscape:

• Loss is a function of all weights.
• 175 billion parameters → 175-billion-dimensional surface.
• Find minimum via gradient descent.

Local vs global minimum:

• Loss surface has many local minima.
• For deep nets, most local minima are equally good (high-dim phenomenon).
• Saddle points more common than local minima in high dim.

Regularization: add term to loss to prevent overfitting:

• L2 (weight decay): + λ * sum(W²). Encourages small weights.
• L1: + λ * sum(|W|). Encourages sparsity.
• Dropout: randomly zero some neurons during training.

Class imbalance:

• 99% non-spam, 1% spam → naive model "always not spam" gets 99% accuracy but 0% recall.
• Weight loss by class frequency.
• Or: F1, AUC instead of accuracy.