Training Loop

Putting it together:

Standard MNIST setup:

• 60k train, 10k test images.
• Each image: 28x28 = 784 pixels (flatten or 2D for CNN).
• Labels: 10 classes (digits).
• Simple feedforward NN: 96-99% test accuracy.
• CNN: 99.5%+.

Train/val/test split:

• Train: optimize weights.
• Validation: tune hyperparameters, early stopping.
• Test: final unbiased evaluation. Touch ONCE.

Overfitting:

• Train accuracy 100%, test accuracy 80% → overfitting.
• Model memorized train data, didn't generalize.
• Mitigations: more data, simpler model, regularization, dropout, data augmentation.

Underfitting:

• Train accuracy 70% → underfitting.
• Model too simple for the task.
• Add layers, neurons, train longer.

Learning curves:

• Plot train + val loss over epochs.
• Train decreasing, val increasing → overfitting starting.
• Both flat → underfitting.

Hyperparameters to tune:

• Learning rate.
• Batch size.
• Number of layers / neurons per layer.
• Activation function.
• Optimizer.
• Regularization strength.
• Dropout rate.

Use grid search, random search, or Bayesian optimization (Optuna) to explore.

Loss curves are diagnostic: smooth decreasing = good. Spikes = lr too high. Flat = lr too low.