MNIST: A Working Network

MNIST is the "hello world" of NN: 28x28 grayscale digits, 60k train + 10k test.

Simple architecture:

Input: 784 (flattened)
  ↓
Dense(128) + ReLU
  ↓
Dense(64) + ReLU
  ↓
Dense(10) + Softmax
  ↓
Output: 10 (digit probabilities)

Parameters:

• Layer 1: 784*128 + 128 = 100,480
• Layer 2: 128*64 + 64 = 8,256
• Layer 3: 64*10 + 10 = 650
• Total: ~109k.

Hardware:

• CPU: ~1 minute per epoch for 60k samples, batch 64.
• GPU: ~5 seconds per epoch.

Convergence:

• After 1 epoch: ~93% test accuracy.
• After 10 epochs: ~97%.
• After 50 epochs: ~98%.

Improvements:

• Add CNN layers (Conv2d): better spatial feature learning. Get to 99%+.
• Data augmentation: shift, rotate slightly. More effective data.
• Dropout: prevents overfitting on small dataset.

CNN basic:

Why CNN > dense for images?

• Translation invariance: same filter applies everywhere.
• Parameter sharing: 9-pixel filter, not 784-input dense.
• Hierarchical features: edges → shapes → digits.

Modern variants for MNIST:

• ResNet-18: ~99.7% on MNIST. Overkill.
• LeNet-5 (1998, Yann LeCun): the classic ConvNet for MNIST — building on his 1989 zip-code ConvNet.

Beyond MNIST:

• CIFAR-10/100: 32x32 RGB classification. Harder.
• ImageNet: 1.2M images, 1000 classes. Where deep learning showed off in 2012.
• Hugging Face datasets: thousands of options.

Iterative refinement: get baseline working, then improve. Don't optimize before working.