MNIST: A Working Network

MNIST is the "hello world" of neural networks: 70,000 pictures of handwritten digits. A network never sees a picture, though; it sees a vector of numbers and a target vector. After this lesson your program turns a labelled image into exactly that.

The data

Each MNIST image is 28 by 28 grayscale pixels. A pixel is an integer from 0 (blank paper) to 255 (full ink). Each image carries a label from 0 to 9. The standard split is 60,000 training images and 10,000 test images, and the test images are for the final score only.

Flatten the picture

A dense layer wants one vector. Flattening reads the rows top to bottom and joins them. A 2 by 2 image with rows [51, 204] and [10, 0] becomes [51, 204, 10, 0]; a 28 by 28 image becomes 784 numbers. Row-major order matters only in that you must use the same order everywhere.

Scale the pixels

Raw 0-255 inputs make the first layer's pre-activations huge and unevenly sized. Two steps fix that:

1. Divide by 255 so every value lies in 0..1.
2. Standardize with the training set's mean and standard deviation: (value - 0.1307) / 0.3081. These are the standard MNIST constants, measured once on the 60,000 training images.

For the pixel 51: 51 / 255 = 0.2, then (0.2 - 0.1307) / 0.3081 = 0.2249. Blank paper (0) maps to -0.4242 and full ink (255) to 2.8215. Use the training statistics even on test images; using the test set's own statistics leaks information about it.

One-hot labels

The label 3 is not "a bit more than 2". Feeding the integer would tell the network that 7 sits between 6 and 8. Instead the target is a vector with one entry per class, 1 at the class and 0 elsewhere. With 5 classes, label 3 becomes [0, 0, 0, 1, 0]. The network's final layer then produces 5 probabilities, and the loss compares the two vectors.

The network and what to expect

A first network for MNIST is 784 inputs, a hidden layer of 128 with ReLU, one of 64 with ReLU, and 10 softmax outputs. Its parameter count is 784128 + 128 = 100,480, then 12864 + 64 = 8,256, then 64*10 + 10 = 650, about 109 thousand in total. Trained with mini-batches, a network like this reaches roughly 97% accuracy on the test set in a few epochs. A convolutional network, which reuses one small filter across the image, passes 99%. The rest of the course builds the pieces: autograd, optimizers, normalization, convolution.

Common mistakes

• Dividing the integer pixel by the integer 255. In several languages that is integer division and every pixel except 255 becomes 0. Convert to floating point first.
• Dividing by 256 instead of 255, so full ink never reaches 1.0.
• Writing the one-hot vector one position off, or with K - 1 entries.
• Standardizing before scaling to 0..1, which applies a 0..1 mean to 0..255 values.

Your exercise

The first line gives the number of classes K. Every other line is label image, where the image rows are joined by | and the pixels in a row by commas. The starter flattens each image and prints your result. Implement one_hot and standardize.