Layers as Matrix Operations

Many neurons in parallel = a layer. Each neuron has its own weights.

A dense layer with N inputs and M outputs:

• Weights: M × N matrix.
• Bias: M-element vector.

output = activation(W @ x + b)

Where W @ x is matrix-vector multiplication: M outputs, each being a dot product.

Stacking layers: output of one = input of next.

For batches, vectorize: process N inputs at once.

• Input: N × 784 matrix.
• W is 128 × 784.
• W @ X.T → 128 × N (each column = output for one input).
• Or: X @ W.T → N × 128 (each row = output).

Convention varies. Here we'll use rows-as-samples.

Layer dimensions:

• More neurons per layer = more capacity but more parameters.
• More layers = deeper, can learn more complex functions, harder to train.

Modern practice:

• Hidden layers: 64-2048 neurons typical for small to medium models.
• Depth: 5-100+ layers.
• LLMs (GPT-3): 96 layers, 12288 neurons per layer, 175B parameters.

Skip connections (ResNet 2015):

• output = layer(x) + x.
• Solves vanishing gradients in deep nets.
• Made 100+ layer networks practical.