Layers as Matrix Operations

A layer is many neurons reading the same inputs side by side. After this lesson your program runs a whole dense layer, for one input or a batch of them, from a flat list of weights.

M neurons, one table of weights

A dense (fully connected) layer with N inputs and M outputs owns an M x N weight matrix W and a bias vector b with M entries. Row j of W is the weight vector of neuron j, so output j is exactly the single-neuron formula from the last lesson:

z[j] = W[j][0]*x[0] + W[j][1]*x[1] + ... + W[j][N-1]*x[N-1] + b[j]
y[j] = sigmoid(z[j])

In matrix notation that is y = sigmoid(W x + b). Nothing new is computed; the matrix is only bookkeeping for M dot products.

Flat storage and a worked trace

Files and wire formats store the matrix as one flat list in row-major order: row 0 first, then row 1, and so on. Element W[j][i] lives at position j*N + i. Take N = 3, M = 2:

flat W = 0.5, 0.0, 0.25,  -1.0, 1.0, 0.5      b = 0.1, -0.2      x = 1, -1, 2

row 0 = 0.5, 0.0, 0.25   z0 = 0.5*1 + 0.0*(-1) + 0.25*2 + 0.1 = 1.1    sigmoid = 0.7503
row 1 = -1.0, 1.0, 0.5   z1 = -1*1 + 1*(-1) + 0.5*2 - 0.2    = -1.2   sigmoid = 0.2315

Both outputs use the whole input vector x, and each output has its own bias. The layer turned 3 numbers into 2.

Stacking layers

The output of one layer is the input of the next, so shapes must chain: an M x N layer feeds an M' by M layer. Feed [0.7503, 0.2315] into a second layer with weights [1, -1] and bias 0 and you get z = 0.5188, sigmoid = 0.6269. A deep network is just this repeated; the parameter count of one layer is M*N + M.

Batches

To process several samples at once, keep one row per sample and apply the same W and b to each row independently. Sample 2's output never depends on sample 1. Frameworks do this with a single matrix multiply, but the arithmetic is identical to looping over samples.

Common mistakes

• Reading W column-major. With N = 3 and M = 2, W[1][0] is the fourth number, not the second. A square test hides this; a non-square one exposes it.
• Swapping M and N when slicing the flat list.
• Adding one bias to every output instead of b[j] to output j.
• Applying the bias before finishing the dot product, or forgetting the activation on the last layer.

Your exercise

The input gives one or more INPUT lines, then the flat WEIGHTS, the BIAS, and a final M N line. The starter parses all of it and prints one output line per INPUT. Implement layer, returning the M activations for one input, with each dot product accumulated left to right and the bias added last.