RNN Cell & Sequence Rollout

Dense, convolutional and pooling layers see a fixed-size input and forget it immediately. A recurrent network reads a sequence one element at a time while carrying a hidden state forward, so each step can depend on everything before it. After this lesson your program runs a vanilla RNN over a sequence and prints its hidden state at every step.

The cell

At step t the cell combines the new input x_t with the previous hidden state h_(t-1):

h_t = tanh( Wxh * x_t + Whh * h_(t-1) + b )

• Wxh has shape H by D and projects the D-dimensional input into the H-dimensional hidden space.
• Whh has shape H by H and projects the previous state into the same space. This recurrent connection is the memory.
• b is a bias of length H.
• tanh squashes every component into -1..1, which keeps the state from growing without bound.

Both matrices are the same at every step, which is why the network handles sequences of any length with a fixed number of parameters.

Matrix times vector, spelled out

Row i of the product is the dot product of row i of the matrix with the vector:

(M * v)[i] = M[i][0]*v[0] + M[i][1]*v[1] + ...

Matrices arrive row by row, so the first line of numbers is row 0. Reading them column by column is the classic transposition bug.

A worked trace

Take D = 1, H = 2, Wxh = [[1], [-1]], Whh = [[0.5, 0], [0, 0.5]], b = [0, 0], start with h = [0, 0], and feed the input 1 twice.

step 1  Wxh*x = [1, -1]      Whh*h = [0, 0]              h = tanh([1, -1])       = [ 0.7616, -0.7616]
step 2  Wxh*x = [1, -1]      Whh*h = [0.3808, -0.3808]   h = tanh([1.3808, -1.3808]) = [ 0.8811, -0.8811]

The same input produced two different states: the second step has "seen" the first. That difference is the whole point of recurrence.

Rollout

Rolling out a sequence means applying the same cell repeatedly, feeding each new state back in (pseudocode):

h = h0                       (zeros unless told otherwise)
for each input x in the sequence:
    h = cell(x, h)
    record h

The final state is a compressed summary of the whole sequence and is what a classifier head reads in tasks like sentiment analysis. The list of every state is what a sequence-to-sequence tagger reads.

Why plain RNNs struggle

Backpropagation through time multiplies the gradient by the cell's Jacobian at every step. If that factor is below 1 the gradient from 50 steps ago has shrunk by something like 0.9 to the 50th, about 0.005, so early inputs barely influence learning; above 1 it explodes. Gradient clipping tames the explosion. LSTM and GRU cells fix the fading with gates that let information (and gradient) pass through a step nearly unchanged. Transformers, next, avoid the chain entirely.

Common mistakes

• Transposing Wxh or Whh, which only shows when H and D differ.
• Starting every step from zeros instead of the previous state.
• Forgetting the bias, or adding it twice.
• Applying tanh to only one of the two matrix products.
• Resetting h between steps of the same sequence.

Your exercise

Commands describe the cell, then feed inputs. The starter parses everything and keeps the hidden state and the history. Implement matvec and rnn_step.