Networks with memory

Feedforward layers and even convolutions process a fixed-size input in one shot — they have no notion of "what came before." Recurrent neural networks (RNNs) handle sequences by maintaining a hidden state that gets updated at every timestep, carrying information forward from everything seen so far.

The vanilla RNN cell

At each timestep t, the cell combines the current input x_t with the previous hidden state h_{t-1}, then squashes the result through a nonlinearity:

$h_t = \tanh(W_{xh}\, x_t + W_{hh}\, h_{t-1} + b)$

Breaking this down:

• W_xh (shape H × D) projects the current input into the hidden space.
• W_hh (shape H × H) projects the previous hidden state into the same space — this is the recurrent connection that gives the network memory.
• Both contributions are summed with a bias b, then passed through tanh to keep the hidden state bounded in [-1, 1] (which also helps prevent the hidden state from exploding across many timesteps).

Note that W_xh and W_hh are shared across every timestep — the same two matrices process every step of the sequence, which is what makes the network "recurrent" rather than just a stack of independent feedforward layers, and what lets it generalize to sequences of any length.

Matrix-vector products, spelled out

For a hidden size H and input size D, W_xh @ x_t is a matrix-vector product:

(W_xh @ x_t)[i] = Σ_j  W_xh[i][j] * x_t[j]      for i in 0..H

Same shape of computation for W_hh @ h_{t-1}, just square (H × H) instead of H × D. Implementing a small matvec(M, v) helper — row i of the output is the dot product of row i of M with v — keeps the cell logic itself clean and reads almost exactly like the math.

Rolling out a sequence

"Rollout" means applying the same cell repeatedly, once per input in the sequence, always feeding the previous step's output hidden state back in as the next step's h_{t-1}:

h = h0  (usually zeros)
for x_t in sequence:
    h = tanh(W_xh @ x_t + W_hh @ h + b)
    record(h)

The hidden state after the last timestep is a compressed summary of the entire sequence so far — this is what gets fed into a classifier head for tasks like sentiment analysis, or what an encoder-decoder architecture passes to its decoder.

Practical notes and edge cases

• Initial hidden state defaults to zero unless explicitly overridden — this is what "the network knows nothing yet" looks like at t=0.
• Row-major matrices: row 0 of a matrix input is literally the first line of numbers you read, row 1 the second, and so on — a transposition bug here (reading rows as columns) is one of the most common mistakes when hand-rolling matrix code.
• State persists across STEP calls — each STEP doesn't reset h; it continues from wherever the last step (or H0) left off, exactly like processing consecutive words in a sentence.
• Vanilla RNNs like this one are simple to reason about but suffer from vanishing/exploding gradients over long sequences — this is the motivation for the gated variants (LSTM, GRU) that later lessons in this course build on top of the same rollout pattern you're implementing here.