Scalar Autograd — How Backpropagation Really Works

Every deep learning framework — PyTorch, TensorFlow, the neural network you're building in this course — relies on the same trick to compute gradients: reverse-mode automatic differentiation. Instead of hand-deriving d(loss)/d(weight) for every parameter using the chain rule on paper, you build a small computation graph as the forward pass runs, then walk that graph backward once to get every gradient at once. This lesson strips the idea down to scalars so you can see the mechanism with nothing else in the way — everything your network does with matrices later is this same idea, just vectorized.

The computation graph

Every arithmetic operation (+, -, *, /, **) produces a new node that remembers:

1. its value (the forward-pass result),
2. its parents (the inputs that produced it), and
3. a local backward rule — how to push a gradient from this node onto its parents.

Building an expression like a * b + c doesn't compute a gradient yet — it just wires up a little DAG of Value nodes, each holding the local derivative rule for its own operation.

The chain rule, mechanically

Each operator's _backward closure implements exactly one line of the chain rule, in terms of the already-known gradient of its output:

• out = a + b → a.grad += out.grad, b.grad += out.grad (addition just passes the gradient straight through to both parents)
• out = a * b → a.grad += b.data * out.grad, b.grad += a.data * out.grad (product rule: the gradient w.r.t. one factor is the other factor, scaled by the incoming gradient)
• out = a / b → a.grad += (1/b.data) * out.grad, b.grad += (-a.data / b.data**2) * out.grad
• out = a ** k (k a literal) → a.grad += k * a.data**(k-1) * out.grad

Notice the pattern: **= (+=, not =) matters — if a node is used in more than one place in the expression (e.g. x*x, where x is the same node fed into * twice), gradients from every use must accumulate, not overwrite. This is the single most common autograd bug: forgetting to accumulate causes wrong gradients whenever a variable is reused.

The backward pass: topological order

To call every node's _backward correctly, you must guarantee that by the time you call a node's _backward, every node that consumes its output has already run its own _backward — otherwise out.grad wouldn't be fully accumulated yet. That means: build a topological sort of the DAG (a DFS post-order works), then walk it in reverse, seeding the final output's gradient to 1.0 (since d(out)/d(out) = 1) before starting.

What you're implementing

Your exercise parses a small expression string into this Value DAG (leaf variables from VAR, operators from the EXPR line), runs the backward pass once per EXPR, and answers GRAD <name> queries by reading off .grad on the corresponding leaf. This exact mechanism — build-graph-forward, then topo-sort-and-walk-backward-once — is precisely what a neural network's training step does at every iteration, just with matrices of numbers standing in for scalars and millions of nodes instead of a handful.