Scalar Autograd: How Backpropagation Really Works

Earlier you derived a neuron's gradient by hand. That does not scale to a network with millions of operations, so every framework records the arithmetic as it runs and replays it backwards. This lesson builds that machine for single numbers. After it, your program differentiates any expression made of + - * / and integer powers with one backward sweep.

The graph is the program you ran

Every operation produces a node that remembers three things: its value, which nodes produced its inputs (its parents), and which operation made it. Evaluating (a + b) * a with a = 2 and b = 3 builds:

n1 = a + b      value 5     parents a, b
n2 = n1 * a     value 10    parents n1, a

Nothing is differentiated yet; the forward pass only leaves a trail.

Local rules

Each operation knows how to hand a gradient to its parents. If g is the gradient arriving at the output node (how much the final result changes per unit change of this node):

Backward, with a trace

Seed the final node with gradient 1, since a change of one unit in the result changes the result by one unit. Then visit nodes from the newest to the oldest. For n2 = n1 * a (so n1 = 5, a = 2):

n2.grad = 1
multiply:  n1 gets a*1 = 2          a gets n1*1 = 5
add (n1):  a gets 2                 b gets 2
a.grad = 5 + 2 = 7        b.grad = 2

Check with calculus: (a+b)*a = a^2 + a*b, so the slope in a is 2a + b = 7 and in b is a = 2.

Accumulate, never overwrite

Notice a received two contributions, 5 and 2, and the answer is their sum. A node used in several places collects a gradient from each use. Writing = instead of += keeps only the last contribution, which is the classic autograd bug and gives wrong answers whenever a variable is reused (x*x, a*b + a).

Order is everything

A node's gradient must be complete before it is passed on. It is complete when every node that consumed it has already pushed its share. That requires visiting consumers before producers: a reverse topological order. A depth-first search builds one. There is a shortcut worth noticing: a node is created only after its parents exist, so the creation order is already a topological order, and walking the list from newest to oldest is a valid backward sweep.

Common mistakes

• Using = where the rule says add the gradient to the parent.
• Forgetting to seed the root with 1, so every gradient is 0.
• Walking the nodes oldest to newest.
• Mixing up the two parents in a / b, or forgetting to square b.
• Creating one node per mention of a variable. A variable named twice must be one shared node, or the contributions never meet.

Your exercise

You get a tiny language: VAR, EXPR, GRAD. The starter tokenizes and parses each EXPR into calls to add, sub, mul, div and power and holds the node arrays. Implement those five constructors (value and bookkeeping) and backward.