A Single Neuron

Lesson 0.0 said a neural network is matrix multiplies plus nonlinearities. Before stacking matrices, shrink the whole idea to its atom — one neuron:

y = activation(w · x + b)

• x = [x1, ..., xn] — inputs (features).
• w = [w1, ..., wn] — learned weights, one per input.
• b — learned bias.
• w · x = w1*x1 + w2*x2 + ... + wn*xn — the dot product: a weighted vote.

Each weight says how strongly (and in which direction) its feature pushes the neuron toward firing. The bias sets the firing threshold.

A concrete spam detector. Features: x = [exclamation marks, ALL-CAPS words, links]. Say training learned w = [0.8, 0.5, 1.2], b = -2.0, and an email has x = [3, 2, 1]:

z = 0.8*3 + 0.5*2 + 1.2*1 - 2.0 = 2.6
sigmoid(2.6) ≈ 0.93        → "93% spam"

The bias is doing real work: on an all-zero input the neuron outputs sigmoid(-2.0) ≈ 0.12 — a quiet default instead of the coin-flip sigmoid(0) = 0.5. Without a bias, every decision boundary is forced through the origin.

In pure Python — the same style this course's graded exercises use, no NumPy required:

(NumPy spells the middle line np.dot(w, x) + b — identical math, vectorized; the next lesson uses that form.)

The idiom: compute the pre-activation z once and keep it. Backpropagation (lesson 1.1) needs it — the sigmoid derivative is sigmoid(z) * (1 - sigmoid(z)) — so every serious implementation caches z during the forward pass.

The trap: math.exp overflows. sigmoid(-800) calls math.exp(800), which exceeds float range (the ceiling is ≈ 709.78) and raises OverflowError instead of quietly returning 0. Real implementations branch on the sign of z, or use the max-subtract trick you'll meet with softmax in Chapter 4.

Why the nonlinearity is load-bearing. Delete activation and a neuron is a linear function. Stack ten layers of linear neurons and you've composed ten linear maps — which collapses to a single matrix. A 100-layer "network" without nonlinearities has exactly the power of one-layer linear regression. The activation is the only thing that makes depth mean anything.

Geometry. w · x + b = 0 is a hyperplane. One neuron can only draw a straight boundary — which is why a single neuron famously cannot compute XOR (Minsky & Papert, 1969): no line separates (0,1) and (1,0) from (0,0) and (1,1). Two hidden neurons draw two lines; an output neuron combines the regions. That's the entire argument for hidden layers, in miniature.

The activation menu (each returns later in the course):

• Sigmoid 1/(1+e^-z): outputs 0..1 — read as a probability. Derivative peaks at 0.25 (at z = 0) and is ≈ 0 for |z| > 4: saturation, the root of vanishing gradients (lesson 1.1).
• Tanh: outputs -1..1, zero-centered for better gradient flow. The RNN cell you build in Chapter 6 is exactly a layer of tanh neurons.
• ReLU max(0, z): gradient exactly 1 while active — cheap, and it doesn't saturate. Risk: a neuron whose z stays negative goes "dead"; Leaky ReLU max(0.01z, z) keeps a trickle flowing.
• Softmax: not per-neuron — it turns a whole output layer into a probability distribution over classes (e^zi / Σ e^zj). You implement it as a graded exercise in Chapter 4.

One neuron = one dot product plus one nonlinearity. The next lesson stacks M of them side by side so a whole layer becomes a single W @ x + b — the same arithmetic, vectorized.

Your exercise

No code grader on this page — the quizzes below are the checkpoint. But hand-trace these two numbers now, because the course's real graders check them later, to exactly four decimals:

• sigmoid(2) = 0.8808. Softmax over two logits is the sigmoid of their gap, so the Chapter 4 softmax grader expects the exact stdout 0.8808,0.1192 for logits 2.0,0.0.
• tanh(0.5) = 0.4621. The Chapter 6 RNN-cell grader pushes STEP 0.5,-0.5 through identity weights and expects exactly 0.4621,-0.4621.

The mistake those graders catch most often is formatting, not math: print(round(p, 4)) drops trailing zeros and prints 0.5 where a hidden test (logits 1000.0,1000.0) demands 0.5000,0.5000. Format with f"{p:.4f}" — and note that same hidden test also detonates a naive exp(1000): exactly the overflow trap above.