What Neural Networks Solve

A neural network is a function whose numbers are found by fitting examples instead of being typed in by a programmer. In this lesson you will meet the smallest nonlinear piece of that function, the sigmoid, and write it so it never overflows.

A function you fit

Writing the rule "spam if it has too many links and exclamation marks" means choosing thresholds by hand. A network keeps the shape of such a rule and lets data pick the numbers:

output = nonlin( W3 x nonlin( W2 x nonlin( W1 x input + b1 ) + b2 ) + b3 )

Each W is a table of weights, each b a list of biases, each nonlin a fixed squashing function. Training means nudging every W and b until the outputs match labelled examples. Everything in this course, from the neuron to the transformer, is that one sentence made concrete.

Why the nonlinearity is not optional

Take two "layers" with no squashing between them: the first multiplies by 2, the second multiplies by 3. Feeding in 5 gives 10, then 30. But a single multiplication by 6 gives the same 30 for every input. Two linear steps always collapse into one, so a hundred of them are no more expressive than one. The squashing function between the steps is what stops the collapse and makes depth worth having.

The sigmoid

The classic squashing function maps any real number into the open interval 0 to 1:

sigmoid(x) = 1 / (1 + e^(-x))

Read it as a probability dial: large positive input means "yes", large negative means "no", zero means "undecided". It is symmetric, sigmoid(-x) = 1 - sigmoid(x), and it is steepest at 0, flattening out at both ends. That flat tail will matter in the backprop lesson.

The overflow trap

A 64-bit float tops out near 1.8e308, and e^710 already exceeds that. Feed the formula above x = -800 and it must compute e^800: some languages raise an overflow error, others quietly produce infinity and hope the division rescues them. The trap is purely numerical, because the true answer is simply "almost exactly 0".

The fix is algebra. Multiply top and bottom of the formula by e^x and you get a second, equal form:

x >= 0 :  1 / (1 + e^(-x))        (exponent is <= 0, safe)
x <  0 :  e^x / (1 + e^x)         (exponent is <  0, safe)

Pick the branch by the sign of x and the exponential only ever sees a non-positive number. For x = -800 the second form computes e^-800, which underflows harmlessly to 0, giving 0 / (1 + 0) = 0. Every real framework does this.

Common mistakes

• Printing 0.5 instead of 0.5000. The grader compares text, so always print exactly four decimals.
• Truncating instead of rounding: sigmoid(5) = 0.99330715..., which must print 0.9933, not 0.9932.
• Using only the textbook form and crashing, or printing nan/inf, on an input like -1000.
• Choking on blank lines or stray spaces around a number.

Your exercise

The starter already reads every line and prints your result with four decimals. Write the sigmoid function so it returns the right value for every finite input, including the enormous ones in the hidden tests. Decide the branch on the sign of the input before you call the exponential.