What Neural Networks Solve

A neural network is a function f(x) → y learned from examples. Given enough labeled data, it can:

• Classify images: f(pixels) → "cat".
• Translate language: f(English sentence) → French sentence.
• Predict prices: f(features) → price.
• Generate text, art, audio.

The function itself is a series of matrix multiplies + nonlinearities:

output = nonlin(W3 @ nonlin(W2 @ nonlin(W1 @ x + b1) + b2) + b3)

Each Wn is a learned matrix; bn is a learned bias. Nonlinearities (ReLU, sigmoid, tanh) make the function flexible.

Why "neural"?

• Inspired loosely by biological neurons.
• Modern NNs share little with brains beyond name.

Why does it work?

• Universal approximation theorem (1989): a 1-hidden-layer net with enough neurons can approximate any continuous function.
• Optimization (gradient descent) finds good parameters.
• Lots of compute + data → empirically works.

History:

• 1958: perceptron.
• 1986: backpropagation popularized.
• 2012: AlexNet wins ImageNet (deep CNN).
• 2017: Transformer paper.
• 2022+: GPT-3/4, Claude, Llama, etc.

Real frameworks:

• PyTorch: research-favored.
• TensorFlow / Keras: production-favored.
• JAX: functional, JIT-compiled.
• MLX (Apple): native to Apple Silicon.

We'll build a tiny NN with manual backprop: NumPy in the walkthroughs, and pure-Python stdin/stdout kernels in the graded exercises. By the end you understand what pytorch.nn does under the hood.

Reference: 3blue1brown YouTube series, micrograd by Karpathy, Deep Learning (Goodfellow).