Gradient Descent

After computing gradients, update each weight:

W -= learning_rate * dW
b -= learning_rate * db

Learning rate (lr): hyperparameter.

• Too high: training unstable, loss oscillates or explodes.
• Too low: training takes forever.
• Typical: 0.001 - 0.1.

Variants:

SGD (Stochastic Gradient Descent):

• Compute gradient on one sample at a time.
• Updates often, noisy.

Mini-batch SGD:

• Compute gradient on a batch (32-256 samples).
• Standard. Less noisy than SGD, faster than full-batch.

Full-batch GD:

• Gradient on all data.
• Stable but slow on large datasets.

Momentum:

• Accumulate gradient over time.
• Like a ball rolling down a hill: gains speed in consistent direction.
• v = β * v + grad; W -= lr * v.
• Helps with shallow valleys.

Adam (adaptive moment estimation):

• Combines momentum + per-parameter learning rates.
• Default optimizer for many tasks.
• Slightly more memory but converges faster.

Learning rate scheduling:

• Start high, decay over time.
• Cosine schedule: smooth decay over training.
• Warmup: low → high → decay.

Modern training uses Adam (or AdamW for weight decay) + cosine schedule + warmup. Standard recipe.