Modern Frameworks

NumPy from scratch is educational. Production uses frameworks:

PyTorch (Facebook → Meta, 2017):

• Dynamic computation graph (define-by-run).
• Pythonic; easy to debug.
• Massive research adoption.
• TorchScript for production deployment.

TensorFlow / Keras:

• Static graph (TF1) or eager (TF2).
• Strong production tooling (TF Serving, TF.js).

JAX (Google):

• Functional, composable.
• jit: just-in-time compile.
• vmap: auto vectorize.
• grad: auto differentiation.
• Used by DeepMind, scientific computing.

MLX (Apple):

• Native Apple Silicon optimization.
• Apple's PyTorch alternative.

GPU acceleration:

• CUDA (NVIDIA).
• ROCm (AMD).
• Metal (Apple).
• Frameworks abstract.

Distributed training:

• Data parallel: each GPU has full model, batch split.
• Model parallel: model split across GPUs (for huge models).
• ZeRO / FSDP: shard optimizer + gradients across GPUs.

Tensor cores:

• Specialized hardware for matrix multiplication.
• 100x faster than general FP32 ops.
• Mixed precision training: FP16 forward + FP32 master weights.

Modern training libraries:

• Hugging Face Transformers: pretrained models, easy fine-tuning.
• PyTorch Lightning: training loop boilerplate.
• Deepspeed: distributed + memory efficient.
• Accelerate: simple multi-GPU.

For learning: PyTorch is the most popular starting point.