Production ML

A model that scores well in a notebook still has to survive real data, real latency and real change. After this lesson you can name where a deployed network usually fails, and your program computes the evaluation numbers that expose the failures a single accuracy figure hides.

From prototype to service

1. Data pipeline: collect, clean, split into train / validation / test before looking at the labels. Leakage (the same customer, or a near-duplicate, on both sides of the split) inflates every score you will ever report.
2. Training: log the loss curves, save checkpoints, tune on validation only.
3. Deployment: serve behind an API (REST or gRPC) or run it in batch. Latency and cost now matter. Quantization stores weights as 8-bit integers instead of 32-bit floats: a 4 times smaller model, usually a little less accurate. Portable formats such as ONNX move a model between frameworks.
4. Monitoring: the world changes. When live inputs or outputs drift away from the training data, accuracy decays silently, with no error message to tell you.
5. Retraining: on a schedule, or triggered by drift.

The most common production bug is not in the network. It is preprocessing done differently at serving time than at training time.

Accuracy can lie

Suppose 100 emails arrive and only 4 are spam. A "model" that says "not spam" every time is right 96 times: accuracy 0.96, and it catches no spam at all. Accuracy hides which classes are failing, so we look at the confusion matrix: a table whose row is the true class, whose column is the predicted class, and whose cell counts examples.

A worked trace

Eight emails, true labels S S S N N N N N and predictions S N S N S N N N (S = spam):

              predicted S   predicted N
true S             2             1
true N             1             4

Accuracy is the diagonal over the total: (2 + 4) / 8 = 0.75. For each class c, tp is the diagonal cell, fp is the rest of column c (predicted c, but wrong) and fn is the rest of row c (really c, but missed):

precision = tp / (tp + fp)      "when I say c, how often am I right?"
recall    = tp / (tp + fn)      "of the real c, how many did I find?"
F1        = 2 * precision * recall / (precision + recall)

For S: tp 2, fp 1, fn 1, so precision = recall = F1 = 0.6667. For N: tp 4, fp 1, fn 1, so all three are 0.8. Macro F1 is the plain average of the per-class F1 scores, 0.7333 here, so every class counts equally however rare it is. In the always-"not spam" example, F1 for spam is 0 and macro F1 collapses to about 0.49 although accuracy looks excellent.

If a denominator is 0 (the class was never predicted, or never occurs), define that precision, recall or F1 as 0.

Common mistakes

• Reading the matrix transposed, so precision and recall swap.
• Integer division. 2 / 3 is 0 in several languages; convert counts to floating point first.
• Averaging F1 over only the classes that appear, instead of all K.
• Dividing by zero on an empty class instead of using the 0 convention.
• Reporting accuracy alone on imbalanced data.

Your exercise

The first line gives K, the number of classes; every other line is a true predicted pair. The starter parses the pairs and prints the results. Implement confusion, which builds the K by K matrix, and metrics, which returns accuracy and macro F1.