Model Evaluation Metrics

Every model you ship is judged by a number, and that number decides whether it launches, whether it gets rolled back, and whether you get credit for it. Choosing that number is a modelling decision as consequential as choosing the algorithm: a fraud model that reports 99% accuracy can be useless, a recommender with a higher AUC can make less money, and a "0.02 improvement" can be pure sampling noise.

Data science interviews probe this relentlessly because it separates people who run model.score() from people who understand what the model is for. Expect to derive precision from a confusion matrix, explain why accuracy collapses under class imbalance, say what an AUC of 0.85 actually means, choose a threshold given costs, and explain what NDCG rewards. This subject covers all of that with worked numbers.

Evaluation protocol — how to split data, k-fold cross-validation, nested CV for hyperparameter search — is covered in the Bias-Variance and Cross-Validation subject. Here we assume you already have honest held-out predictions and focus on what to compute from them.

The Confusion Matrix and Everything Derived From It

For a binary classifier at a fixed threshold, every prediction lands in one of four cells:

                        Predicted
                    Positive   Negative
Actual  Positive      TP         FN      ← recall = TP / (TP+FN)
        Negative      FP         TN      ← specificity = TN / (TN+FP)
                       ↑
              precision = TP / (TP+FP)

Worked example. A fraud model scores 10,000 card transactions; 100 are fraud (1% prevalence). At threshold 0.5:

                 Pred fraud   Pred legit
Actual fraud        60           40        (100 fraud)
Actual legit        40         9,860       (9,900 legit)

Matthews correlation coefficient is the Pearson correlation between predicted and actual labels, using all four cells:

MCC ranges from -1 to +1, is 0 for a constant or random classifier, and is the only single-number summary here that is symmetric in both classes and sensitive to all four cells. F1 ignores TN entirely.

F-beta: weighting recall vs precision

\beta > 1 weights recall more; \beta < 1 weights precision more. Take the same model at threshold 0.1, where it flags more transactions: TP = 83, FN = 17, FP = 125. Then P = 83/208 = 0.399, R = 0.83:

F2 prefers the low threshold (missing fraud is expensive), F0.5 prefers the high threshold (false alarms are expensive), F1 splits the difference. Which is right depends on the cost structure, not on the metric's popularity.