C9Evaluation & selection

Evaluation and model selection are how you find out whether a model is actually good or just lucky, and how you choose among competing models without fooling yourself. This is the most underrated discipline in machine learning: a wrong metric or a leaky validation split can make a useless model look brilliant right up until it meets real data.

You will define classification metrics (precision, recall, F1, ROC and AUC) and regression metrics (MSE, MAE, R-squared, MAPE, quantile losses), learning when each one misleads. You will diagnose whether predicted probabilities are trustworthy with reliability diagrams and Brier scores, recalibrate with Platt scaling and isotonic regression, choose the right cross-validation scheme (stratified, group, or time-series) for the test distribution you actually face, and tune hyperparameters with grid, random, and Bayesian search while watching the bias and variance of the search itself.

Every other subsection in this pillar, and every pillar after it, ultimately reports a number from here. Getting evaluation right is what separates a credible result from a misleading one.