C4Instance & probabilistic

Instance-based and probabilistic classifiers are two of the oldest and most transparent ways to label data. Instance-based methods, like k-nearest neighbors, make no model at all: they just remember the training set and let nearby points vote. Probabilistic classifiers, like naive Bayes and discriminant analysis, instead model how each class generates its features and then invert that with Bayes' rule. Both are excellent baselines and surprisingly hard to beat on the right problem.

You will implement k-NN for classification and regression, choose k by cross-validation, and confront the curse of dimensionality that erodes distance-based methods as features pile up. Then you will build naive Bayes from the conditional-independence assumption, derive linear and quadratic discriminant analysis by modeling classes as Gaussians, and meet Fisher's LDA as a supervised projection that maximizes class separation, the discriminative twin of PCA.

The Bayes-rule reasoning here is foundational for probabilistic graphical models later, the distance intuitions feed directly into nearest-neighbor search and vector databases, and LDA-style projections recur throughout dimensionality reduction.