A2Probability & info theory

Probability is how you reason quantitatively about uncertainty, and information theory measures how much a distribution actually tells you. Together they are the modeling language of machine learning: every loss function, every generative model, and every notion of how confident a prediction should be is, underneath, a statement about probability.

You will start from the Kolmogorov axioms and random variables, work through expectation, variance, and the moment generating function, then the joint, marginal, and conditional view that leads to Bayes' rule. From there you meet the workhorse distributions and the exponential family, the Markov, Chebyshev, and Jensen inequalities, and the law of large numbers and central limit theorem that explain why averages behave. The discipline then turns to estimation (maximum likelihood, MAP, conjugate priors) and to information theory: entropy, KL divergence, cross-entropy, and mutual information, closing with Markov chains and a first look at Fisher information and information geometry.

You will leave able to write a likelihood, derive an estimator, and recognize cross-entropy loss as maximum likelihood in disguise. That vocabulary underpins pillar B statistics, the VAEs and diffusion models of pillar D, the RLHF penalty in pillar E, and the graphical models of pillar F.