A3Optimization theory

Optimization is how a model learns: you write down a loss surface and then search it for the parameters that make the loss small. Training a neural network, fitting a logistic regression, and tuning a support vector machine are all the same problem in different clothing, which is minimizing a function you can differentiate.

You will formalize what a minimum is through the gradient and Hessian, study convex sets and convex functions (the cases where local means global), and then build the descent methods in order: gradient descent and its convergence guarantees, stochastic gradient descent, momentum and Nesterov acceleration, and the adaptive AdaGrad/RMSProp/Adam family that trains real networks. The discipline also covers constrained optimization through Lagrange multipliers, duality, and the KKT conditions, then second-order and specialized methods (Newton, BFGS and L-BFGS, proximal gradient for nonsmooth penalties, and Bayesian optimization for expensive black boxes).

After this you can read an optimizer's update rule and know why it behaves as it does, diagnose slow or unstable training, and choose a method to match a problem's structure. Every training loop in pillars C through H is an instance of what you study here.