C3Tree-based methods

Tree-based methods split the feature space into axis-aligned regions and predict a constant in each one. A single tree is interpretable but brittle; the breakthrough is combining many of them, and gradient-boosted trees are still the quiet winner of a large share of real-world tabular competitions and production systems. If you only learn one family of models well, this is a strong candidate.

You will grow a decision tree by recursive partitioning, choose splits with Gini, entropy, or MSE, and tame overfitting with pruning and cost-complexity control. Then you will move to ensembles: bagging and random forests to cut variance, feature importance done honestly with permutation, AdaBoost and its exponential-loss view, and gradient boosting framed as functional gradient descent. The track ends with XGBoost and LightGBM, where second-order objectives and histogram splits make boosting both fast and accurate.

These models handle mixed feature types, missing values, and nonlinear interactions with almost no preprocessing, which is why they remain the default for structured data and a recurring baseline for forecasting and ranking tasks downstream.