Stacking and blending

Bagging and boosting combine many copies of the same learner. Stacking combines different learners by training a meta-learner to weight their predictions[1]. With the right setup, it gives small but reliable gains over any single model on a held-out set, and it is the standard way to win machine-learning competitions.

The pattern

You have $M$ base learners (logistic regression, random forest, gradient boosting, k-NN, neural network: any mix). Train each on the training data. For a query $x$:

where $g$ is the meta-learner trained to combine the base predictions into the final output.

The catch is leakage: if you train $g$ on the same data the base learners were trained on, $g$ learns to trust the most overfitting base model. The fix is out-of-fold predictions.

Out-of-fold (OOF) construction

Use $k$-fold cross-validation to generate base-model predictions on data the base model has not seen:

1. Split the training data into $k$ folds.
2. For each fold $i$: train each base learner on the other $k - 1$ folds, predict on fold $i$. Stack predictions across folds into an OOF matrix $\Phi \in \mathbb{R}^{n \times M}$, one column per base learner.
3. Train the meta-learner $g$ on $(\Phi, y)$.
4. For inference: retrain each base learner on the full training set, then predict with the meta-learner.

The OOF construction ensures the meta-learner sees base predictions on data the base learners have not memorized[2].

Choosing the meta-learner

A common choice: logistic regression (for classification) or linear regression (for regression), with L2 regularization. The meta-learner is deliberately simple to avoid over-weighting any one base model. Tree-based meta-learners are also used but more prone to overfitting on the small $\Phi$ matrix.

The simplest meta-learner is a convex combination: predict the weighted average $\sum_m w_m f_m(x)$ with $w_m \ge 0$, $\sum_m w_m = 1$. Pick the weights by minimizing held-out loss (a constrained least-squares problem).

Blending

Blending is the lazy version of stacking: instead of $k$-fold OOF, hold out a single validation set, train base learners on the rest, and predict on the validation set to train the meta-learner. Cheaper but uses less data and is more prone to overfitting on the small validation set.

When stacking wins

• Base learners are heterogeneous: very different model families with different inductive biases. Two random forests with different hyperparameters contribute much less than a random forest plus a linear model.
• Base learners are uncorrelated in their errors. Check the OOF correlation matrix; weights on highly correlated learners cancel out.
• Enough training data to fit good base learners and have meaningful OOF predictions for the meta-learner.

When it does not

• Single learner is already near-optimal. The marginal gain rarely justifies the engineering cost.
• Production constraints: stacking $M$ models means $M$ times the inference cost and $M$ times the deployment surface area. Often a single tuned gradient-boosted model is the better engineering choice.

Check it on arrays

import numpy as np

rng = np.random.default_rng(0)

# Synthetic 2-class problem
n = 600; d = 5
X = rng.normal(size=(n, d))
y = (X[:, 0] + X[:, 1]**2 - X[:, 2] * X[:, 3] > 0).astype(int)

# Two base learners with different biases:
# (1) logistic regression  (2) k-NN
def fit_logistic(X, y, n_iter=200, lr=0.1):
    X1 = np.column_stack([np.ones(len(X)), X])
    beta = np.zeros(X1.shape[1])
    for _ in range(n_iter):
        p = 1 / (1 + np.exp(-X1 @ beta))
        beta += lr * X1.T @ (y - p) / len(y)
    return beta

def logistic_predict_proba(beta, X):
    X1 = np.column_stack([np.ones(len(X)), X])
    return 1 / (1 + np.exp(-X1 @ beta))

def knn_predict_proba(X_train, y_train, X_query, k=15):
    d = np.sqrt(((X_query[:, None] - X_train[None, :])**2).sum(axis=2))
    nn = np.argsort(d, axis=1)[:, :k]
    return y_train[nn].mean(axis=1)

# Hold out a final test set the meta-learner never sees, so the comparison
# is honest. Use 5-fold OOF on the training data: build phi_tr for fitting
# the meta-learner, and AVERAGE per-fold test predictions to build phi_te
# (so train and test base predictions have the same distribution).
test_idx = rng.permutation(n)[: n // 5]
train_idx = np.setdiff1d(np.arange(n), test_idx)
X_tr, y_tr = X[train_idx], y[train_idx]
X_te, y_te = X[test_idx], y[test_idx]

K = 5
folds = np.array_split(rng.permutation(len(X_tr)), K)
phi_tr = np.zeros((len(X_tr), 2))
phi_te_acc = np.zeros((len(X_te), 2))
for fold in folds:
    inner_train = np.setdiff1d(np.arange(len(X_tr)), fold)
    beta = fit_logistic(X_tr[inner_train], y_tr[inner_train])
    phi_tr[fold, 0] = logistic_predict_proba(beta, X_tr[fold])
    phi_tr[fold, 1] = knn_predict_proba(X_tr[inner_train], y_tr[inner_train],
                                        X_tr[fold], k=15)
    # Per-fold test-set predictions; average across folds at the end.
    phi_te_acc[:, 0] += logistic_predict_proba(beta, X_te)
    phi_te_acc[:, 1] += knn_predict_proba(X_tr[inner_train], y_tr[inner_train],
                                          X_te, k=15)
phi_te = phi_te_acc / K

# Train the meta-learner on OOF predictions
beta_meta = fit_logistic(phi_tr, y_tr)
print(f"meta-learner coefficients: intercept={beta_meta[0]:.3f}, "
      f"w_logistic={beta_meta[1]:.3f}, w_knn={beta_meta[2]:.3f}")

# Evaluate every method on the same held-out test set
stack_pred = logistic_predict_proba(beta_meta, phi_te)
acc_stack = np.mean((stack_pred > 0.5) == y_te)
acc_logistic_alone = np.mean((phi_te[:, 0] > 0.5) == y_te)
acc_knn_alone = np.mean((phi_te[:, 1] > 0.5) == y_te)
print(f"held-out test accuracy: logistic alone = {acc_logistic_alone:.3f}, "
      f"k-NN alone = {acc_knn_alone:.3f}, stacked = {acc_stack:.3f}")

The meta-coefficients show how the meta-learner weights each base model. On this small two-learner toy with k-NN already dominating, stacking does not clearly beat the best single learner: the win from stacking comes from heterogeneity and from problems where no single model is best across the data, not from any two random base learners.

Exercises

Exercise 1. A stacking pipeline uses $5$-fold cross-validation with $M = 3$ base learners on $n = 1000$ training rows. How many base-model fits are performed to build the out-of-fold matrix $\Phi$, and how many additional fits to prepare for inference? What are the dimensions of $\Phi$?

Exercise 2. The simplest meta-learner is a convex combination $\sum_m w_m f_m$ with $w_m \ge 0$, $\sum_m w_m = 1$. Take two base predictors $f_1, f_2$ with equal error variance $\sigma^2$ and error correlation $\rho$. Find the weight $w$ on $f_1$ (blend $w f_1 + (1-w) f_2$) that minimizes the blend's error variance, and evaluate it at $\rho = -1$.

Exercise 3 (code). Construct two base regressors whose errors are perfectly anti-correlated ($f_1 = y + e$, $f_2 = y - e$). Search over convex weights $w$ and confirm the best blend has lower mean squared error than either base model, with the optimum at $w \approx 0.5$.

import numpy as np

rng = np.random.default_rng(0)

n = 5000
y = rng.normal(size=n)

# Base 1 overshoots by e, base 2 undershoots by e: anti-correlated errors.
e = rng.normal(scale=1.0, size=n)
f1 = y + e
f2 = y - e

def mse(pred):
    return np.mean((pred - y) ** 2)

mse1, mse2 = mse(f1), mse(f2)
ws = np.linspace(0, 1, 101)
mses = [mse(w * f1 + (1 - w) * f2) for w in ws]
w_star = ws[int(np.argmin(mses))]
mse_blend = min(mses)
print(f"MSE f1={mse1:.3f}, f2={mse2:.3f}, best blend (w={w_star:.2f})={mse_blend:.4f}")

assert mse_blend < min(mse1, mse2)           # blend beats both bases
assert np.isclose(w_star, 0.5, atol=0.05)    # optimum at the equal blend
assert mse_blend < 0.05                      # anti-correlated errors cancel

The next lesson covers cheap alternatives to true stacking: snapshot ensembles and voting.