Soft-margin SVM and hinge loss

Hard-margin SVM is infeasible on nonseparable data and brittle near outliers. The soft-margin SVM relaxes the constraints with slack variables, charging a penalty for every margin violation[1]. Rewriting the slack penalty as a loss gives the hinge loss, which is what production SVMs actually minimize.

Slack variables

For each example, introduce a nonnegative slack $\xi_i \ge 0$ that absorbs any margin shortfall:

Three regimes:

• $\xi_i = 0$: the constraint holds with room to spare; the point sits outside or on the margin.
• $0 < \xi_i \le 1$: the point is inside the margin band but on the correct side.
• $\xi_i > 1$: the point is misclassified.

The total violation $\sum_i \xi_i$ should be small.

The slack is the margin shortfall, so its size reads off directly from the signed margin $z_i = y_i(w^\top x_i + b)$, which is $\xi_i = \max(0, 1 - z_i)$. Several points spanning the range, to fix what a unit of slack means:

• A confidently correct point at $z = 2.0$ sits well outside the margin: $\xi = 0$. No penalty.
• A point right on the margin, $z = 1.0$: $\xi = 0$. The constraint binds but is not violated.
• A point inside the margin band but still on the correct side, $z = 0.4$: $\xi = 0.6$.
• A point on the decision boundary, $z = 0$: $\xi = 1.0$. Slack of exactly one marks the boundary between "inside the margin" and "misclassified".
• A point misclassified with room, $z = -1.5$: $\xi = 2.5$. Slack grows without bound as a point moves deeper onto the wrong side.

The $C\sum_i \xi_i$ term sums these, so the objective pays $C$ per unit of this shortfall.

The QP

The regularization knob $C > 0$ trades margin width against constraint violation:

• Large $C$: heavy penalty on violations, narrow margin, close to hard-margin.
• Small $C$: tolerates violations, wider margin, more bias.

The problem is convex (quadratic objective + affine constraints).

Hinge loss: same problem, no slacks

Minimize over $\xi_i$ for fixed $(w, b)$. The optimal slack is $\xi_i = \max(0, 1 - y_i(w^\top x_i + b))$: just enough to satisfy the constraint, zero if not needed. Substituting,

The per-example loss

is the hinge loss. It is zero on correctly classified points outside the margin ($z \ge 1$), and grows linearly inside the margin and across the boundary.

The soft-margin SVM is regularized hinge-loss minimization, the same shape as L2-regularized logistic regression but with hinge instead of cross-entropy[2].

Convex but not differentiable

The hinge is not differentiable at $z = 1$ (the kink). Standard remedies:

• Subgradient descent: replace the derivative at the kink with any value in the subdifferential (typically $-y x$ if $z < 1$ else $0$).
• Proximal / smoothing: replace the hinge with a smoothed version (e.g. squared hinge).
• Dual / SMO: solve the dual QP, as in the original SVM literature.

For small to medium problems, the dual is standard; for large ones, stochastic subgradient descent on the primal (Pegasos) is the workhorse.

Choosing C

$C$ is the central hyperparameter. Cross-validate over $C \in \{10^{-3}, \dots, 10^3\}$ on a log scale. Larger $C$ may help on clean, well-separated data; smaller $C$ helps when outliers or label noise are present. The trade is exactly the bias-variance tradeoff with a different parameterization: $1/C$ plays the role of $\lambda$ in ridge.

Check it on arrays

import numpy as np

rng = np.random.default_rng(0)

# Nonseparable 2D data: two overlapping Gaussians
n = 200
X = np.vstack([rng.normal([0.8, 0], scale=1.0, size=(n//2, 2)),
               rng.normal([-0.8, 0], scale=1.0, size=(n//2, 2))])
y = np.array([1.0] * (n//2) + [-1.0] * (n//2))

def soft_margin_svm(X, y, C=1.0, lr=0.01, steps=5000):
    n, d = X.shape
    w = np.zeros(d)
    b = 0.0
    for _ in range(steps):
        margins = y * (X @ w + b)
        viol = margins < 1.0                            # which points violate
        grad_w = w - C * (y[viol, None] * X[viol]).sum(axis=0)
        grad_b = - C * y[viol].sum()
        w -= lr * grad_w / n
        b -= lr * grad_b / n
    return w, b

# Sweep C: more weight on violations -> tighter margin
for C in [0.01, 1.0, 100.0]:
    w, b = soft_margin_svm(X, y, C=C, lr=0.1)
    margin = 1.0 / np.linalg.norm(w)
    acc = np.mean(np.sign(X @ w + b) == y)
    viol = np.mean(y * (X @ w + b) < 1.0)
    print(f"C={C:>6}: margin={margin:.3f}, accuracy={acc:.3f}, fraction with violation={viol:.3f}")

# Verify the hinge-loss equivalence: optimal slack is max(0, 1 - y(w^T x + b))
w, b = soft_margin_svm(X, y, C=1.0, lr=0.1)
hinge_loss = np.maximum(0, 1 - y * (X @ w + b)).sum()
print(f"\ntotal hinge loss = {hinge_loss:.3f}  (= sum of optimal slacks)")

Larger $C$ tightens the margin and reduces the fraction of margin violations; the total hinge loss equals the sum of optimal slacks.

Exercises

Exercise 1. Four points have signed margins $z_i = y_i(w^\top x_i + b)$ equal to $2.0$, $0.7$, $0.0$, and $-1.0$. Compute the optimal slack $\xi_i$ for each, the per-example hinge loss, and classify each point as outside the margin, inside the band, on the boundary, or misclassified.

Exercise 2. For fixed $(w, b)$, the soft-margin objective in the slack $\xi_i$ is $\min_{\xi_i \ge 0} C\xi_i$ subject to $\xi_i \ge 1 - z_i$ with $z_i = y_i(w^\top x_i + b)$. Show the minimizer is $\xi_i^\star = \max(0, 1 - z_i)$, which is why the slack penalty collapses to the hinge loss.

Exercise 3 (code). On two overlapping Gaussian classes, sweep $C \in \{0.01, 1, 100\}$ with subgradient descent and confirm that the margin width $1/\lVert w\rVert$ is non-increasing in $C$ (larger $C$ penalizes violations harder, so the margin tightens).

import numpy as np

rng = np.random.default_rng(0)
n = 200
X = np.vstack([rng.normal([0.8, 0], 1.0, (n // 2, 2)),
               rng.normal([-0.8, 0], 1.0, (n // 2, 2))])
y = np.array([1.0] * (n // 2) + [-1.0] * (n // 2))

def soft_margin_svm(X, y, C, lr=0.1, steps=5000):
    n, d = X.shape
    w = np.zeros(d)
    b = 0.0
    for _ in range(steps):
        viol = y * (X @ w + b) < 1.0               # points inside the margin
        grad_w = w - C * (y[viol, None] * X[viol]).sum(axis=0)
        grad_b = -C * y[viol].sum()
        w -= lr * grad_w / n
        b -= lr * grad_b / n
    return w, b

margins = []
for C in [0.01, 1.0, 100.0]:
    w, b = soft_margin_svm(X, y, C)
    margins.append(1.0 / np.linalg.norm(w))

print("margins by C:", [round(m, 3) for m in margins])
assert margins[0] >= margins[1] >= margins[2]      # margin shrinks as C grows

The next lesson casts the SVM in its dual form, where the support-vector structure becomes explicit.