Gated FFNs: SwiGLU and GeGLU

The position-wise feed-forward network (FFN) inside a transformer block is, in the original design, a two-layer MLP with a ReLU in the middle. Modern LLMs (PaLM, LLaMA, Mistral) replace it with a gated variant that consistently lowers loss at equal compute[1]. This lesson derives that variant and the width adjustment that makes the comparison fair.

1. The baseline FFN

For a token representation $x \in \mathbb{R}^{d}$ (here $d = d_{\text{model}}$), the original FFN is

where $\phi$ is an elementwise activation (ReLU in the 2017 model) and $d_{ff}$ is the hidden width, conventionally $d_{ff} = 4d$. Ignoring biases, this costs

parameters.

2. The gated linear unit

A gated linear unit (GLU) splits the first projection into two: one branch produces values, the other produces a multiplicative gate that decides, per coordinate, how much of the value to pass. With two projections $W, V \in \mathbb{R}^{d_{ff}\times d}$,

where $\odot$ is the elementwise (Hadamard) product and $\sigma$ is the logistic sigmoid. The gate $\sigma(Vx) \in (0,1)^{d_{ff}}$ attenuates each hidden coordinate.

Swapping the gate nonlinearity gives the named variants. With the Swish (also called SiLU) function $\text{Swish}(z) = z\,\sigma(z)$:

With the Gaussian error linear unit $\text{GELU}(z) = z\,\Phi(z)$, where $\Phi$ is the standard normal CDF:

The full block then applies a third matrix $W_2$ to project back to $d$:

What the gate buys is per-coordinate control that a plain ReLU MLP cannot express. Take one hidden coordinate with value branch $W_1 x = 2.0$ and watch the SwiGLU output as the gate pre-activation $W_3 x$ swings sign:

• Gate open. $W_3 x = +3$ gives $\text{Swish}(3) = 3\,\sigma(3) \approx 2.858$, so the coordinate passes through amplified: $2.0 \times 2.858 \approx 5.72$.
• Gate shut. $W_3 x = -3$ gives $\text{Swish}(-3) \approx -0.142$, so the same value branch is squeezed to $2.0 \times (-0.142) \approx -0.285$, near zero.

The value $W_1 x = 2.0$ is identical in both rows; only the gate input differs, and it moves the output across a factor of twenty. A ReLU MLP would key the gate to the same coordinate's own sign; the GLU lets a separate projection $W_3$ decide, per coordinate, how much of the value to let through.

3. The 2/3 width rule

The gated FFN uses three weight matrices instead of two, so a naive swap at $d_{ff}=4d$ raises the parameter count by 50%. To compare architectures at equal size, shrink the hidden width. The gated count is

Setting $P_{\text{gated}} = P_{\text{FFN}} = 8d^2$ and solving:

The new hidden width is two-thirds of the old one. This is why LLaMA reports $d_{ff} \approx \tfrac{8}{3}d$ (rounded to a convenient multiple) rather than $4d$: it holds parameters and FLOPs fixed so the gain is attributable to the gating, not to extra capacity.

4. Worked check

The block below builds a ReLU FFN and a parameter-matched SwiGLU FFN, confirms their parameter counts agree, and runs one forward pass.

import numpy as np

def swish(z):
    return z / (1.0 + np.exp(-z))

d = 512
d_ff_relu = 4 * d
d_ff_glu = (8 * d) // 3        # the 2/3 rule, rounded down

# baseline ReLU FFN
W1 = np.random.randn(d_ff_relu, d) * 0.02
W2 = np.random.randn(d, d_ff_relu) * 0.02
relu_params = W1.size + W2.size

# parameter-matched SwiGLU FFN (three matrices)
Wg = np.random.randn(d_ff_glu, d) * 0.02   # value branch
Wv = np.random.randn(d_ff_glu, d) * 0.02   # gate branch
Wo = np.random.randn(d, d_ff_glu) * 0.02
glu_params = Wg.size + Wv.size + Wo.size

x = np.random.randn(d)
relu_out = W2 @ np.maximum(0.0, W1 @ x)
glu_out = Wo @ ((Wg @ x) * swish(Wv @ x))

print("ReLU FFN params:  ", relu_params)
print("SwiGLU FFN params:", glu_params)
print("ratio:", round(glu_params / relu_params, 3))
print("output shapes:", relu_out.shape, glu_out.shape)

The two parameter counts land within rounding of each other (ratio near 1.0), so any loss difference between the architectures is a difference in form, not in size.

Exercises

Exercise 1. With Swish defined as $\text{Swish}(z) = z\,\sigma(z)$, compute the SwiGLU output of one hidden coordinate whose value branch is $W_1 x = 2.0$, for gate inputs $W_3 x = +3$ and $W_3 x = -3$. ($\sigma(3) \approx 0.9526$, $\sigma(-3) \approx 0.0474$.)

Exercise 2. Derive the 2/3 width rule. A baseline ReLU FFN with hidden width $d_{ff} = 4d$ has two weight matrices; a gated FFN has three. Find the gated hidden width $d_{ff}'$ that holds the parameter count fixed.

Exercise 3 (code). Build a ReLU FFN at $d_{ff} = 4d$ and a SwiGLU FFN at the 2/3 width, and confirm their parameter counts agree within rounding (ratio close to $1$).

import numpy as np

d = 512
d_ff_relu = 4 * d
d_ff_glu = (8 * d) // 3          # 2/3 rule, rounded down -> 1365

# ReLU FFN: two matrices of size d x d_ff
relu_params = 2 * d * d_ff_relu
# SwiGLU FFN: three matrices of size d x d_ff'
glu_params = 3 * d * d_ff_glu

ratio = glu_params / relu_params
assert d_ff_glu == 1365                    # floor(8*512/3)
assert abs(ratio - 1.0) < 0.01             # matched within rounding
print("relu", relu_params, "glu", glu_params, "ratio", round(ratio, 4))

Next, multi-head latent attention attacks the other half of the block, compressing the attention KV cache the same way this lesson reshaped the FFN.