Positional encodings: sinusoidal, RoPE, ALiBi

Attention is permutation-equivariant: if you shuffle the input tokens, the output is shuffled by the same permutation. The model has no intrinsic notion of "this token comes before that one." Yet order is fundamental to language -- "dog bites man" and "man bites dog" must have different meanings.

Positional encodings inject order information into the representations.

Sinusoidal absolute encodings (Vaswani et al., 2017)

Add a fixed, deterministic vector $\mathbf{p}_t$ to each token embedding at position $t$[1]:

for feature index $i = 0, 1, \dots, d/2 - 1$.

Why this form?

• Different frequencies mean positions are distinguishable.
• Linearity: $\mathbf{p}_{t+k}$ can be written as a linear function of $\mathbf{p}_t$ for any offset $k$ (rotation in 2D planes). The model can potentially learn to attend based on relative offset, the idea made explicit by encoding relative positions directly in the attention scores[2].
• Unbounded: can generate encodings for any sequence length.

Limitation: position information is added to the embedding, not the attention scores. As sequences grow beyond the training length, the embeddings become out-of-distribution.

import numpy as np
import matplotlib
matplotlib.use('Agg')

def sinusoidal_pe(T, d):
    """Returns (T, d) sinusoidal positional encoding matrix. d must be even."""
    assert d % 2 == 0, "d must be even for sinusoidal PE"
    pe = np.zeros((T, d))
    pos = np.arange(T)[:, None]          # (T, 1)
    dim = np.arange(0, d, 2)[None, :]   # (1, d/2)
    angles = pos / 10000 ** (dim / d)   # (T, d/2)
    pe[:, 0::2] = np.sin(angles)        # (T, d/2) -> even columns
    pe[:, 1::2] = np.cos(angles)        # (T, d/2) -> odd columns
    return pe

pe = sinusoidal_pe(T=64, d=128)
print("PE shape:", pe.shape)
print("PE[0, :4]:", pe[0, :4].round(4))
print("PE[1, :4]:", pe[1, :4].round(4))
# Verify: columns 0,1 are sin/cos of position / 1.0 = position
# columns d-2,d-1 are nearly constant (very long wavelength)

Rotary Position Encoding (RoPE)

RoPE (Su et al., 2022; used by LLaMA, Mistral, Qwen) encodes position in the attention score rather than the embedding[3]. The key insight: rotate the query and key vectors by an angle proportional to their position, so that the attention score between positions $t$ and $s$ depends only on the relative offset $t - s$.

For a 2D slice of query $\mathbf{q}$ at position $t$:

Apply the same rotation to keys with angle $s \theta_i$. The dot product $\mathbf{q}_t \cdot \mathbf{k}_s$ then equals the dot product of the unrotated vectors premultiplied by a rotation of angle $(t - s)\theta_i$ -- a function of the relative offset only.

Key properties:

• Relative position: attention scores depend on $t - s$, not on $t$ and $s$ independently.
• Extrapolation: works well at lengths somewhat beyond training (unlike sinusoidal absolute).
• Zero overhead: applied to Q and K before attention, not to the residual stream.

import numpy as np

def apply_rope(x, positions):
    """x: (T, d), positions: (T,). Returns RoPE-rotated x."""
    T, d = x.shape
    assert d % 2 == 0
    half = d // 2
    thetas = 10000 ** (-2 * np.arange(half) / d)      # (half,)
    angles = positions[:, None] * thetas[None, :]     # (T, half)
    cos_a, sin_a = np.cos(angles), np.sin(angles)

    x_even = x[:, 0::2]   # (T, half)
    x_odd  = x[:, 1::2]

    x_rotated = np.empty_like(x)
    x_rotated[:, 0::2] = x_even * cos_a - x_odd * sin_a
    x_rotated[:, 1::2] = x_even * sin_a + x_odd * cos_a
    return x_rotated

T, d = 8, 16
Q = np.random.randn(T, d)
Q_rope = apply_rope(Q, positions=np.arange(T))
print("RoPE output shape:", Q_rope.shape)

The relative-offset claim is checkable on numbers. Fix one query vector and one key vector ($d = 8$) and rotate them to various absolute positions, then read off $\mathbf{q}_t \cdot \mathbf{k}_s$:

• Offset $t - s = 2$, three placements. At $(t, s) = (2, 0)$, $(5, 3)$, and $(10, 8)$ the score is $-1.7631$ in all three cases. The absolute positions slide by $8$; the score does not move, because only $t - s$ enters.
• Offset $0$ (a token against itself). At $t = s = 0$, $5$, $10$ the score is $-1.4680$ every time, and it equals the dot product of the unrotated $\mathbf{q}$ and $\mathbf{k}$: rotating both vectors by the same angle leaves their dot product unchanged.

That is the named distinction RoPE is built on: sinusoidal encoding gives each absolute position a fingerprint, while RoPE makes the attention score a function of the gap between positions, so $(2, 0)$ and $(10, 8)$ are indistinguishable to it.

ALiBi: Attention with Linear Biases

ALiBi (Press et al., 2022; used by MPT, BLOOM) adds a fixed linear bias to the pre-softmax attention scores, proportional to the distance between positions:

where $m_i$ is a head-specific slope (a geometric sequence across heads, not learned).

Key properties:

• No modification to embeddings or projections.
• Distance penalty grows linearly, so distant tokens are explicitly penalized.
• Excellent length extrapolation: models trained on length 2048 often maintain performance at 4096+ without any fine-tuning. The linear penalty is monotonically stronger for larger offsets, which is a natural inductive bias.

import numpy as np

def alibi_bias(T, n_heads):
    """Returns (n_heads, T, T) ALiBi bias matrix."""
    # Slopes: geometric sequence 2^{-8/h}, 2^{-16/h}, ...
    slopes = 2 ** (-8 * np.arange(1, n_heads + 1) / n_heads)   # (h,)
    # Relative distances: |t - j| for each (t, j) pair
    distances = np.abs(np.arange(T)[:, None] - np.arange(T)[None, :])  # (T, T)
    # bias: (h, T, T)
    return -slopes[:, None, None] * distances[None, :, :]

bias = alibi_bias(T=8, n_heads=4)
print("ALiBi bias[0]:\n", bias[0].round(3))  # head 0: steepest penalty

Comparison

Modern LLMs (LLaMA 3, Qwen, Mistral) universally use RoPE. ALiBi is preferred when extrapolation is a hard requirement without fine-tuning.

Exercises

Exercise 1. Write the sinusoidal encoding vector $\mathbf{p}_0$ at position $t = 0$ for any dimension $d$. What does it equal, and why does that make $t = 0$ a natural reference point?

Exercise 2. RoPE rotates each 2D slice of $\mathbf{q}$ and $\mathbf{k}$ by an angle proportional to position. Prove that rotating both vectors by the same angle leaves their dot product unchanged, then explain why this makes the attention score depend only on the offset $t - s$.

Exercise 3 (code). Implement RoPE and verify both claims numerically: equal offsets give equal scores at different absolute placements, and a zero offset reproduces the unrotated dot product.

import numpy as np

def apply_rope(x, positions):
    T, d = x.shape
    assert d % 2 == 0
    half = d // 2
    thetas = 10000 ** (-2 * np.arange(half) / d)
    angles = positions[:, None] * thetas[None, :]
    cos_a, sin_a = np.cos(angles), np.sin(angles)
    x_even, x_odd = x[:, 0::2], x[:, 1::2]
    out = np.empty_like(x)
    out[:, 0::2] = x_even * cos_a - x_odd * sin_a
    out[:, 1::2] = x_even * sin_a + x_odd * cos_a
    return out

rng = np.random.default_rng(0)
q = rng.normal(size=(1, 8))
k = rng.normal(size=(1, 8))

def score(t, s):
    qt = apply_rope(q, np.array([t]))
    ks = apply_rope(k, np.array([s]))
    return (qt @ ks.T)[0, 0]

# same offset (= 2) at three placements
s_far = [score(2, 0), score(5, 3), score(10, 8)]
assert np.allclose(s_far, s_far[0])              # offset fixes the score

# zero offset equals the unrotated dot product
assert np.isclose(score(0, 0), (q @ k.T)[0, 0])
print("offset-2 score", round(s_far[0], 4), "| zero offset == raw dot")