GloVe and fastText

Skip-gram trains on individual (center, context) pairs sampled sequentially. The same co-occurrence information is already summarized in the word-word co-occurrence matrix -- if we could fit directly to that matrix, we could exploit global statistics in one shot. GloVe (Global Vectors) does exactly this[1].

GloVe: global co-occurrence

The co-occurrence matrix

Let $X_{ij}$ be the number of times word $j$ appears in the context of word $i$ (within a window), and $X_i = \sum_j X_{ij}$. The ratio

encodes the relative affinity of probe word $k$ for context $i$ vs $j$. This ratio is the key quantity: it equals approximately 1 for words equally associated with both $i$ and $j$, very large for words specifically associated with $i$, and very small for words specifically associated with $j$.

Deriving the objective

We want embeddings $\mathbf{v}_i, \mathbf{u}_k$ such that some function $F(\mathbf{v}_i, \mathbf{v}_j, \mathbf{u}_k)$ captures the co-occurrence ratio. Requiring $F$ to be linear in the embedding differences and compatible with the ratio form forces:

This is the GloVe log-bilinear model: the inner product of center and context embeddings, plus per-word biases, should approximate the log co-occurrence count.

The GloVe objective is a weighted least-squares fit to this:

where the weighting function $f(X_{ik}) = \min\!\left(\!(X_{ik}/X_{\max})^{0.75},\, 1\right)$ down-weights rare pairs (which have noisier counts) and caps the weight of very frequent pairs.

import numpy as np
from collections import defaultdict, Counter

# Build co-occurrence matrix from a tiny corpus
sentences = [
    "the cat sat on the mat".split(),
    "the dog sat on the floor".split(),
    "the cat and the dog are animals".split(),
    "animals live on the floor".split(),
]
vocab = list(set(w for s in sentences for w in s))
w2i = {w: i for i, w in enumerate(vocab)}
V = len(vocab)
window = 2

X = np.zeros((V, V))
for sent in sentences:
    for i, center in enumerate(sent):
        for j in range(max(0, i - window), min(len(sent), i + window + 1)):
            if j != i:
                X[w2i[center], w2i[sent[j]]] += 1

# GloVe training
d = 8
x_max = 100.0
rng = np.random.default_rng(0)
V_emb = rng.normal(0, 0.1, (V, d))
U_emb = rng.normal(0, 0.1, (V, d))
b_v   = np.zeros(V)
b_u   = np.zeros(V)
lr    = 0.05

for epoch in range(200):
    loss = 0.0
    for i in range(V):
        for k in range(V):
            if X[i, k] == 0:
                continue
            w = min((X[i, k] / x_max) ** 0.75, 1.0)
            diff = V_emb[i] @ U_emb[k] + b_v[i] + b_u[k] - np.log(X[i, k])
            loss += w * diff ** 2
            grad = 2 * w * diff
            v_i = V_emb[i].copy()
            V_emb[i] -= lr * grad * U_emb[k]
            U_emb[k] -= lr * grad * v_i
            b_v[i]   -= lr * grad
            b_u[k]   -= lr * grad
    if (epoch + 1) % 100 == 0:
        print(f"Epoch {epoch + 1}: loss = {loss:.3f}")

# Combine center and context embeddings (GloVe convention)
W = V_emb + U_emb

fastText: subword embeddings

Word2vec and GloVe treat each word as an atomic unit. This breaks on:

• Rare words: "antidisestablishmentarianism" has very few training occurrences.
• Morphological variants: "play," "plays," "playing" share a root but have separate embeddings with no guaranteed similarity.
• Out-of-vocabulary (OOV): a word not in the training vocabulary gets no embedding.

fastText (Bojanowski et al., 2017)[2] represents each word as the sum of its character n-gram embeddings plus its full-word embedding.

For word $w$ with character n-grams $\mathcal{G}_w$ (for $n \in [3, 6]$):

where $\mathbf{v}_{[w]}$ is the full-word embedding and $\mathbf{z}_g$ is the embedding of n-gram $g$ (shared across all words containing that n-gram).

For example, "apple" (with $\langle$ and $\rangle$ as boundary markers) generates n-grams: <ap, app, ppl, ple, le>, <app, appl, pple, ple>, <appl, apple, pple>, <apple>, plus the whole-word special token <apple>.

Key properties

Morphology: "play" and "playing" share n-grams pla, lay, <pl, etc., so their embeddings are close by construction.

OOV: for a word not in the vocabulary, sum the embeddings of its character n-grams. The resulting vector is meaningful even if the word was never seen.

Rare words: each character n-gram is shared across many words, so even rare words receive informative updates from all words sharing the same substrings.

# fastText is best used via the official library
# pip install fasttext  (or fasttext-wheel)
import fasttext

# Train on a corpus file
model = fasttext.train_unsupervised("corpus.txt", model="skipgram",
                                    dim=100, minCount=1, minn=3, maxn=6)

# Embedding of an unseen word
vec = model.get_word_vector("antidisestablishmentarianism")
print("Vector shape:", vec.shape)

# Nearest neighbors (works for OOV too)
model.get_nearest_neighbors("plays")

GloVe vs word2vec vs fastText

Exercises

Exercise 1. The GloVe weighting is $f(x) = \min\!\left((x / x_{\max})^{0.75},\, 1\right)$ with $x_{\max} = 100$. Compute $f(x)$ by hand for co-occurrence counts $x \in \{1, 10, 100, 500\}$, and state in one sentence what the weighting does to the contributions of a count-1 pair versus a count-100 pair.

Exercise 2. A word $w$ and its plural $w\text{s}$ differ by one character. Using fastText's decomposition $\mathbf{v}_w = \mathbf{v}_{[w]} + \sum_{g \in \mathcal{G}_w} \mathbf{z}_g$, explain why their vectors are close even if $w\text{s}$ was never seen in training, and contrast this with what word2vec or GloVe would produce for an unseen $w\text{s}$.

Exercise 3 (code). The GloVe update in the implementation above uses $\nabla = 2 f(X_{ik})\, (\mathbf{v}_i \cdot \mathbf{u}_k + b_i + b_k - \log X_{ik})$. Verify the analytic gradients of one squared term with respect to $\mathbf{v}_i$ and $b_i$ against finite differences.

import numpy as np

rng = np.random.default_rng(0)
d = 6
v_i = rng.normal(0, 1, d)
u_k = rng.normal(0, 1, d)
b_i, b_k = 0.3, -0.2
X_ik, x_max = 12.0, 100.0
w = min((X_ik / x_max) ** 0.75, 1.0)

def term(v_i, u_k, b_i, b_k):
    diff = v_i @ u_k + b_i + b_k - np.log(X_ik)
    return w * diff ** 2

diff = v_i @ u_k + b_i + b_k - np.log(X_ik)
g_vi = 2 * w * diff * u_k          # analytic: d term / d v_i
g_bi = 2 * w * diff                # analytic: d term / d b_i

eps = 1e-6
g_vi_fd = np.zeros(d)
for j in range(d):
    e = np.zeros(d); e[j] = eps
    g_vi_fd[j] = (term(v_i + e, u_k, b_i, b_k)
                  - term(v_i - e, u_k, b_i, b_k)) / (2 * eps)
g_bi_fd = (term(v_i, u_k, b_i + eps, b_k)
           - term(v_i, u_k, b_i - eps, b_k)) / (2 * eps)

assert np.allclose(g_vi, g_vi_fd, atol=1e-5)
assert np.isclose(g_bi, g_bi_fd, atol=1e-5)
print("v_i grad max diff:", np.abs(g_vi - g_vi_fd).max())

All three produce static embeddings -- one vector per word type. The next lesson introduces RNNs, which produce a hidden state that reads the entire sequence and can be used to build the first contextual representations.