Topic models: LDA and NMF

A document is a mixture of topics; each topic is a distribution over words. Topic models discover these latent themes from a corpus without supervision, grouping words that tend to appear together and assigning each document a mixture of those groups.

The two canonical approaches are Latent Dirichlet Allocation (LDA), a probabilistic graphical model, and Non-negative Matrix Factorization (NMF), a deterministic matrix decomposition[2].

Latent Dirichlet Allocation

The generative story

LDA assumes documents are generated by the following process[1]:

1. For each document $d$, draw a topic mixture $\theta_d \sim \text{Dir}(\alpha)$.
2. For each of the $K$ topics $k$, draw a word distribution $\phi_k \sim \text{Dir}(\beta)$.
3. For each word position $n$ in document $d$: a. Draw a topic assignment $z_{dn} \sim \text{Categorical}(\theta_d)$. b. Draw a word $w_{dn} \sim \text{Categorical}(\phi_{z_{dn}})$.

The Dirichlet priors $\alpha$ and $\beta$ control sparsity: small $\alpha$ produces documents that concentrate on few topics; small $\beta$ produces topics that concentrate on few words.

Plate notation

Shaded nodes ($w$) are observed; all others are latent.

Inference: collapsed Gibbs sampling

Exact posterior inference is intractable; the standard approach is collapsed Gibbs sampling, which integrates out $\theta$ and $\phi$ analytically and samples only the topic assignments $\mathbf{z}$.

The conditional posterior for a single assignment $z_{dn} = k$ (given all others):

where $n_{dk}^{-dn}$ is the number of words in document $d$ assigned to topic $k$ (excluding position $n$), and $n_{kw}^{-dn}$ is the number of times word $w$ is assigned to topic $k$ (excluding position $n$).

Each iteration cycles over every word in the corpus and resamples its topic assignment. After burn-in, the averaged counts estimate $\theta$ and $\phi$.

Reading the output

After convergence:

• $\phi_k$: distribution over vocabulary for topic $k$. The top-10 words characterize the topic.
• $\theta_d$: distribution over topics for document $d$.

import numpy as np
from sklearn.decomposition import LatentDirichletAllocation
from sklearn.feature_extraction.text import CountVectorizer

docs = [
    "The president signed the new healthcare bill into law today.",
    "Stock markets rose sharply on strong earnings reports.",
    "Scientists discover a new species of deep-sea fish.",
    "Senate debates immigration reform and border security funding.",
    "Tech company reports record profits driven by cloud services.",
    "Marine biologists study coral reef bleaching in the Pacific.",
]

vectorizer = CountVectorizer(stop_words="english")
X = vectorizer.fit_transform(docs)
vocab = vectorizer.get_feature_names_out()

lda = LatentDirichletAllocation(n_components=3, random_state=0, max_iter=100)
lda.fit(X)

for k, phi in enumerate(lda.components_):
    top_words = [vocab[i] for i in phi.argsort()[-5:][::-1]]
    print(f"Topic {k}: {', '.join(top_words)}")

Non-negative Matrix Factorization

NMF factorizes the document-term matrix $\mathbf{X} \in \mathbb{R}^{D \times V}_{\ge 0}$ into two non-negative factors:

• $\mathbf{H}_{k,:}$: the word distribution for topic $k$ (analogous to $\phi_k$ in LDA).
• $\mathbf{W}_{d,:}$: the topic weight vector for document $d$ (analogous to $\theta_d$).

Non-negativity forces additive, parts-based decompositions: topics can only combine positively, which produces interpretable components.

Objective: minimize the Frobenius reconstruction error with a multiplicative update:

Multiplicative update rules (Lee and Seung, 2001):

Division and multiplication are elementwise. These updates guarantee non-negativity is preserved and the objective never increases.

from sklearn.decomposition import NMF

nmf = NMF(n_components=3, random_state=0, max_iter=500)
W = nmf.fit_transform(X)
H = nmf.components_

for k in range(3):
    top_words = [vocab[i] for i in H[k].argsort()[-5:][::-1]]
    print(f"NMF topic {k}: {', '.join(top_words)}")

LDA vs NMF

Exercises

Exercise 1. In collapsed Gibbs sampling, the unnormalized resampling weight for assigning word $w$ in document $d$ to topic $k$ is the product of a document-topic affinity and a topic-word affinity. A corpus has $K = 2$ topics, $V = 5$ vocabulary words, hyperparameters $\alpha = 0.1$, $\beta = 0.1$. Excluding the current word, document $d$ has counts $n_{d,0}^{-dn} = 4$, $n_{d,1}^{-dn} = 0$ (so $n_d^{-dn} = 4$); the current word $w$ has topic counts $n_{0,w}^{-dn} = 3$, $n_{1,w}^{-dn} = 0$, with topic totals $n_0^{-dn} = 20$, $n_1^{-dn} = 20$. Compute the unnormalized weight for each topic and the resulting probability of topic 0.

Exercise 2. NMF minimizes $\|\mathbf{X} - \mathbf{W}\mathbf{H}\|_F^2$ with the multiplicative update $\mathbf{H} \leftarrow \mathbf{H} \odot \frac{\mathbf{W}^\top \mathbf{X}}{\mathbf{W}^\top \mathbf{W} \mathbf{H}}$. Show that if $\mathbf{X}$, $\mathbf{W}$, and $\mathbf{H}$ start non-negative, the update keeps $\mathbf{H}$ non-negative, and that a fixed point of the update satisfies the stationarity condition $(\mathbf{W}^\top \mathbf{X})_{kj} = (\mathbf{W}^\top \mathbf{W} \mathbf{H})_{kj}$ wherever $\mathbf{H}_{kj} > 0$.

Exercise 3 (code). Implement NMF with the multiplicative updates in pure numpy on a small non-negative document-term matrix. Run 200 iterations and confirm the factors stay non-negative and the Frobenius reconstruction error decreases monotonically.

import numpy as np

rng = np.random.default_rng(0)
D, V, K = 6, 8, 3
# A non-negative document-term matrix built from latent parts so structure exists
W_true = rng.random((D, K))
H_true = rng.random((K, V))
X = W_true @ H_true

W = rng.random((D, K)) + 1e-3
H = rng.random((K, V)) + 1e-3
eps = 1e-9

errors = []
for _ in range(200):
    H = H * (W.T @ X) / (W.T @ W @ H + eps)
    W = W * (X @ H.T) / (W @ H @ H.T + eps)
    errors.append(np.linalg.norm(X - W @ H))

errors = np.array(errors)
assert (W >= 0).all() and (H >= 0).all()         # non-negativity preserved
# Error is non-increasing (allow a tiny numerical slack from the eps guard)
assert (np.diff(errors) <= 1e-6).all()
assert errors[-1] < errors[0]
print("start err:", round(errors[0], 4), " end err:", round(errors[-1], 6))

Both produce interpretable topics when $K$ is in the range 10-100 and documents are reasonably long. Modern LLMs have largely replaced topic models for semantic search (embeddings from E2 are more expressive), but topic models remain useful for exploratory analysis at scale with no labeled data.