AI Knowledge TreeE1Classical NLP
7 lessons. Study in order; each one assumes the previous.
Classical NLP is where language modeling began: treat text as a sequence of tokens and count. Before any neural network, the discipline solved real problems (predicting the next word, tagging parts of speech, finding named entities, retrieving relevant documents) with probability tables, dynamic programming, and clever weighting schemes. The ideas here are not museum pieces; they are the foundations every modern system quietly reuses.
You will build n-gram language models from the Markov approximation and counting, fix their zero-probability holes with Laplace and Kneser-Ney smoothing, and measure quality with perplexity. You will train byte-pair encoding tokenizers and contrast WordPiece and SentencePiece, decode part-of-speech and entity tags with HMMs and Viterbi, sharpen them into discriminative CRFs, and rank documents with TF-IDF and BM25.
This matters because tokenization still feeds every LLM, BM25 still anchors hybrid retrieval in production RAG, and perplexity is the yardstick you carry through the rest of the pillar.
Before you start
- A2 Probability: distributions, conditional probability, and Bayes' rule.
- A2 Information theory: entropy and cross-entropy (perplexity builds on these).
- Comfortable Python with numpy for counting and small matrix operations.