E3Transformer LMs

Transformer language models are the architecture behind essentially every modern LLM, and this discipline builds one from the attention operation up. It replaces recurrence with a mechanism that lets every token attend to every other token in parallel, which is what made training at today's scale possible.

You will derive scaled dot-product attention (including why the 1/sqrt(d_k) factor is there), stack heads into multi-head attention, and assemble a full transformer block with residual connections and layer norm, tracing tensor shapes through a forward pass. You will add positional information from sinusoidal encodings up to RoPE and ALiBi, then specialize the block into the three families: decoder-only GPT models with causal masking, encoder-only BERT with bidirectional masked pretraining, and encoder-decoder T5 and BART with cross-attention. The discipline closes with the efficiency tricks production models rely on: grouped-query and multi-query attention, and mixture-of-experts routing.

This is the load-bearing center of the pillar: training, inference, agents, and multimodal models all assume you understand the block built here.