D3Sequence modeling

Sequence modeling is about data where order carries meaning: language, time series, audio, code. The central difficulty is reach, letting a model relate a token to context that may sit hundreds of steps away without losing or exploding the signal in between. This track traces how the field solved that problem and arrived at the architecture behind every modern language model.

You will start with word embeddings, then build recurrent networks and watch their gradients vanish over long ranges, fix that with LSTM and GRU gates, and add attention to encoder-decoder translation. From there self-attention and the full Transformer come into focus, along with positional encodings (sinusoidal, ALiBi, RoPE) and the newer linear-time alternatives: state space models like Mamba, sub-quadratic attention including FlashAttention, and hybrid designs such as Jamba and Griffin.

The Transformer you assemble here is the same backbone used across NLP, modern reinforcement learning, and nearly all of AI engineering, which makes this one of the most load-bearing disciplines in the tree.