D7Training systems

Training systems is the engineering that turns a working architecture into a model you can train at billion-parameter scale without running out of memory or waiting weeks. Once models outgrow a single GPU, the bottleneck shifts from math to systems: numeric precision, memory budgets, and how computation is split across many devices and kept in sync.

You will learn mixed-precision training with fp16 or bf16 and loss scaling, trade compute for memory with gradient checkpointing, and shard work across devices through data parallelism (DDP, FSDP, ZeRO) and model parallelism (pipeline and tensor splits like GPipe and Megatron). The track also covers compilation through torch.compile, XLA, and Triton, the IO-aware tiling behind FlashAttention, and the cluster infrastructure (storage, schedulers, NCCL collectives, observability) that holds a large training run together.

This is mostly conceptual and infrastructure-flavored, but it is what separates research code from systems that actually train modern models, and it leads naturally into the MLOps and serving concerns of later pillars.