E4Training LLMs

Training LLMs is the pipeline that turns raw web text into a model that follows instructions and respects human preferences. It answers the questions that decide whether a base model becomes something useful: what data to train on, how big to make it for a given compute budget, and how to align its behavior after pretraining.

You will study data curation (quality filtering, deduplication, domain mixing), then the Kaplan and Chinchilla scaling laws that govern how to spend compute between parameters and tokens. From there you move through alignment: supervised fine-tuning, Bradley-Terry reward modeling, the four-model RLHF loop with PPO's clipped objective and KL penalty, and the offline preference methods (DPO, IPO, KTO, ORPO) that drop the reward model. You will also cover GRPO behind reasoning models, Constitutional AI and RLAIF, and parameter-efficient fine-tuning with LoRA, QLoRA, and DoRA.

This is where the abstract Transformer becomes a deployable assistant, and the alignment ideas here reappear directly in pillar H's RL and pillar J's applied fine-tuning.