D5Self-supervised learning

Self-supervised learning is the answer to a practical bottleneck: labels are expensive, but raw data is nearly free. The idea is to invent a pretext task the data labels itself, so a network can learn rich, general-purpose representations from unlabeled images or text and then transfer them to downstream tasks with very little supervision.

You will pretrain representations contrastively by pulling augmented views together with the InfoNCE loss (SimCLR, MoCo), extend that across modalities to align images and captions in CLIP, and see how non-contrastive methods like BYOL and DINO drop negatives entirely through predictors and student-teacher distillation. On the masked-prediction side you will work through BERT-style masked language modeling and its image analogue, MAE-style masked image modeling.

This paradigm is why foundation models exist at all: nearly every large pretrained encoder, and the embeddings that power retrieval and multimodal systems, comes out of a self-supervised objective like the ones here.