E7Multimodal

6 lessons. Study in order; each one assumes the previous.

Multimodal models break language out of text alone, joining it with images, audio, and video so a single system can see, hear, and generate across modalities. The central trick is alignment: mapping different kinds of data into a shared representation where a caption and its picture, or a question and the image it asks about, can be compared and reasoned over together.

You will train CLIP's joint image-text encoders with a contrastive InfoNCE loss and watch zero-shot classification fall out of cosine similarity. From there you bridge a frozen vision encoder to an LLM with a projection layer (LLaVA) or a Q-Former (BLIP-2), fine-tune large multimodal models, and handle audio with Whisper and Wav2Vec. The discipline closes on generation: text-to-image diffusion with classifier-free guidance (DALL-E, Stable Diffusion) and text-to-video with spacetime diffusion transformers (Sora, Veo).

This connects the pillar back to vision and generative modeling in pillar D, and it is the direction frontier assistants are moving as they become natively multimodal.

Before you start

E3 Transformer LMs: encoder-only and decoder-only models.
D5 Self-supervised learning (contrastive methods) and D4 generative modeling (diffusion).
E4 Supervised fine-tuning for the multimodal fine-tuning lessons.

Start with lesson 1: CLIP: contrastive vision-language pretraining →