Self-supervised vision (DINO, MAE)

Labeled image datasets are expensive; unlabeled images are essentially free. Self-supervised pretraining turns this asymmetry into model quality: pretrain a vision Transformer[2] on hundreds of millions of unlabeled images, then fine-tune on a small labeled set. Two methods dominate: DINO (student-teacher distillation, no labels needed) and MAE (masked image modeling, BERT for pixels). The D5 SSL track covers the general theory; this lesson is the vision-specific story.

Why SSL for vision

Supervised ImageNet pretraining was the standard for years. Its limits:

• 1.3M images is a small dataset by foundation-model standards.
• Many real-world tasks have a domain gap from ImageNet.
• ImageNet labels are noisy (~10% errors at the class level).

Self-supervised pretraining on much larger unlabeled corpora (LAION, internal image dumps) consistently beats supervised pretraining on downstream transfer, while needing no labels at all.

DINO

Caron et al. (2021), Facebook AI Research.

Setup: two networks, the student $g_\theta$ and the teacher $g_\xi$, both vision Transformers. They receive different augmented crops of the same image and must produce similar features. The teacher's parameters are an exponential moving average of the student's; this prevents collapse without negative pairs.

Loss: cross-entropy between the student's and teacher's softmaxed features, where each feature vector is interpreted as a probability distribution. Centering of the teacher's features (subtracting a running mean) and temperature sharpening keep the representations informative.

DINO produces remarkable emergent properties:

• The pretrained [CLS] token attention highlights the object of interest, even though the model was never told what the object is.
• k-NN classification on DINO features matches supervised ImageNet classification at 78% accuracy without any labels.
• Linear probes on top of frozen DINO features approach supervised fine-tuning accuracy.

DINOv2 (2023) scaled this to a curated 142M-image dataset and produced general-purpose visual features used as backbones for classification, segmentation, depth, and 3D tasks. Closest vision has to a general-purpose feature extractor.

MAE (Masked Autoencoders)

He et al. (2022). The image analogue of BERT's masked language modeling[1].

Setup: mask 75% of the image patches at random. An asymmetric encoder-decoder ViT:

• The encoder sees only the unmasked 25% of patches (much cheaper than processing all patches).
• The decoder (small, ~8 layers) receives the encoder's output plus learned mask tokens at masked positions and predicts the original pixel values.

Loss: mean squared error on pixel values of the masked patches.

Why it works:

• The 75% mask rate forces the model to learn high-level semantic features (low-level texture is not enough to fill in 75% of an image).
• The asymmetric encoder is efficient: it processes a quarter of the patches in the forward pass, making 1.6B-parameter MAEs trainable with modest compute.

After pretraining, throw away the decoder and use the encoder as a backbone. Fine-tuning matches or beats DINO and supervised pretraining on classification and detection.

SimCLR / MoCo / iBOT (background)

Other SSL methods used in vision before DINO and MAE:

• SimCLR (2020): contrastive learning with augmented pairs (similar views close, others apart). Needs huge batches and many negatives.
• MoCo (2020): contrastive with a memory bank of negatives; works at smaller batches.
• BYOL (2020): student-teacher without negatives, similar to DINO.
• iBOT (2022): combines DINO-style distillation with masked-image prediction for the best of both.

The D5 SSL track covers the family broadly; here the focus is on the practical vision-specific recipe.

Practical recipe

For a typical vision project:

1. Start with DINOv2 or MAE pretrained on a large corpus. Freeze the backbone, train a small task head. Excellent for fast experimentation.
2. Fine-tune the whole backbone for your task when you have enough data and compute. Improves performance.
3. Train SSL from scratch on your domain only if the domain is far from natural images (medical, satellite, microscopy) and you have millions of images.

For most teams, option 1 is what works. DINOv2 weights, MAE weights, and CLIP image-encoder weights are all reasonable starting points.

Exercises

Exercise 1. MAE masks $75\%$ of patches and its encoder processes only the visible $25\%$. Self-attention is $O(N^2)$ in the token count $N$. By what factor is the encoder's attention cost reduced relative to processing all $N$ patches?

Solution

Step 1. A full encoder attends over all $N$ tokens, costing $\propto N^2$.

Step 2. The MAE encoder sees only the visible fraction $0.25 N$, so its attention cost is $\propto (0.25 N)^2 = 0.0625 N^2$.

Step 3. The ratio is $\frac{0.0625 N^2}{N^2} = 0.0625 = \frac{1}{16}$, a $16\times$ reduction in attention cost. Dropping the masked tokens from the encoder (rather than feeding them as mask tokens) is what makes large MAEs cheap to pretrain; the masked positions reappear only in the small decoder.

Exercise 2. In DINO the teacher's parameters are an exponential moving average of the student's, and the loss pushes the student to match the teacher on different crops of the same image. A naive shortcut would make both networks output the same constant vector for every image (a collapse that drives the matching loss to zero). Explain the role of teacher centering and sharpening in preventing this.

Solution

Step 1. The loss is minimized trivially if every output is the same distribution regardless of input: the student and teacher then always agree. This degenerate "constant output" solution carries no information about the image, so it must be blocked.

Step 2. Centering subtracts a running mean of the teacher's outputs before the softmax. If one coordinate starts to dominate every output (the route to a constant), centering removes that shared component, penalizing the collapse toward a single direction.

Step 3. Sharpening (a low softmax temperature on the teacher) pushes the teacher's target toward a confident, peaked distribution. Centering alone could instead collapse to the uniform distribution; sharpening counteracts that by rewarding peaked, input-specific targets. The two forces balance: centering blocks the one-coordinate collapse, sharpening blocks the uniform collapse, so the only stable solution is informative, input-dependent features. The EMA teacher provides a stable, slowly-moving target so this balance does not oscillate.

Exercise 3. DINO learns by matching features across two crops; MAE learns by reconstructing masked pixels. A practitioner wants frozen features that work well under a $k$-NN or linear probe with no fine-tuning. Which method's training objective is better aligned with that use, and why?

Solution

Step 1. A $k$-NN or linear probe judges features by how well distance or a linear boundary separates classes in the frozen embedding space, so the goal is an embedding whose geometry already groups semantically similar images.

Step 2. DINO's objective directly shapes this geometry: it forces two augmented crops of the same image to map to nearby feature distributions, which clusters views of the same content together and pushes the [CLS] embedding toward a semantic summary. This is why DINO reports strong $k$-NN accuracy without labels.

Step 3. MAE optimizes pixel reconstruction of masked patches. Its encoder learns features useful for filling in pixels, which are excellent after fine-tuning but not necessarily linearly separable when frozen, since nothing in the objective enforces a clean class geometry in the raw embedding. For frozen-feature probing, DINO's invariance objective is better aligned; for fine-tuned transfer, MAE is competitive or better.

The big picture

Modern vision moved from "supervised ImageNet pretraining" to "self-supervised pretraining on much larger unlabeled corpora", with fine-tuning on smaller labeled task datasets. SSL methods are now the default upstream training; supervised fine-tuning is downstream.

The next (and final) D2 lesson moves to a completely different vision application: 3D scene reconstruction from images, via NeRF and Gaussian Splatting.

---

References

1. [1]He, K., Chen, X., Xie, S., Li, Y., Dollár, P., & Girshick, R. (2022). "Masked autoencoders are scalable vision learners (MAE)". CVPR. arXiv:2111.06377. Masked image modeling with an asymmetric encoder-decoder.↩
2. [2]Dosovitskiy, A., et al. (2021). "An image is worth 16x16 words: transformers for image recognition at scale (ViT)". ICLR. arXiv:2010.11929. The vision Transformer backbone these SSL methods pretrain.↩