Video generation: Sora and Veo

Text-to-image generation maps a prompt to a single image. Video generation extends this to a sequence of temporally coherent frames: the model must maintain consistent physics, scene appearance, and object identity across time while producing smooth motion[1].

The video generation challenge

Three new difficulties beyond image generation:

Temporal consistency: objects must maintain their appearance across frames. A dog cannot spontaneously change breed or color between seconds 2 and 3.

Motion modeling: realistic physics -- gravity, inertia, fluid dynamics -- must be captured. A ball thrown in frame 1 must follow a plausible trajectory in frames 2-30.

Compute: a 5-second 1080p video at 24fps is $5 \times 24 = 120$ images. At the quality of a diffusion model, this is $120\times$ the compute of a single image. The spatial-temporal attention matrix scales as $O(T^2)$ in sequence length, which at video resolution becomes infeasible without careful design.

Spacetime patches: the key abstraction

Both Sora and Veo frame video as a sequence of spacetime patches -- 3D volumetric patches covering both space and time:

• Spatial patch size: $p_s \times p_s$ pixels (e.g., $8 \times 8$).
• Temporal patch size: $p_t$ frames (e.g., $2$ frames).

A $256 \times 256 \times 16$ frame video (16 frames, 256px) compressed to latents at $32 \times 32 \times 2$ latents becomes: $\frac{32}{4} \times \frac{32}{4} \times \frac{2}{2} = 8 \times 8 \times 1 = 64$ tokens with 4x spatial patch size and 2x temporal patch size.

This is the same operation as ViT's image patching, extended to 3D.

Sora (OpenAI, 2024)

Architecture: a Diffusion Transformer (DiT) operating on spacetime patches in a compressed latent space, the same latent-diffusion idea Stable Diffusion uses for images, extended to spacetime[2].

Key technical choices:

• Variable duration and resolution: the model is trained on videos of many different lengths, resolutions, and aspect ratios. The patch tokenizer handles variable-length sequences without resizing. This enables the model to generate any duration at inference.
• Large model scale: rumored 3B+ parameters, trained on hundreds of millions of video clips with captions (generated by a captioning model).
• Text conditioning: T5-XXL or similar large text encoder; cross-attention at every DiT block.

Capabilities (from the technical report):

• Up to 60 seconds at 1080p.
• Strong understanding of physics, object permanence, and camera motion.
• Can simulate camera panning, zooming, and multi-character scenes.

Limitations: Sora still fails on long-range physics (falling objects sometimes hover), complex hand motions, and interactions between many agents.

Veo (Google DeepMind, 2024)

Veo (and its successors Veo 2, Veo 3) is Google's video generation model. Key reported characteristics:

• Also uses a DiT architecture with spacetime patches.
• Stronger cinematic quality and camera control commands.
• Can incorporate reference images for character consistency.
• Veo 3 adds audio generation (synchronized sound effects and speech).

Both Sora and Veo outperform earlier diffusion-based video models (ModelScope, CogVideo, AnimateDiff) by a large margin on temporal consistency and motion realism.

Training challenges

Data: high-quality, high-resolution, diverse video with dense captions requires a massive data pipeline. Most training data comes from YouTube, stock video, and movie clips; captions are auto-generated.

Compute: training a Sora-scale model requires thousands of H100s for months.

Evaluation: automatic metrics (FVD -- Frechet Video Distance, CLIPSIM) correlate weakly with human preferences. Human evaluation is expensive; generating reference videos is non-trivial.

Temporal alignment: the model must learn not just what scenes look like, but how they change over time. Position encodings must handle 3D (space + time); most models use 3D RoPE or factored 2D spatial + 1D temporal encodings.

Open-source video generation

CogVideoX (Zhipu AI, 2024): 5B parameter DiT-based video model; generates 10-second clips at 720p. Open weights.

Wan2.1 (Alibaba, 2025): 14B parameter model with strong text-to-video quality; open weights available for research.

Mochi-1 (Genmo, 2024): 10B parameter open model with state-of-the-art motion quality.

These open models enable research and fine-tuning, but production-quality video generation at the level of Sora or Veo remains in the proprietary domain as of 2025.

Exercises

Exercise 1. A 5-second clip at 24 fps and a 10-second clip at 24 fps are both generated. How many frames does each contain? If each frame costs the same as one diffusion image, by what factor is the 10-second clip more expensive than a single image, and how much more expensive than the 5-second clip?

Solution

Step 1. Frames equal duration times frame rate. The 5-second clip is $5 \times 24 = 120$ frames; the 10-second clip is $10 \times 24 = 240$ frames.

Step 2. At one-image cost per frame, the 10-second clip is $240\times$ a single image, and the 5-second clip is $120\times$.

Step 3. Between the two clips the ratio is $240 / 120 = 2$, exactly the duration ratio, because frame count is linear in time. This per-frame counting is the optimistic bound; spatiotemporal attention adds a super-linear term on top, which the next exercises probe.

Exercise 2. A $256 \times 256$ video of 16 frames is compressed by the video VAE to $32 \times 32 \times 2$ latents. The patchifier uses a 4x spatial patch and a 2x temporal patch. How many spacetime tokens enter the DiT? If the spatial resolution doubles to $512 \times 512$ (latents $64 \times 64 \times 2$) with the same patch sizes, how many tokens result, and how does the DiT's self-attention cost change?

Solution

Step 1. Tokens are the latent grid divided by the patch sizes along each axis:

$$\frac{32}{4} \times \frac{32}{4} \times \frac{2}{2} = 8 \times 8 \times 1 = 64 \text{ tokens}.$$

Step 2. Doubling spatial resolution gives latents $64 \times 64 \times 2$, so

$$\frac{64}{4} \times \frac{64}{4} \times \frac{2}{2} = 16 \times 16 \times 1 = 256 \text{ tokens},$$

a $4\times$ increase in sequence length (each spatial side doubled, so $2 \times 2 = 4$).

Step 3. Self-attention is $O(T^2)$ in the token count, so cost scales as $(256 / 64)^2 = 4^2 = 16\times$. A 2x resolution bump costs 16x more attention, which is why video models lean on aggressive VAE compression and large patches before the transformer.

Exercise 3. Position encodings for video must handle three axes (two spatial, one temporal). One design uses full 3D RoPE; another factors it into 2D spatial plus 1D temporal encodings. State one advantage of the factored approach for the "variable duration and resolution" training Sora relies on, and one risk it carries compared to a joint 3D encoding.

Solution

Step 1. Factoring separates the spatial encoding from the temporal one, so the same spatial encoding applies at any clip length and the temporal encoding extends independently. This lets the model train and generate at variable durations and resolutions without relearning positions, matching Sora's variable-length patch sequences.

Step 2. The cost is expressiveness: a factored encoding treats space and time as separable, so it represents couplings between a position and a moment less directly than a joint 3D encoding. Phenomena where spatial location and timing interact (an object's motion trajectory) are harder to encode crisply, risking weaker long-range physics, exactly the failure mode the lesson notes for falling objects that hover.

Step 3. The trade-off is generalization across duration and resolution (factored) versus richer spatiotemporal coupling (joint 3D), and most systems favor the factored or 3D-RoPE designs because variable-length training is the harder constraint to give up.

These spacetime-token and position-encoding choices are what let one diffusion transformer span images and minute-long video, closing the multimodal arc this pillar opened with CLIP.

---

References

1. [1]Ho, J., et al. (2022). "Video diffusion models". NeurIPS. arXiv:2204.03458↩
2. [2]Rombach, R., et al. (2022). "High-resolution image synthesis with latent diffusion models (Stable Diffusion)". CVPR. arXiv:2112.10752↩