Encoder-decoder transformers: T5 and BART

For tasks that map one sequence to another -- translation, summarization, question answering with generated answers -- the encoder-decoder architecture combines the full-context reading of an encoder with the sequential generation of a decoder[1].

Architecture

The encoder is a stack of bidirectional transformer blocks (no causal mask) that reads the source sequence and produces a sequence of contextual representations $\mathbf{h}^e \in \mathbb{R}^{S \times d}$.

The decoder is a stack of transformer blocks with two attention layers per block:

1. Masked self-attention: causal, so the decoder can only attend to already-generated tokens.
2. Cross-attention: attends over the encoder output $\mathbf{h}^e$.

Cross-attention

In each decoder block, the cross-attention mechanism:

• Queries come from the decoder's current hidden state.
• Keys and values come from the encoder output $\mathbf{h}^e$ (fixed for the whole decoding pass).
• No causal mask -- the decoder can attend to all encoder positions.

The cross-attention matrix $\alpha \in \mathbb{R}^{T \times S}$ shows which encoder positions each decoder position attends to -- like the Bahdanau attention matrix, but with the full Transformer's key-query parameterization.

T5: text-to-text transfer transformer

Raffel et al. (2020) unified all NLP tasks under a single text-to-text format: every task is a sequence-to-sequence problem where both input and output are plain text[2].

T5 pretrains with span corruption (a generalization of MLM): randomly select spans of tokens, replace each span with a single sentinel token <extra_id_k>, and train the model to reconstruct the original spans in the output.

T5 uses a relative positional encoding rather than absolute; the decoder uses causal self-attention with position bias.

BART: denoising sequence-to-sequence

Lewis et al. (2020) pretrain an encoder-decoder with a denoising objective: corrupt the source text with various noising functions, then reconstruct the original.

Noising functions (BART uses all of these during pretraining):

• Token masking: replace tokens with [MASK].
• Token deletion: delete tokens (decoder must infer their positions, unlike MLM).
• Text infilling: replace a span with a single [MASK] (generalizes token masking).
• Sentence permutation: shuffle sentences.
• Document rotation: rotate so the document starts at a random token.

The model is pretrained to output the original uncorrupted text. This pretrains both the encoder (to understand corrupted input) and the decoder (to generate text).

BART is especially strong on generation tasks (summarization, dialogue, translation) because the decoder is trained as a full language model on clean text.

Comparison

from transformers import T5Tokenizer, T5ForConditionalGeneration

tokenizer = T5Tokenizer.from_pretrained("t5-small")
model     = T5ForConditionalGeneration.from_pretrained("t5-small")

# Summarization
inputs = tokenizer(
    "summarize: The cat sat on the mat. The mat was red. The cat was black.",
    return_tensors="pt"
)
outputs = model.generate(**inputs, max_new_tokens=20)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

When to use encoder-decoder vs decoder-only

Encoder-decoder: supervised text-to-text tasks (translation, structured summarization) where the input and output have a clear boundary and are potentially different lengths. Cross-attention gives a direct pathway from source to target.

Decoder-only: open-ended generation, few-shot prompting, instruction following. Simpler architecture, easier to scale, and with instruction tuning can perform all encoder-decoder tasks via prompting.

Exercises

Exercise 1. In a decoder block, the masked self-attention uses a causal mask but the cross-attention does not. Explain why each choice is correct: what would break if the cross-attention were causally masked, and what would break if the self-attention were not?

Exercise 2. T5 frames sentiment classification as text-to-text, emitting the literal string positive or negative rather than a logit over two classes. Give one concrete advantage and one concrete cost of this choice compared to a BERT-style classification head on [CLS].

Exercise 3 (code). Implement decoder-to-encoder cross-attention directly: queries from a length-$T$ decoder state, keys and values from a length-$S$ encoder output, no causal mask. Verify the attention matrix has shape $T \times S$, each row is a distribution, and every decoder position attends to all $S$ encoder positions.

import numpy as np

def softmax(x, axis=-1):
    e = np.exp(x - x.max(axis=axis, keepdims=True))
    return e / e.sum(axis=axis, keepdims=True)

rng = np.random.default_rng(0)
T, S, d = 4, 7, 8          # 4 decoder positions, 7 encoder positions
dec_state = rng.standard_normal((T, d))
enc_out = rng.standard_normal((S, d))
Wq = rng.standard_normal((d, d))
Wk = rng.standard_normal((d, d))
Wv = rng.standard_normal((d, d))

Q = dec_state @ Wq          # queries from the decoder
K = enc_out @ Wk            # keys and values from the (fixed) encoder output
V = enc_out @ Wv
alpha = softmax(Q @ K.T / np.sqrt(d))   # (T, S), no mask
out = alpha @ V                          # (T, d)

assert alpha.shape == (T, S)
assert out.shape == (T, d)
assert np.allclose(alpha.sum(axis=1), 1.0)   # each decoder row sums to 1
assert np.all(alpha > 0)                     # attends to every source position
print("cross-attention matrix:", alpha.shape, "-> output:", out.shape)

Modern practice increasingly favors large decoder-only models for both tasks.