Supervised fine-tuning

A pretrained language model generates plausible continuations of text -- it is a good completion engine. It is not (yet) an instruction follower: ask it a question and it may continue generating questions rather than answering them. Supervised fine-tuning (SFT) bridges this gap by training on (instruction, response) pairs where responses are human-written or human-curated[1].

The SFT objective

SFT is standard next-token prediction, but restricted to the completion tokens:

The prompt tokens are included in the context (the model reads them) but are not included in the loss (the model is not penalized for the prompt text). Only the model's response tokens count.

This is implemented by applying a mask to the loss: zero out positions corresponding to the prompt, compute cross-entropy only on response positions.

Data format

SFT data consists of conversations formatted with role markers:

<|system|>
You are a helpful, harmless, and honest assistant.
<|user|>
Explain the difference between precision and recall.
<|assistant|>
Precision measures how many of your positive predictions were actually positive...
<|end|>

The exact format (special tokens, delimiters) varies by model family. What matters: the model learns to recognize the structure and generate the assistant turn.

Data quality over quantity

SFT with a small number of high-quality examples often outperforms SFT with a large number of mediocre ones.

• LIMA (Zhou et al., 2023): 1000 carefully curated examples achieved near-RLHF-quality instruction following on many tasks.
• Alpaca vs WizardLM: WizardLM's "Evol-Instruct" (LLM-generated more complex variants of simple instructions) significantly outperformed the original Alpaca dataset (52K GPT-3 distillation examples) despite similar size.

Sources of SFT data:

• Human demonstrations (expensive but highest quality).
• LLM distillation: generate responses with a stronger teacher model (GPT-4, Claude), curate.
• RLHF-selected responses: use a reward model to select good completions.

Formatting matters

SFT implicitly teaches the model the conversation format it should follow at inference. Inconsistent formatting (different system prompt styles, missing end tokens) in the SFT data leads to format errors at inference.

Catastrophic forgetting

Fine-tuning on SFT data can degrade capabilities learned during pretraining if:

• The SFT dataset is too small or domain-narrow.
• The learning rate is too high.

Mitigations:

• Mix a small fraction of pretraining data into SFT training.
• Use low learning rates (1e-5 to 5e-5).
• LoRA fine-tuning (next lesson) avoids forgetting by keeping most pretrained weights frozen.

SFT vs prompting

An alternative to SFT is few-shot prompting: include examples of (instruction, response) in the prompt context and let the base model continue. SFT is preferable when:

• The desired format is consistent and learnable.
• Response quality matters more than cost (SFT bakes the style in rather than spending tokens on examples every inference).
• The target behavior is hard to specify through examples alone.

from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments, Trainer
from datasets import Dataset

model_name = "meta-llama/Llama-3.2-1B"
tokenizer  = AutoTokenizer.from_pretrained(model_name)
model      = AutoModelForCausalLM.from_pretrained(model_name)

sft_data = [
    {"instruction": "What is backpropagation?",
     "response": "Backpropagation is the algorithm used to compute gradients..."},
    {"instruction": "Explain gradient descent.",
     "response": "Gradient descent is an optimization algorithm that..."},
]

def format_and_tokenize(example):
    prompt = f"<|user|>\n{example['instruction']}\n<|assistant|>\n"
    full   = prompt + example["response"] + tokenizer.eos_token
    tokens = tokenizer(full, truncation=True, max_length=512)
    # Mask the prompt tokens in labels
    prompt_len  = len(tokenizer(prompt)["input_ids"])
    labels      = [-100] * prompt_len + tokens["input_ids"][prompt_len:]
    tokens["labels"] = labels
    return tokens

dataset = Dataset.from_list(sft_data).map(format_and_tokenize)
args    = TrainingArguments(output_dir="sft_out", num_train_epochs=3, learning_rate=2e-5)
trainer = Trainer(model=model, args=args, train_dataset=dataset)
trainer.train()

Exercises

Exercise 1. An SFT example has a prompt of 12 tokens and a response of 8 tokens. The cross-entropy loss is averaged over the positions that contribute to the loss. The model assigns probability $0.5$ to each of the 8 response tokens (and whatever it likes to the prompt tokens). Compute the per-example SFT loss in nats, and state how many positions enter the average.

Exercise 2. Two SFT runs use the same data and model. Run A averages the loss over all tokens (prompt + response); run B averages over response tokens only (the correct masking). For a batch where prompts are on average 3 times longer than responses, and the model already predicts prompt tokens well (low loss $\approx 0.2$ nats there) but predicts response tokens poorly (loss $\approx 2.0$ nats there), compute both reported losses and explain which run trains the response behavior harder.

Exercise 3 (code). Implement the masked SFT loss in numpy. Given per-position next-token log-probabilities and a boolean mask marking response positions, compute the mean cross-entropy over response positions only. Verify on the Exercise 1 numbers that prompt positions do not change the result.

import numpy as np

def sft_loss(token_logprobs, response_mask):
    """token_logprobs: (T,) log p assigned to the realized next token at each position.
       response_mask: (T,) bool, True where the position is a response token."""
    token_logprobs = np.asarray(token_logprobs, float)
    response_mask  = np.asarray(response_mask, bool)
    nll = -token_logprobs                       # per-position cross-entropy
    return nll[response_mask].mean()            # average over response positions only

# 12 prompt tokens + 8 response tokens. Response tokens each get log p = log 0.5.
T_prompt, T_resp = 12, 8
logp = np.concatenate([
    np.full(T_prompt, np.log(0.9)),             # prompt positions, arbitrary
    np.full(T_resp,  np.log(0.5)),              # response positions
])
mask = np.concatenate([np.zeros(T_prompt, bool), np.ones(T_resp, bool)])

loss = sft_loss(logp, mask)
assert np.isclose(loss, -np.log(0.5))           # = log 2 ~ 0.6931, independent of prompt

# Changing the prompt log-probs must not move the masked loss.
logp2 = logp.copy()
logp2[:T_prompt] = np.log(0.01)
assert np.isclose(sft_loss(logp2, mask), loss)
print("masked SFT loss:", round(loss, 4), "nats")

SFT is the first of three stages in RLHF[2]. The SFT model is the starting point for reward modeling and policy optimization, which the next lessons derive.