How LLMs Are Trained

This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production.

This section’s bibliography lists the reference material consulted while preparing these pages.

Examples on this page target TRL (Transformer Reinforcement Learning) 1.14 and HF transformers 5.x, and the Chinchilla, InstructGPT, DPO and DeepSeekMath papers cited below, all verified as current on 2026-09-28. The previous pages in this sub-section covered what a model is (What Is an LLM?) and the architecture it runs (Transformer Architecture). This page covers how that architecture’s weights actually get set: the sequence of training stages that turns a randomly initialized network into a model that predicts text, then follows instructions, then reasons.

Pre-training: learning from raw text

Pre-training is the first and by far most expensive stage. A model with randomly initialized weights is trained, purely self-supervised, on next-token prediction over a very large corpus of raw text: no human writes a single label. The training signal is simply "predict the next token", applied at every position of every document in the corpus, at a scale of trillions of tokens.

Three things about pre-training data matter more than any individual architectural choice:

Scale — the corpus is assembled from web crawls (filtered and deduplicated), books, code repositories, scientific articles, and other large text sources, because next-token prediction only produces a broadly capable model when the data itself covers broad, varied knowledge and writing styles.
Quality filtering and deduplication — raw web text is noisy and highly repetitive; heuristic and model-based filters remove low-quality or boilerplate pages, and near-duplicate documents are removed so the model does not simply memorize a handful of pages that appear thousands of times.
Mixture weighting — different data sources (web text, code, books, reference material) are not sampled in their natural proportions; a training-data mixture up-weights sources known to improve downstream capability (code, for instance, is generally over-represented relative to its share of raw internet text) and down-weights or removes sources known to be harmful or low-value.

The resulting base model is a raw next-token predictor: it completes text plausibly, but it has no notion of "answer the user’s question" versus "continue whatever pattern comes next" — that behavior is what the later post-training stages instill.

Scaling laws and Chinchilla

Pre-training a model at a given scale is enormously expensive, so an obvious question is: for a fixed compute budget, how should that budget be split between a larger model and more training tokens? Kaplan et al.'s original scaling-law work (2020) suggested favoring model size heavily. Hoffmann et al.'s Chinchilla paper (2022) re-ran that analysis over more than 400 training runs and reached a different, now widely adopted, conclusion: model size and training-token count should scale at roughly equal rates.

The training compute (in FLOPs) of a transformer is well approximated as a function of only its parameter count and the number of training tokens :

This is a FLOPs approximation (about 6 floating-point operations per parameter per training token, forward plus backward pass), not the scaling result itself: it only defines the budget that and have to share. Chinchilla’s compute-optimal result comes from fitting a parametric loss to its training runs and minimizing it under that budget:

The paper’s fit gives , , , and . Its three estimation approaches put the exponents at and . Either way, for a fixed compute budget , loss is minimized when and are both scaled up together, rather than favoring one at the expense of the other. Chinchilla’s own calibration puts the resulting optimal ratio at about 20 tokens per parameter — for example, their 70-billion-parameter Chinchilla model was trained compute-optimally on roughly 1.4 trillion tokens. A model trained on far fewer tokens than that ratio (as many earlier large models were) is "undertrained" for its size: a smaller model trained on proportionally more tokens would have reached the same loss more cheaply.

Two caveats matter for reading this rule correctly. First, Chinchilla optimizes pre-training compute only; it says nothing about inference cost, and a production model is usually served far more times than it is trained, so many current models are deliberately trained well past the compute-optimal ratio ("overtrained") in exchange for a smaller, cheaper-to-serve model at deployment time. Second, the rule describes pre-training loss on raw text, not the downstream capability that post-training (below) later builds on top of it.

Supervised fine-tuning (SFT)

A base model completes text; it does not reliably follow instructions or hold a conversation, because nothing in next-token prediction over raw web text rewards that specific behavior. Supervised fine-tuning is the first post-training stage that changes this: the model is further trained, with the same next-token-prediction loss as pre-training, but on a much smaller, curated dataset of (prompt, ideal response) pairs written or selected specifically to demonstrate the target behavior — answering questions, following instructions, refusing unsafe requests, using a consistent conversational format.

SFT does not teach the model new facts or capabilities so much as it teaches it a behavior: given a prompt in the conversational format, produce a response in the style and structure the demonstrations show, drawing on knowledge the much larger pre-training stage already encoded. This is why SFT datasets are small (thousands to low hundreds of thousands of examples) compared with a pre-training corpus, and why SFT is comparatively cheap: it is a short fine-tuning run over a frozen architecture, not a from-scratch training run.

Preference tuning: RLHF and DPO

SFT alone leaves a gap: demonstrating one ideal response per prompt does not tell the model how to compare several possible responses and prefer the better one, and writing an ideal demonstration for every kind of prompt does not scale. Preference tuning closes that gap by training on comparisons instead: humans (or, increasingly, other models) are shown a prompt and two candidate responses, and record which one they prefer. InstructGPT (Ouyang et al., 2022) established the now-standard pipeline for using such comparisons, in two stages.

Reward modeling and RLHF

The first stage trains a reward model: a separate model, usually initialized from the SFT model itself, that takes a prompt and a response and outputs a scalar score. It is trained on the human-preference pairs directly, with the loss pushing the score of the preferred response ("winner") above the score of the dispreferred response ("loser") for the same prompt :

Here is the reward model’s scoring function and is the logistic sigmoid, so this is an ordinary binary-classification loss over which of the two responses scored higher. Once trained, the reward model stands in for a human rater: the language model is then optimized (with a reinforcement-learning algorithm, historically PPO — Proximal Policy Optimization) to produce responses that score highly under the reward model, while a KL-divergence penalty against the original SFT model keeps it from drifting into responses that game the reward model without actually being better. This full pipeline — reward model, then RL against it — is RLHF (Reinforcement Learning from Human Feedback).

DPO: preference tuning without a separate reward model or RL loop

RLHF’s two-stage pipeline (train a reward model, then run an RL loop against it) is effective but operationally heavy: it needs a second model, an RL training loop with its own instabilities, and careful tuning of the KL penalty. Direct Preference Optimization (Rafailov et al., 2023) reformulates the same preference-learning problem as a single, ordinary supervised loss on the language model itself, with no separate reward model and no RL loop:

is the model being trained and is a frozen reference copy (typically the SFT model before preference tuning); is an inverse-temperature that controls how strongly the loss penalizes deviation from . The key move is algebraic: DPO’s authors show that the reward function an RLHF reward model would learn can be re-expressed directly in terms of the policy’s own log-probabilities relative to the reference model, which is what lets the same preference pairs \((x, y_w, y_l)\) train the language model directly with a closed-form classification loss, shaped exactly like the reward-model loss above but applied to the policy itself instead of to a separate reward model.

RL for reasoning: RLVR and GRPO

RLHF and DPO both optimize for human preference, which is well suited to open-ended qualities such as helpfulness, tone or safety, but is a poor fit for tasks with an objectively checkable answer, such as a math problem or a piece of code that either passes its tests or does not. RLVR (Reinforcement Learning with Verifiable Rewards) instead scores a response with an automatic verifier — did the final numeric answer match, did the generated code pass its test suite — rather than a learned reward model trained on human comparisons, which removes the reward-model training stage (and the reward-hacking risk that comes with a learned, imperfect stand-in for human judgment) entirely for tasks where a verifier exists.

GRPO (Group Relative Policy Optimization), introduced in the DeepSeekMath paper (Shao et al., 2024) and used to train DeepSeekMath 7B to strong performance on the MATH benchmark, is the RL algorithm now closely associated with this style of training. Like PPO, it is a policy-gradient method, but it removes PPO’s separate value/critic model (which estimates how good a state is expected to be) entirely. Instead, for a given prompt, GRPO samples a group of several candidate responses from the current policy, scores each one with the verifiable reward, and computes each response’s advantage by normalizing its reward against the group’s own mean and standard deviation — a response that scores above the group average gets a positive advantage, one that scores below it gets a negative advantage, with no separate model needed to say what "average" should have been. This is what lets GRPO scale RL training to reasoning tasks without the memory and training-stability overhead of maintaining a full critic model alongside the policy, and it is the same family of RL training behind the "thinking" behavior covered in Reasoning Models.

Why post-training shapes behaviour

Every stage above changes a different aspect of what the model does, and later stages build on earlier ones rather than replacing them:

Pre-training determines what the model knows — the breadth of facts, language and patterns it can draw on — and does so through next-token prediction alone, with no notion of a conversation.
SFT determines the format and behavior of a response — that it answers instead of merely continuing text, and in what structure — without teaching new facts.
Preference tuning (RLHF or DPO) determines which of several acceptable responses is better — more helpful, better formatted, safer — by training directly on human comparisons rather than single demonstrations.
RL for reasoning (RLVR/GRPO) determines whether the model works through a problem before answering, trained against an automatic, verifiable signal rather than either kind of preference data.

A model released as, say, "instruction-tuned" or "reasoning" is naming which of these stages it went through beyond pre-training, and prompting it well depends on knowing which stages actually shaped its behavior: a base model needs to be prompted with a continuation it should complete, while a heavily preference-tuned or RL-trained model responds to direct instructions and goals.

Hands-on fine-tuning — actually running SFT or a small preference-tuning pass on a downloaded model, rather than reading about the stages in the abstract — is covered in Fine-Tuning with PEFT & TRL; this page stays at the level of what each stage is and why it exists.

Example: a conceptual SFT sketch with TRL

The following sketch shows the shape of an SFT run with Hugging Face’s TRL library — it is conceptual; see Fine-Tuning with PEFT & TRL for a complete, runnable walkthrough with a real dataset and model:

from datasets import load_dataset
from trl import SFTConfig, SFTTrainer

# a dataset of prompt/response pairs already in a "messages" chat format
dataset = load_dataset("json", data_files="docs_assistant_sft_examples.jsonl", split="train")

training_args = SFTConfig(
    output_dir="docs-assistant-sft",
    max_length=2048,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    num_train_epochs=1,
    learning_rate=2e-5,
)

trainer = SFTTrainer(
    model="Qwen/Qwen2.5-1.5B-Instruct",  # base or already-instruction-tuned model to continue training
    train_dataset=dataset,
    args=training_args,
)
trainer.train()

See the TRL SFTTrainer documentation for the current SFTConfig/SFTTrainer API this sketch follows, including dataset format requirements and the completion-only-loss default.

Example: inspecting a model’s chat template

SFT trains on prompt/response pairs rendered through the model’s chat template — the same Jinja template apply_chat_template uses at inference time — so the format the model is trained on is exactly the format it is later prompted with. Inspecting tokenizer.chat_template shows that format directly:

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-1.5B-Instruct")

print(tokenizer.chat_template)  # the raw Jinja template the model was trained/fine-tuned against

messages = [
    {"role": "system", "content": "You are DocsAssistant, an assistant that answers questions about this documentation site."},
    {"role": "user", "content": "What does ef_search control in an HNSW index?"},
    {"role": "assistant", "content": "ef_search sets how many candidates HNSW keeps while searching: higher is more accurate but slower."},
]

# add_generation_prompt=False, with the assistant turn included, renders the exact text an SFT example is trained on
rendered = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=False)
print(rendered)

See the HF transformers chat templating guide for apply_chat_template, tokenizer.chat_template and how a training pipeline renders conversational examples through it.

Pre-train to RL: the training pipeline

flowchart LR A[Pre-training\nnext-token prediction\non raw text] --> B[SFT\nprompt/response\ndemonstrations] B --> C[Preference tuning\nRLHF reward model + PPO\nor DPO] C --> D[RL for reasoning\nRLVR / GRPO]