How LLMs are built
A large language model is built in stages: it is first pretrained to predict the next token on a very large amount of text, then tuned on example conversations and on human preferences so that it answers like an assistant.
Last updated: 08 Oct, 2026
The chat models used in this course went through those stages before they reached an API. Knowing them explains why a model continues text, why it follows instructions and why its answers vary from run to run.
Turning text into tokens
A model does not read letters or whole words. A tokenizer splits text into pieces called tokens, often a short word or part of a word, and replaces each piece with a number from a fixed vocabulary. The model works only with those numbers. The usual method, byte-pair encoding, is taught in Subword tokenization (BPE and WordPiece).
Pretraining: predicting the next token
Pretraining gathers a very large amount of text from web pages, books and code, and trains the model on one task: given the tokens so far, predict the next one. Each guess is compared with the real next token, and the model's numbers, called weights, are adjusted to make the real token more likely. Repeated over a huge amount of text, the model has to learn grammar, facts and patterns of reasoning in order to guess well. The result is a base model: it continues any text you give it, but it does not yet answer questions.
A real model does this with a neural network, the transformer, covered in Transformer architecture. The idea itself fits in a table of counts. The code below trains the smallest possible version on a few short sentences.
Counting what follows each word
from collections import Counter, defaultdict
text = "the cat sat on the mat . the dog sat on the rug . the cat saw the dog ."
words = text.split()
follows = defaultdict(Counter) # follows["the"] counts the words that come after "the"
for current, nxt in zip(words, words[1:]):
follows[current][nxt] += 1Turning counts into probabilities
options = follows["the"]
total = sum(options.values())
probabilities = {word: round(count / total, 2) for word, count in options.items()}Measuring how well the table predicts
Training lowers the average surprise of the real next word. The surprise of one guess is the negative log of the probability the model gave to the word that followed in the text, and the average over all the guesses is the loss. Guessing at random among the vocabulary gives a loss of log of the vocabulary size.
import math
pairs = list(zip(words, words[1:]))
loss = -sum(math.log(follows[a][b] / sum(follows[a].values())) for a, b in pairs) / len(pairs)
random_loss = math.log(len(set(words)))Generating by sampling
import random
random.seed(7)
word, generated = "the", ["the"]
for _ in range(8):
options = follows[word]
word = random.choices(list(options), weights=list(options.values()))[0]
generated.append(word)Training and sampling a word-count model
All four pieces together, with the printouts that show each step. This table has no hidden layers; it is a stand-in that makes the training objective visible, not a neural network.
import math
import random
from collections import Counter, defaultdict
text = "the cat sat on the mat . the dog sat on the rug . the cat saw the dog ."
words = text.split()
follows = defaultdict(Counter)
for current, nxt in zip(words, words[1:]):
follows[current][nxt] += 1
total = sum(follows["the"].values())
print({w: round(c / total, 2) for w, c in follows["the"].items()})
pairs = list(zip(words, words[1:]))
loss = -sum(math.log(follows[a][b] / sum(follows[a].values())) for a, b in pairs) / len(pairs)
print("loss", round(loss, 3), "vs", round(math.log(len(set(words))), 3), "for random guessing")
random.seed(7)
word, generated = "the", ["the"]
for _ in range(8):
options = follows[word]
word = random.choices(list(options), weights=list(options.values()))[0]
generated.append(word)
print(" ".join(generated)){'cat': 0.33, 'mat': 0.17, 'dog': 0.33, 'rug': 0.17}
loss 0.566 vs 2.197 for random guessing
the cat sat on the dog sat on theReading the word-count model
- The first line is what the table learned about the word after
the:catanddogeach followed it in a third of the cases,matandrugin a sixth. - The loss line compares the table with random guessing. Guessing among the nine distinct tokens of this text gives 2.197, and the trained table gives 0.566: the words that followed in the text were far more probable under the table. Pretraining a language model lowers this same number, on far more text and with far more weights.
- The last line is text sampled from the table, one word at a time, with a fixed seed so it repeats. Every pair of neighbouring words in it occurred in the training text, and the sentence as a whole means nothing: the table has no idea what a cat is.
Instruction tuning: from continuing text to following instructions
A base model asked "What is the capital of France?" may continue with more quiz questions, because that is a likely continuation. Instruction tuning trains the base model further on many example prompts paired with the replies a good assistant would give, in a chat format with system, user and assistant roles. After it, the model treats a prompt as a request.
Preference tuning: learning what people prefer
Several replies can follow instructions and still differ in quality. In reinforcement learning from human feedback (RLHF), people rank candidate replies, a reward model learns to predict those rankings, and the language model is trained to produce replies that score higher. The InstructGPT paper, Training language models to follow instructions with human feedback, describes this recipe: instruction examples first, then a reward model from human rankings, then reinforcement learning. A later method, Direct Preference Optimization, learns from pairs of preferred and rejected replies without a separate reward model.
What the stages mean for the apps you build
- Replies vary because the next token is sampled. The setting that controls how much is called temperature, taught in Temperature.
- A model knows what its training text contained and nothing newer, which is why Retrieval-augmented generation (RAG) and tools exist.
- The chat roles come from tuning. The system, human and AI messages in Messages and prompts are the format the model was tuned on.
- Tuning continues after release. Fine-tuning LLMs covers adapting a model to your own examples, and Quantization covers making it smaller.
Pretraining vs instruction tuning vs preference tuning
| Pretraining | Instruction tuning | Preference tuning | |
|---|---|---|---|
| Learns from | A very large amount of raw text and code | Example prompts with good replies | Rankings or pairs of replies |
| Goal | Predict the next token | Follow an instruction | Prefer the replies people rate higher |
| Result | A base model that continues text | A model that answers requests | A model whose answers people prefer |
| Relative cost | By far the largest | Small beside pretraining | Small beside pretraining |
Where you use what the stages explain
- Choosing a model: a name ending in instruct or chat is tuned to follow instructions, and a base model is for continuing text or for further training.
- Deciding how to change behaviour: change the prompt first, add documents with RAG for facts, and fine-tune only when those are not enough.
- Reading a model card: it lists the training data, the tuning steps and the cut-off of the model's knowledge.
Related
- Previous: Generative AI vs AI agents vs agentic AI
- Next: LangChain essentials
- Go deeper: Transformers and Next-token prediction
- Change the text in the example to three different sentences and print
follows["the"]. How does the loss change? - Change
random.seed(7)torandom.seed(8)and generate again. Why is the sentence different? - Start the generation from
"cat"instead of"the".
This is what real progress feels like.