Generative AI and Agentic AILangChain 1.4 · LangGraph 1.2 · Python 3.12 or 3.13
Dashboard
0%

How LLMs are built

A large language model is built in stages: it is first pretrained to predict the next token on a very large amount of text, then tuned on example conversations and on human preferences so that it answers like an assistant.

Last updated: 08 Oct, 2026

The chat models used in this course went through those stages before they reached an API. Knowing them explains why a model continues text, why it follows instructions and why its answers vary from run to run.

Six stages in a row: text data, a tokenizer, pretraining by predicting the next token, instruction tuning on example prompts and replies, preference tuning on which replies people prefer, and the finished chat model. Pretraining builds a base model that continues text; the two tuning stages turn it into an assistant.

Turning text into tokens

A model does not read letters or whole words. A tokenizer splits text into pieces called tokens, often a short word or part of a word, and replaces each piece with a number from a fixed vocabulary. The model works only with those numbers. The usual method, byte-pair encoding, is taught in Subword tokenization (BPE and WordPiece).

Pretraining: predicting the next token

Pretraining gathers a very large amount of text from web pages, books and code, and trains the model on one task: given the tokens so far, predict the next one. Each guess is compared with the real next token, and the model's numbers, called weights, are adjusted to make the real token more likely. Repeated over a huge amount of text, the model has to learn grammar, facts and patterns of reasoning in order to guess well. The result is a base model: it continues any text you give it, but it does not yet answer questions.

A real model does this with a neural network, the transformer, covered in Transformer architecture. The idea itself fits in a table of counts. The code below trains the smallest possible version on a few short sentences.

Counting what follows each word

python
from collections import Counter, defaultdict

text = "the cat sat on the mat . the dog sat on the rug . the cat saw the dog ."
words = text.split()

follows = defaultdict(Counter)       # follows["the"] counts the words that come after "the"
for current, nxt in zip(words, words[1:]):
    follows[current][nxt] += 1

Turning counts into probabilities

python
options = follows["the"]
total = sum(options.values())
probabilities = {word: round(count / total, 2) for word, count in options.items()}

Measuring how well the table predicts

Training lowers the average surprise of the real next word. The surprise of one guess is the negative log of the probability the model gave to the word that followed in the text, and the average over all the guesses is the loss. Guessing at random among the vocabulary gives a loss of log of the vocabulary size.

python
import math

pairs = list(zip(words, words[1:]))
loss = -sum(math.log(follows[a][b] / sum(follows[a].values())) for a, b in pairs) / len(pairs)
random_loss = math.log(len(set(words)))

Generating by sampling

python
import random

random.seed(7)
word, generated = "the", ["the"]
for _ in range(8):
    options = follows[word]
    word = random.choices(list(options), weights=list(options.values()))[0]
    generated.append(word)

Training and sampling a word-count model

All four pieces together, with the printouts that show each step. This table has no hidden layers; it is a stand-in that makes the training objective visible, not a neural network.

ExamplePure Python, Python 3.12
import math
import random
from collections import Counter, defaultdict

text = "the cat sat on the mat . the dog sat on the rug . the cat saw the dog ."
words = text.split()

follows = defaultdict(Counter)
for current, nxt in zip(words, words[1:]):
    follows[current][nxt] += 1

total = sum(follows["the"].values())
print({w: round(c / total, 2) for w, c in follows["the"].items()})

pairs = list(zip(words, words[1:]))
loss = -sum(math.log(follows[a][b] / sum(follows[a].values())) for a, b in pairs) / len(pairs)
print("loss", round(loss, 3), "vs", round(math.log(len(set(words))), 3), "for random guessing")

random.seed(7)
word, generated = "the", ["the"]
for _ in range(8):
    options = follows[word]
    word = random.choices(list(options), weights=list(options.values()))[0]
    generated.append(word)
print(" ".join(generated))

Reading the word-count model

  • The first line is what the table learned about the word after the: cat and dog each followed it in a third of the cases, mat and rug in a sixth.
  • The loss line compares the table with random guessing. Guessing among the nine distinct tokens of this text gives 2.197, and the trained table gives 0.566: the words that followed in the text were far more probable under the table. Pretraining a language model lowers this same number, on far more text and with far more weights.
  • The last line is text sampled from the table, one word at a time, with a fixed seed so it repeats. Every pair of neighbouring words in it occurred in the training text, and the sentence as a whole means nothing: the table has no idea what a cat is.

Instruction tuning: from continuing text to following instructions

A base model asked "What is the capital of France?" may continue with more quiz questions, because that is a likely continuation. Instruction tuning trains the base model further on many example prompts paired with the replies a good assistant would give, in a chat format with system, user and assistant roles. After it, the model treats a prompt as a request.

Preference tuning: learning what people prefer

Several replies can follow instructions and still differ in quality. In reinforcement learning from human feedback (RLHF), people rank candidate replies, a reward model learns to predict those rankings, and the language model is trained to produce replies that score higher. The InstructGPT paper, Training language models to follow instructions with human feedback, describes this recipe: instruction examples first, then a reward model from human rankings, then reinforcement learning. A later method, Direct Preference Optimization, learns from pairs of preferred and rejected replies without a separate reward model.

What the stages mean for the apps you build

  • Replies vary because the next token is sampled. The setting that controls how much is called temperature, taught in Temperature.
  • A model knows what its training text contained and nothing newer, which is why Retrieval-augmented generation (RAG) and tools exist.
  • The chat roles come from tuning. The system, human and AI messages in Messages and prompts are the format the model was tuned on.
  • Tuning continues after release. Fine-tuning LLMs covers adapting a model to your own examples, and Quantization covers making it smaller.

Pretraining vs instruction tuning vs preference tuning

PretrainingInstruction tuningPreference tuning
Learns fromA very large amount of raw text and codeExample prompts with good repliesRankings or pairs of replies
GoalPredict the next tokenFollow an instructionPrefer the replies people rate higher
ResultA base model that continues textA model that answers requestsA model whose answers people prefer
Relative costBy far the largestSmall beside pretrainingSmall beside pretraining

Where you use what the stages explain

  • Choosing a model: a name ending in instruct or chat is tuned to follow instructions, and a base model is for continuing text or for further training.
  • Deciding how to change behaviour: change the prompt first, add documents with RAG for facts, and fine-tune only when those are not enough.
  • Reading a model card: it lists the training data, the tuning steps and the cut-off of the model's knowledge.
Watch out. A model does not store its training documents the way a database does. It stores weights shaped by them, so it cannot point to where a fact came from, and it can produce a fact that was never in its data. Treat a bare answer as unverified.
Try it yourself
  • Change the text in the example to three different sentences and print follows["the"]. How does the loss change?
  • Change random.seed(7) to random.seed(8) and generate again. Why is the sentence different?
  • Start the generation from "cat" instead of "the".

This is what real progress feels like.