Temperature: how random the choice is

Instead of always taking the top token, a model can pick at random, weighted by probability. Temperature reshapes those probabilities before the pick.

import torch

inputs = tokenizer("Sorry, your parcel", return_tensors="pt")
with torch.no_grad():
    logits = model(**inputs).logits[0, -1]

for temperature in (0.2, 1.0, 1.8):
    probs = torch.softmax(logits / temperature, dim=-1)
    top = torch.topk(probs, 3)
    print(temperature, [(tokenizer.decode(i), round(p.item(), 2)) for p, i in zip(top.values, top.indices)])

Temperature divides the logits before softmax. Below 1, the gaps between scores grow, and the top token takes almost all the probability. Above 1, the gaps shrink and unlikely tokens get a real chance. At 1 the probabilities are the model's own.

What that does to an answer

The next examples ask for an apology through the model's chat format, which lesson 9 explains. start is where the prompt ends, so only new tokens are printed:

messages = [{"role": "user", "content": "Write a one-sentence apology to a customer whose parcel is late."}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt")
start = inputs.input_ids.shape[1]

for temperature in (0.2, 1.0, 1.8):
    torch.manual_seed(0)
    output = model.generate(**inputs, max_new_tokens=30, do_sample=True, temperature=temperature)
    print(temperature, tokenizer.decode(output[0][start:], skip_special_tokens=True))

do_sample=True switches from greedy to picking at random, and torch.manual_seed(0) fixes the random choices so a run can be repeated (lesson 8).

Low temperature gives the safe, predictable sentence, and higher values drift further from it. At 1.8 it still reads well, because this model's default settings also cut off unlikely tokens; lesson 6 switches that off and shows what 1.8 really does. For sorting tickets or extracting data, keep temperature low or use greedy. For writing where variety helps, around 0.7 to 1 is common.