top_p and top_k: cutting off unlikely tokens

Temperature makes rare tokens likelier, absurd ones included. top_k and top_p cut the unlikely tail before the pick, so answers vary without derailing.

Example

import torch

inputs = tokenizer("Sorry, your parcel", return_tensors="pt")
with torch.no_grad():
    probs = torch.softmax(model(**inputs).logits[0, -1], dim=-1)

sorted_probs, sorted_ids = torch.sort(probs, descending=True)
cumulative = torch.cumsum(sorted_probs, dim=-1)
kept = int((cumulative < 0.9).sum()) + 1
print(kept, "tokens make up 90% of the probability")
print([tokenizer.decode(i) for i in sorted_ids[:10]])

Sort the tokens from most to least likely and add up probabilities as you go. top_p=0.9 keeps just enough tokens to reach 90% and throws the rest away, then picks among those. top_k=50 keeps the 50 most likely, however much probability that is.

Out of about 150,000 tokens, only this many carry nine tenths of the probability. Everything past them is where nonsense comes from.

High temperature, with and without a cut

Examplethe apology request from lesson 5

messages = [{"role": "user", "content": "Write a one-sentence apology to a customer whose parcel is late."}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt")
start = inputs.input_ids.shape[1]

Example

for top_p in (1.0, 0.5):
    torch.manual_seed(1)
    output = model.generate(**inputs, max_new_tokens=30, do_sample=True, temperature=1.8, top_p=top_p, top_k=0)
    print(top_p, tokenizer.decode(output[0][start:], skip_special_tokens=True))

top_k=0 turns top_k off so only top_p acts. With top_p=1.0 nothing is cut, and temperature 1.8 can pick from the whole tail. With 0.5 the same temperature only chooses among the tokens holding the top half of the probability.

What a model uses unless told otherwise

A model on Hugging Face ships its own defaults in model.generation_config. Print it: this one sets its own temperature, top_p and top_k, which is why the chat examples need do_sample=False to be greedy. Hosted APIs have defaults too; check them before comparing results.

Try it yourself

Print model.generation_config.
Count how many tokens make up 90% after "The", a start with far more possibilities.
Set top_k=5 and top_p=1.0 at temperature 1.8.

PreviousTemperature: how random the choice is

Little by little, you're building something great.