Seeds: why the same prompt gives different answers

Ask a sampling model the same thing three times and you get three answers. That is expected behaviour, and knowing where it comes from tells you when to switch it off.

messages = [{"role": "user", "content": "Write a one-sentence apology to a customer whose parcel is late."}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt")
start = inputs.input_ids.shape[1]

output = model.generate(**inputs, max_new_tokens=30, do_sample=True, temperature=1.0, num_return_sequences=3)
for row in output:
    print("-", tokenizer.decode(row[start:], skip_special_tokens=True))

num_return_sequences=3 samples three answers from one prompt. Same prompt, same model, same settings, different words, because each token was drawn at random. Your three sentences will not match the ones above; the next example is how to make them repeatable.

Fixing the random choices

for run in range(2):
    torch.manual_seed(42)
    output = model.generate(**inputs, max_new_tokens=20, do_sample=True, temperature=1.0)
    print(run, tokenizer.decode(output[0][start:], skip_special_tokens=True))

With the same seed before each run, the random numbers repeat, and so does the answer. That is useful when you are debugging a prompt and want to change one thing at a time.

A seed is not a guarantee across machines. A different processor, library version or batch of requests can change the result even with the same seed, and most hosted APIs treat a seed as best effort. When you need the same answer every time, use temperature 0 or greedy, and store the answer rather than regenerating it.