Generating text: one token at a time

Lesson 3 predicted one token. Pick it, add it to the input, and predict again: that loop is how every answer from every language model is written.

import torch

ids = tokenizer.encode("Your refund", return_tensors="pt")

for _ in range(8):
    with torch.no_grad():
        logits = model(ids).logits
    next_id = logits[0, -1].argmax()
    ids = torch.cat([ids, next_id.view(1, 1)], dim=1)

print(tokenizer.decode(ids[0]))

argmax takes the most likely token, the first in the lesson 3 list. torch.cat joins it onto the end of the ids, so the next pass reads one more token. Eight passes add eight tokens.

Picking the most likely token every time is called greedy decoding. It gives the same text every run.

The same thing, built in

inputs = tokenizer("Your refund", return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=8, do_sample=False)
print(tokenizer.decode(output[0]))

generate runs the loop for you. do_sample=False means greedy, and max_new_tokens is how many passes to make at most. The text matches the hand-written loop, because it is the same loop.

Why answers arrive word by word: each token needs a full pass through the model, so a model has nothing more to send until it has run again. The streaming you saw in APIs for AI is these tokens sent as they are made.

Likely is not the same as true

inputs = tokenizer("Your refund", return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=60, do_sample=False)
print(tokenizer.decode(output[0]))

Nobody told the model this shop's refund policy. It wrote one anyway, 30 days and a full refund, because that is what usually follows the words Your refund policy. A model produces likely text, not checked facts, and this is exactly how a confident wrong answer is made. Giving the model the real policy in the prompt, and checking its answers, are the defences later lessons build.