max_new_tokens and stopping: how an answer ends

An answer ends in one of two ways: the model produces its end-of-message token, or it hits the length limit you set. Only the first means the answer is finished.

messages = [{"role": "user", "content": "Write a one-sentence apology to a customer whose parcel is late."}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt")
start = inputs.input_ids.shape[1]

for limit in (8, 60):
    output = model.generate(**inputs, max_new_tokens=limit, do_sample=False)
    new = output[0][start:]
    print(len(new), "tokens:", tokenizer.decode(new, skip_special_tokens=True))

With a limit of 8 the sentence is cut off mid-way. With 60 it finished on its own with fewer tokens than allowed. The code decides which happened by checking the last token.

The end token

output = model.generate(**inputs, max_new_tokens=60, do_sample=False)
last = output[0][-1].item()
print(last, repr(tokenizer.decode([last])))
print(last in model.generation_config.eos_token_id)

The model ended by writing a special token that means end of message. eos_token_id, end of sequence, lists the ids that stop generation. skip_special_tokens=True in earlier lessons is what hid it.

Check how an answer ended before using it. Hosted APIs report it as a field, often called finish_reason or stop_reason, with a value like stop or length. A JSON answer cut off by the limit is not valid JSON, and the reason field says why before your parser does.