Chat templates: how messages become one text

You send a chat model messages with roles. It reads one text: a chat template writes the messages into it with special markers, and that text is tokenized.

messages = [
    {"role": "system", "content": "You sort support tickets."},
    {"role": "user", "content": "My parcel has not arrived"},
]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
print(text)

Each message becomes <|im_start|>, the role, the content and <|im_end|>. Those markers are single special tokens the model learned during training to read as message boundaries.

add_generation_prompt=True adds the opening of an assistant message at the end, with nothing after it. The model's next token is the start of its reply, and when it writes <|im_end|> the reply is over: the end token from lesson 7.

Leave the system message out and this template writes one in for you: You are Qwen, created by Alibaba Cloud. You are a helpful assistant. Models differ here, which is one reason the same messages can behave differently on two models.

A helper for the rest of the course

def reply(messages, max_new_tokens=40):
    text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
    inputs = tokenizer(text, return_tensors="pt")
    output = model.generate(**inputs, max_new_tokens=max_new_tokens, do_sample=False)
    new_tokens = output[0][inputs.input_ids.shape[1]:]
    return tokenizer.decode(new_tokens, skip_special_tokens=True)

reply is lessons 4 and 9 in one function: template, tokenize, greedy generate, keep only the new tokens, decode. Greedy, so every answer in part 3 can be compared fairly.

print(reply([{"role": "user", "content": "In one sentence, what is a parcel tracking number?"}]))

Why this matters with an API

Hosted APIs apply the template on their side, so you send the list and never see these markers. The model underneath still reads one long text. A system message has no special power beyond where the template puts it and what the model learned about that position.