Context windows: how much a model can read

A model reads a fixed number of tokens at once, its context window. The system prompt, examples, conversation and the answer all have to fit inside it.

print(model.config.max_position_embeddings)

This model's configuration allows 32,768 token positions. Hosted models range from tens of thousands to over a million. The larger the window, the more you can send, and the more you pay for each call that uses it.

A conversation that grows

The conversation starts with structured, the system prompt from lesson 10.

conversation = [{"role": "system", "content": structured}]
for turn in range(40):
    conversation.append({"role": "user", "content": f"Ticket {turn}: my parcel has not arrived yet"})
    conversation.append({"role": "assistant", "content": '{"category": "shipping", "priority": 3}'})

def count_tokens(messages):
    encoded = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_dict=True)
    return len(encoded["input_ids"])

A support chat with 40 earlier turns. count_tokens applies the chat template and counts the tokens. tokenize=True with return_dict=True gives a dictionary whose input_ids are the token ids; without return_dict, calling len on the result would count the dictionary's two keys instead.

print(count_tokens(conversation[:1]), "tokens: the system prompt")
print(count_tokens(conversation), "tokens: the whole conversation")

Every call re-sends the whole conversation, because the model remembers nothing between calls. A chat that runs long grows the cost of every new message, and eventually stops fitting.

Keeping the newest turns

def trim(messages, budget):
    system, rest = messages[:1], messages[1:]
    while rest and count_tokens(system + rest) > budget:
        rest = rest[2:]
    return system + rest

short = trim(conversation, budget=500)
print(len(conversation), "messages before,", len(short), "after")
print(count_tokens(short), "tokens")
print(short[1]["content"])

trim keeps the system message and drops the oldest user and assistant pair, two messages at a time, until the rest fits the budget. The oldest turn left is now a recent one. Leave room in the budget for the answer, since it shares the window.

Dropping old turns loses what was in them. Other approaches summarise old turns, or store facts about a user separately and add only the relevant ones, which is what memory frameworks like Mem0 do.