Latency: how long an answer takes

A person waiting on a reply notices seconds. Most of the time goes into writing output tokens one by one, so answer length matters more than prompt length.

messages = [{"role": "user", "content": "Write a one-sentence apology to a customer whose parcel is late."}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt")
start = inputs.input_ids.shape[1]

import time

for limit in (10, 60):
    started = time.perf_counter()
    output = model.generate(**inputs, max_new_tokens=limit, min_new_tokens=limit, do_sample=False)
    seconds = time.perf_counter() - started
    made = output.shape[1] - start
    print(f"{made} tokens in {seconds:.2f} seconds, {made / seconds:.0f} tokens a second")

min_new_tokens stops the model from ending early, so both runs write exactly the number asked for. Six times the tokens takes several times as long: each one is a pass through the model, as in lesson 4. Your numbers will differ with your computer; the ratio is what to look at.

Two numbers to track

Time to first token is how long before anything appears. It grows with the prompt, since the whole input is read before the first output token. Tokens per second is how fast the rest follows. Streaming, from APIs for AI, does not make an answer faster; it makes the wait visible by showing tokens as they are made.

Ways to make an answer faster: ask for shorter answers, since JSON with two fields beats a paragraph; use a smaller model where it scores well enough; send fewer examples; and run independent calls at the same time, as the async lessons did.