Chat prompts, placeholders and config

A chat prompt is a list of messages. A placeholder slots in earlier turns of a conversation, and config stores settings such as the model name beside the text.

Lesson 18's prompt was one string. Chat models read a list of messages, and a support chat has earlier turns that must go in the middle, between the instruction and the new question. A chat prompt stores that list.

langfuse.create_prompt(
    name="desk-chat",
    type="chat",
    prompt=[
        {"role": "system", "content": "You answer customers of {{shop}} in one short sentence."},
        {"type": "placeholder", "name": "history"},
        {"role": "user", "content": "{{ticket}}"},
    ],
    config={"model": "shop-model", "temperature": 0},
    labels=["production"],
)

The placeholder named history marks where a list of messages will go. config is any JSON you want versioned with the prompt; here the model and temperature, so changing model becomes a prompt change rather than a code change. The type is fixed when a prompt is created and cannot change later.

prompt = langfuse.get_prompt("desk-chat", type="chat")
history = [{"role": "user", "content": "Where is A17?"},
           {"role": "assistant", "content": "Order A17 shipped on 3 March."}]
messages = prompt.compile(shop="a small online shop", ticket="Thanks. And B22?", history=history)
for message in messages:
    print(f"{message['role']:9} {message['content']}")
print(prompt.config)

python chat_prompt.py

compile filled the variables and replaced the placeholder with the two history messages, giving four messages ready for a chat model. It does not check that the history messages have the right shape; that is up to you.

Sending them to the model

response = client.chat.completions.create(
    model=prompt.config["model"], temperature=prompt.config["temperature"], messages=messages
)
print(response.choices[0].message.content)

python chat_prompt.py

The client from lesson 7 sent the compiled messages with the model and temperature read from the config. The stand-in reads only the last message, which has no Lookup: line, so it asked for an order number; a real model would read the history and see that B22 was the question.