Chat generators: a stand-in and the real thing

A chat generator takes messages and returns replies. OpenAIChatGenerator calls a provider; the course's ShopChat answers from the prompt by word overlap.

Example

generator = OpenAIChatGenerator(model="gpt-4.1-mini")
print(type(generator).__name__, "created")
generator.run(messages=[ChatMessage.from_user("Hello")])

Creating it works without a key. The key is resolved when it runs: its api_key defaults to Secret.from_env_var("OPENAI_API_KEY"), a Haystack Secret that reads the variable at the last moment and is never written into a saved pipeline. With the key exported, run returns {"replies": [ChatMessage]}.

The stand-in

Exampleshop_chat.py

"""A stand-in chat generator for Haystack: word overlap instead of a neural network."""
import re

from haystack import component
from haystack.dataclasses import ChatMessage, ToolCall


def words(text):
    return {w for w in re.findall(r"[a-z]+", text.lower()) if len(w) > 3}


@component
class ShopChat:
    """Answers from the documents in its prompt, and calls a tool when a ticket names an order."""

    @component.output_types(replies=list[ChatMessage])
    def run(self, messages: list[ChatMessage], tools: list | None = None):
        last = messages[-1]

        # A tool ran: answer with its result.
        if last.tool_call_result:
            return {"replies": [ChatMessage.from_assistant(f"Order status: {last.tool_call_result.result}.")]}

        text = last.text or ""
        # An order id, and a tool to look it up with.
        order = re.search(r"A-\d{4}", text)
        if order and tools:
            call = ToolCall(tool_name=tools[0].name, arguments={"order_id": order.group()})
            return {"replies": [ChatMessage.from_assistant(tool_calls=[call])]}

        # A prompt with documents as "- " lines: pick the line sharing most words with the question.
        question = re.search(r"Question: (.*)", text)
        lines = re.findall(r"^- (.*)$", text, re.M)
        if question and lines:
            asked = words(question.group(1))
            best = max(lines, key=lambda line: len(asked & words(line)))
            if asked & words(best):
                return {"replies": [ChatMessage.from_assistant(best)]}
            return {"replies": [ChatMessage.from_assistant("I could not find that in the documents.")]}

        return {"replies": [ChatMessage.from_assistant("I can only answer from documents.")]}

It has the same shape as a real chat generator: run(messages, tools) returning replies, so it can replace OpenAIChatGenerator in any pipeline or agent.
Given a prompt with - document lines and a Question:, it returns the line sharing the most words with the question, or says it could not find it.
Given tools and an order id, it returns a tool call; after the tool runs, it answers with the result (lesson 13).

Example

prompt = [ChatMessage.from_user("- Refunds are paid within five working days.\n- Parcels ship within two days.\nQuestion: when are refunds paid?")]
print(ShopChat().run(messages=prompt)["replies"][0].text)

What the stand-in is for

It copies a sentence; a model writes an answer, and can combine documents or refuse politely. What you learn is the pipeline around it, which stays the same when OpenAIChatGenerator or another generator replaces it.

Try it yourself

Ask a question that shares no words with either line.
Pass api_key=Secret.from_token("sk-test") to OpenAIChatGenerator and print generator.to_dict(). Is the token there?
Look up OllamaChatGenerator in the ollama-haystack integration.

PreviousPrompt templates with ChatPromptBuilder

Little by little, you're building something great.