How do you evaluate multi-turn conversations?

30-second answerSay your answer out loud first, then reveal.

Single-turn evals miss context handling, consistency, clarification behaviour and task completion across a dialogue. Evaluate at conversation level (was the user's goal achieved? was the policy followed throughout?) and at turn level (correct handling of a follow-up). Use replayed real conversations (fixed history, evaluate the next turn), simulated users (an LLM playing a persona with a goal against your bot), and scripted scenario tests, with judges or state checks at the end.

Approaches

Approach	How	Good for
Next-turn eval on real histories	Feed the first N turns from logs, evaluate the bot's next reply	Context handling, regression on real data
User simulator	LLM plays a user with persona, goal, hidden info, patience level	End-to-end task success, many variations
Scripted scenarios	Fixed user messages with assertions per turn	Deterministic policy tests (e.g. must ask for verification)
Conversation-level judge	Judge reads the full transcript against a rubric	Overall helpfulness, tone, compliance

Conversation-level criteria examples

Goal achieved (verify via backend state where possible: was the ticket created correctly?).
Asked necessary clarifying questions; didn't ask redundant ones.
Consistent facts across turns; no contradictions.
Followed escalation rules; respected refusals or boundaries.
Number of turns to resolution (efficiency).

User simulator tips: give the simulator a hidden goal and constraints ("you want a refund but the order is 40 days old"); vary personas; cap turns; and validate that simulator behaviour is realistic by comparing with real conversations.

PreviousHow do you generate useful synthetic test data for evals, and what are the pitfalls?

Little by little, you're building something great.

How do you evaluate multi-turn conversations?

Approaches

Conversation-level criteria examples

Related