How do you define success metrics with a customer for an AI project?

30-second answerSay your answer out loud first, then reveal.

Agree on three layers of metrics before building. Business outcome metrics (handle time, cost per case, revenue, CSAT). Model/task quality metrics measured on an agreed eval set (accuracy, extraction F1, groundedness). Adoption and operational metrics (active users, acceptance rate, latency, cost). Set baselines, targets and how and when each is measured, and write them down. Ambiguous success criteria are the top reason AI projects get declared failures.

Example: claims summarisation assistant

Layer	Metric	Baseline	Target	How measured
Business	Avg. time to review a claim file	25 min	15 min	Time tracking, 4-week pilot
Quality	Summary correctness (SME rubric)	—	≥ 90% "acceptable"	150-case eval set
Quality	Critical omissions	—	< 2%	Same eval set
Adoption	Weekly active adjusters in pilot	—	≥ 70%	Usage logs
Ops	p95 latency	—	< 20s	Monitoring
Ops	Cost per claim	—	< ₹15	Token logs

Principles

Baselines first: measure the current process (even roughly) before claiming improvement.
Agree on the eval set and grading method with SMEs. "90% accurate" means nothing until you know what counts as correct.
Include guardrail metrics that must not get worse (error rate, compliance issues, customer complaints).
Time-bound: "by the end of the 6-week pilot".
Write it down in the project charter and review it at checkpoints.

PreviousHow do you choose the first AI use case to build with a customer?

Every expert started right here.

How do you define success metrics with a customer for an AI project?

Example: claims summarisation assistant

Principles

Related