The AI assistant works for some users but gives errors or empty answers for others. How do you debug it?

30-second answerSay your answer out loud first, then reveal.

Find what distinguishes failing users from working ones: permissions and roles (different data access means empty retrieval), region or tenant configuration, language or locale, data differences (different product lines, document types), client versions or browsers, and usage patterns. Compare traces from both groups side by side, reproduce with a failing user's identity in a safe way, and check logs for auth errors, filtered results and exceptions.

Hypotheses to test

Difference	Symptom	Check
Permissions / ACLs	Empty or "I don't know" answers	Retrieval returns zero docs after permission filter; group sync issues
Group membership sync lag	New employees fail	Identity sync timestamps
Region / tenant config	Errors in one office	Endpoint configs, data residency routing
Language	Hindi queries fail	Embedding model and prompt language handling
Data	Users in division B have docs in a new format	Parsing failures for that source
Client	Old app version / browser blocking streaming	Client logs, user-agent
Rate limits per user	Heavy users fail intermittently	429 logs

Process

Collect examples: user IDs, timestamps, exact queries.
Pull traces, and compare retrieval counts, filters, errors and config.
Reproduce with a test account that mirrors a failing user's permissions.
Fix the root cause, add monitoring for "empty retrieval rate by user group", and add a regression test.

PreviousDuring a pilot, users complain the assistant is "too slow." You're on a call with the customer. How do you diagnose it quickly?

Little by little, you're building something great.

The AI assistant works for some users but gives errors or empty answers for others. How do you debug it?

Hypotheses to test

Process

Related