What should you log and trace for LLM requests? What is OpenTelemetry's role?

30-second answerSay your answer out loud first, then reveal.

Trace each request end to end as a tree of spans: the API request, retrieval, each LLM call, tool calls, guardrail checks, and sub-agents. Record model, versions, parameters, token usage, latency, cost, errors, and (subject to privacy policy) prompts and outputs. OpenTelemetry provides a vendor-neutral standard for traces and metrics, and its GenAI semantic conventions define common attribute names (e.g. model name, input and output token counts), so data flows to any backend: Langfuse, LangSmith, Phoenix, Datadog, Grafana and others.

Per LLM span (example attributes)

text

gen_ai.system / provider, gen_ai.request.model, gen_ai.response.model
gen_ai.request.temperature, gen_ai.request.max_tokens
gen_ai.usage.input_tokens, gen_ai.usage.output_tokens (+ cached tokens if available)
latency_ms, ttft_ms, finish_reason, error_type
app.prompt_version, app.feature, app.tenant_id (non-PII), trace_id
payload: prompt / completion (redacted, sampled per policy)

Practices

Correlate: propagate trace IDs from frontend to backend to providers' request IDs.
Redact PII before export; restrict access to payloads.
Sample payloads (keep all errors and flagged requests) to control storage cost.
Link feedback (thumbs, edits) to the trace ID.
Metrics from traces: derive latency, cost and error dashboards.

Why OpenTelemetry: avoids lock-in to one observability vendor and integrates LLM traces with existing APM for the rest of the stack.

PreviousHow do you define SLOs for an LLM-powered service?

Slow is fine. Stopping is the only problem.

What should you log and trace for LLM requests? What is OpenTelemetry's role?

Per LLM span (example attributes)

Practices

Related