Design an observability and tracing platform for LLM applications handling billions of spans per day.

30-second answerSay your answer out loud first, then reveal.

Instrument apps with OpenTelemetry (GenAI semantic conventions) via SDKs that capture spans for LLM calls, retrieval, tools and agent steps, with tokens, latency, cost, model and prompt versions. Send them through a collector pipeline with sampling, PII redaction and batching into a streaming bus, then into separate stores: a columnar store for metrics and span metadata (fast aggregation), object storage for large payloads (prompts and outputs), and a search index for lookup. Provide trace views, dashboards, alerting, online evaluators on sampled traces, and dataset curation from traces.

Instrumented apps send spans to collectors and a streaming bus, which fans out to a search index, online evaluators, a columnar store and object storage; the stores feed dashboards, trace views and alerts, from which datasets are curated into eval sets.

Design considerations

Data volume: LLM payloads are large (prompts can be 10K+ tokens). Store metadata for 100% of traces, full payloads for a sample (plus all errors and flagged traces), and compress them.
Sampling strategy: head sampling (decide at the start) vs tail sampling (decide after seeing the outcome: keep errors, slow traces, low feedback scores).
Privacy: redaction before storage, tenant isolation, retention policies, role-based access to payloads.
Cost attribution: compute cost from tokens × model price per span; aggregate by feature, team and user.
Query patterns: "p95 latency by model last hour" (columnar aggregate), "show this user's failing traces" (index lookup), "find traces where the tool X errored" (filters).
Online evaluation: asynchronous judges score sampled traces for faithfulness and relevance; results become metrics and alerts.
Integrations: export to existing APM tools; link traces to deploys and versions for regression analysis.

PreviousDesign the safety and abuse-prevention architecture for a public, free-to-use generative AI app.

This is what real progress feels like.

Design an observability and tracing platform for LLM applications handling billions of spans per day.

Design considerations

Related