How do you track and attribute LLM costs in an organisation?

30-second answerSay your answer out loud first, then reveal.

Capture token usage (input, output, cached) and model for every call, compute cost from a maintained price table, and tag each call with feature, team, tenant, user tier and environment. Aggregate it in dashboards with budgets and anomaly alerts. For self-hosted models, allocate GPU costs by usage share (tokens or GPU-seconds). Report unit economics, such as cost per conversation or per resolved ticket, not just total spend.

Implementation

Gateway or SDK instrumentation: every call logs {model, input_tokens, output_tokens, cached_tokens, feature, team, tenant, env}.
Price table: versioned (prices change), including discounts (batch, cached input, committed use).
Aggregation: daily cost per feature, team and tenant; per-request distributions (to find outliers).
Budgets and alerts: per team and feature; anomaly detection for sudden spikes (bugs, abuse, retry loops).
Unit economics: cost per successful outcome; gross margin per customer tier for SaaS pricing.
Showback / chargeback: monthly reports to team owners.

Self-hosted allocation: GPU cost per hour × hours ÷ tokens served gives cost per 1M tokens per model; idle capacity is shown as overhead, which motivates utilisation improvements.

PreviousWhy are feature flags and config-driven behaviour important in LLM applications?

Slow is fine. Stopping is the only problem.

How do you track and attribute LLM costs in an organisation?

Implementation

Related