Callbacks: tracking every call

A gateway's owner needs to know who spent what. A callback is a class LiteLLM calls after every request, with the request, response and cost.

Example

class SpendTracker(CustomLogger):
    def __init__(self):
        super().__init__()
        self.by_team = {}

    async def async_log_success_event(self, kwargs, response_obj, start_time, end_time):
        team = kwargs["litellm_params"]["metadata"]["team"]
        self.by_team[team] = self.by_team.get(team, 0) + kwargs["response_cost"]

A CustomLogger subclass with async_log_success_event, which the custom callbacks page lists as the hook for successful async calls; async_log_failure_event is its partner. kwargs holds the request, including the metadata you attach to it and the call's response_cost.

Example

import asyncio

import litellm
from litellm.integrations.custom_logger import CustomLogger

import shop_llm

litellm.register_model({
    "shop/small": {"input_cost_per_token": 0.0000005, "output_cost_per_token": 0.0000015, "litellm_provider": "shop", "mode": "chat"},
    "shop/large": {"input_cost_per_token": 0.000005, "output_cost_per_token": 0.000015, "litellm_provider": "shop", "mode": "chat"},
})

class SpendTracker(CustomLogger):
    def __init__(self):
        super().__init__()
        self.by_team = {}

    async def async_log_success_event(self, kwargs, response_obj, start_time, end_time):
        team = kwargs["litellm_params"]["metadata"]["team"]
        self.by_team[team] = self.by_team.get(team, 0) + kwargs["response_cost"]


async def main():
    tracker = SpendTracker()
    litellm.callbacks = [tracker]
    for team, model in [("billing", "shop/large"), ("shipping", "shop/small"), ("billing", "shop/large")]:
        await litellm.acompletion(model=model, messages=[{"role": "user", "content": "I was charged twice for one order"}], metadata={"team": team})
    await asyncio.sleep(0.5)
    for team, spent in tracker.by_team.items():
        print(f"{team:9} ${spent:.6f}")


asyncio.run(main())

litellm.callbacks registers the tracker for every call. metadata={"team": ...} travels with the request to the callback without being sent to the model.

The sleep is there on purpose. LiteLLM runs logging in the background so it never slows a reply, which means a callback can finish after acompletion has returned. Reading the totals straight away can miss the last call; a real service reads them later, or from its logging system.

Callbacks are also how LiteLLM connects to observability tools. Its docs list integrations such as Langfuse and Datadog, configured by name instead of a class.

Try it yourself

Add async_log_failure_event and count failures per team using flaky/small.
Remove the sleep and run it a few times.
Record the model as well as the team, and print spend per model.

PreviousErrors: one set of exceptions for every provider

Slow is fine. Stopping is the only problem.