Tokens and cost: what each call costs

Every LiteLLM response carries its cost, from tokens used and a price table. A model with no price costs nothing as far as LiteLLM knows.

Example

litellm.register_model({
    "shop/small": {"input_cost_per_token": 0.0000005, "output_cost_per_token": 0.0000015, "litellm_provider": "shop", "mode": "chat"},
    "shop/large": {"input_cost_per_token": 0.000005, "output_cost_per_token": 0.000015, "litellm_provider": "shop", "mode": "chat"},
})

register_model adds prices to LiteLLM's table, per token in and per token out. The numbers are example prices, a small cheap model and a large one charging ten times as much, in the usual pattern where writing costs more than reading.

Example

import litellm
from litellm import completion

import shop_llm

litellm.register_model({
    "shop/small": {"input_cost_per_token": 0.0000005, "output_cost_per_token": 0.0000015, "litellm_provider": "shop", "mode": "chat"},
    "shop/large": {"input_cost_per_token": 0.000005, "output_cost_per_token": 0.000015, "litellm_provider": "shop", "mode": "chat"},
})

for model in ["shop/small", "shop/large"]:
    response = completion(model=model, messages=[{"role": "user", "content": "I was charged twice for one order"}])
    cost = response._hidden_params["response_cost"]
    print(f"{model:10} {response.usage.prompt_tokens} in, {response.usage.completion_tokens} out, ${cost:.7f}")

response._hidden_params["response_cost"] is where the SDK docs put the cost of the call. The same tokens cost ten times more on the large model: at 50,000 tickets a month that is the difference this whole course is about.

Counting before sending

Example

messages = [{"role": "user", "content": "I was charged twice for one order"}]
print(litellm.token_counter(model="shop/small", messages=messages))
print(f"{litellm.completion_cost(model='shop/large', prompt='I was charged twice for one order', completion='billing'):.7f}")

token_counter estimates tokens before a call; for a model without its own tokenizer it falls back to OpenAI's, per the token usage page, so its count differs from ShopLLM's word count. completion_cost prices a prompt and answer you already have. LLM Fundamentals covered why counts differ between tokenizers.

Try it yourself

Register shop/large at twenty times the small price and rerun the loop.
Remove register_model and print the cost.
Work out the monthly cost of 50,000 tickets on each model.

PreviousStreaming: the answer a piece at a time

Little by little, you're building something great.