Controlling the cost of API calls
Controlling API cost is keeping the bill from an LLM app down by spending fewer tokens and making fewer calls: a cache for repeats, a cap on output length, a smaller model where it suffices, and a rate limit on the whole app.
Last updated: 15 Oct, 2026 · LLMOps
The Serving an API and scaling it lesson grew the endpoint to serve more traffic, and more traffic is more cost. The Detailed LLMOps Project Lifecycle video singles this out as one important thing, how do we optimize the API call cost, because LLM model API calls will be happening on every request. A paid model charges per token in and per token out, and a self-hosted one bills by the GPU hour, so both reward the same habits. This lesson is the set of levers that lower the bill.
The levers, and what each one saves
Cost on a paid model is tokens multiplied by calls, so every lever either cuts the tokens a call spends or cuts how many calls happen. They stack: a cache removes calls, a smaller model lowers the price of the ones that remain, and a token cap bounds the worst case of each.
| Lever | What it does | Effect on cost |
|---|---|---|
| Cache repeats | Return a stored answer for an identical prompt | Removes the call and its tokens entirely |
| Cap max_tokens | Limit the length of the model's output | Bounds the priciest half of each call |
| Trim the prompt | Send less context and fewer examples | Fewer input tokens on every call |
| A smaller model | Route easy requests to a cheaper model | Lower price per token for most traffic |
| Rate limit | Cap calls per user or per minute | Stops a loop or abuse running up a bill |
Prompt size is a cost you pay every call
The input tokens are easy to forget because they are not the answer, but a long system prompt and a pile of retrieved context are sent on every single request and charged every time. Retrieving four tight chunks instead of twenty, and keeping the system prompt short, lowers the floor cost of every call the app ever makes.
Output is the dearer half
On most paid models an output token costs more than an input token, and output is the part that runs unbounded if you let it. A max_tokens cap is the single cheapest guard: it puts a ceiling on the most expensive part of each call and stops a model that decides to write an essay from billing like one.
Where cost control matters
- A high-traffic app, where a cache on common questions removes a large share of calls before they cost anything.
- A retrieval app, where trimming how much context is sent cuts input tokens on every request at once.
- A mixed workload, where routing simple requests to a smaller model keeps the dear model for the hard ones.
Related
- Previous: Serving an API and scaling it
- Next: Deploying an open-source LLM on SageMaker
- Measuring whether a saving hurt quality: Promptfoo overview
- Add a dict cache keyed on the prompt to an app you run, and count the calls it removes over an hour of repeats.
- Set a
max_tokenscap and watch the cost of a long answer stop growing at the ceiling. - Measure the input tokens of your system prompt plus retrieved context, and work out what that floor costs per thousand calls.
You understood something today that you didn't yesterday.