CustomLLM: writing a provider of your own
CustomLLM is a base class in LiteLLM that you subclass to add a model provider of your own, so any code you can write answers a completion() call like a built-in provider does.
Last updated: 12 Oct, 2026 · LiteLLM 1.105
Every provider so far, Groq and Gemini, came built in and needed a key. A custom provider needs neither. It is a Python class, so it can wrap an internal service, a rule-based bot or a test double. Because it needs no key and no network, this lesson's examples give the same output on every run.
The prefix of the model string picks the provider, as Providers and keys: the model string showed. A custom provider adds one more prefix to that list.
Syntax: subclass CustomLLM, write a completion method that fills in model_response, and register the class under a prefix.
class Faq(CustomLLM):
def completion(self, model, messages, model_response, **kwargs):
model_response.choices[0].message.content = "reply text"
return model_response
litellm.custom_provider_map = [{"provider": "faq", "custom_handler": Faq()}]
completion(model="faq/v1", messages=[{"role": "user", "content": "hi"}])Writing the provider class
Answering from a dictionary
The class below is a small FAQ bot. answer() looks for a keyword in the last message and returns the matching reply. LiteLLM calls completion() with keyword arguments. The method takes **kwargs for the ones it does not use. model_response arrives as an empty reply object. Filling in content and model is the whole job.
class Faq(CustomLLM):
ANSWERS = {"refund": "Refunds take 5 working days.", "ship": "Orders ship within 2 days."}
def answer(self, messages):
question = messages[-1]["content"].lower()
for word, reply in self.ANSWERS.items():
if word in question:
return reply
return "I do not know that one."
def completion(self, model, messages, model_response, **kwargs):
model_response.choices[0].message.content = self.answer(messages)
model_response.model = model
return model_responseRegistering the provider
custom_provider_map pairs a prefix with an instance of the class. After this line, any model string that starts with faq/ is sent to Faq. The part after the slash arrives as the model argument.
litellm.custom_provider_map = [
{"provider": "faq", "custom_handler": Faq()}, # "faq/anything" now goes to Faq
]Calling it before it is registered
Without the registration, the prefix faq means nothing to LiteLLM.
import litellm
from litellm import completion
litellm.suppress_debug_info = True # hides LiteLLM's help banner on errors
completion(model="faq/v1", messages=[{"role": "user", "content": "How long does a refund take?"}])Traceback (most recent call last):
File "main.py", line 6, in <module>
completion(model="faq/v1", messages=[{"role": "user", "content": "How long does a refund take?"}])
litellm.exceptions.BadRequestError: litellm.BadRequestError: LLM Provider NOT provided. Pass in the LLM provider you are trying to call. You passed model=faq/v1
Pass model as E.g. For 'Huggingface' inference endpoints pass in `completion(model='huggingface/starcoder',..)` Learn more: https://docs.litellm.ai/docs/providersLiteLLM raised BadRequestError with LLM Provider NOT provided. It does not know the word before the slash, so it cannot pick a provider. Registering the class fixes it.
Calling the custom provider
This is the whole program: the class, the registration and three questions. Each answer comes from the dictionary, so a changed question changes the answer.
import litellm
from litellm import CustomLLM, completion
class Faq(CustomLLM):
ANSWERS = {"refund": "Refunds take 5 working days.", "ship": "Orders ship within 2 days."}
def answer(self, messages):
question = messages[-1]["content"].lower()
for word, reply in self.ANSWERS.items():
if word in question:
return reply
return "I do not know that one."
def completion(self, model, messages, model_response, **kwargs):
model_response.choices[0].message.content = self.answer(messages)
model_response.model = model
return model_response
litellm.custom_provider_map = [
{"provider": "faq", "custom_handler": Faq()}, # "faq/anything" now goes to Faq
]
for question in ["How long does a refund take?", "When will it ship?", "Do you sell tents?"]:
response = completion(model="faq/v1", messages=[{"role": "user", "content": question}])
print(response.choices[0].message.content, "|", response.model, "|", response.choices[0].finish_reason)Refunds take 5 working days. | v1 | stop Orders ship within 2 days. | v1 | stop I do not know that one. | v1 | stop
- The first two questions match a keyword.
refundandshipeach return their own reply. - The third matches nothing. The class returns its fallback sentence, so the reply object is always filled in.
- response.model is v1. It is the part after the slash, set by the line
model_response.model = model. - finish_reason is stop. LiteLLM filled it in because the class did not set it.
Streaming from a custom provider
A call with stream=True goes to a separate method, streaming(). It is a generator that yields one small dictionary per chunk, with the text in text and is_finished set on the last one.
def streaming(self, model, messages, **kwargs):
words = self.answer(messages).split()
for i, word in enumerate(words):
last = i == len(words) - 1
yield {"text": word + " ", "is_finished": last, "finish_reason": "stop" if last else "",
"index": 0, "tool_use": None,
"usage": {"completion_tokens": 1, "prompt_tokens": 0, "total_tokens": 1}}for chunk in completion(model="faq/v1", stream=True,
messages=[{"role": "user", "content": "refund?"}]):
print(repr(chunk.choices[0].delta.content), chunk.choices[0].finish_reason)'Refunds ' None 'take ' None '5 ' None 'working ' None 'days. ' None None stop
Each word arrives as its own chunk, and the last chunk reports stop. A class that defines only completion() has no streaming(), and a streaming call to it fails. LiteLLM's base class raises Not implemented yet! for any method you do not write.
Built-in provider vs CustomLLM
| Built-in provider (groq/, gemini/) | CustomLLM (your prefix) | |
|---|---|---|
| Who writes the call | LiteLLM | You |
| Needs an API key | Yes | Only if your code needs one |
| Streaming | Included | Write streaming() |
| Price and token counts | Filled in from the model table and the reply | Not filled in; usage reads zero unless you set it |
Where you use a custom provider
- Wrapping an internal API. Your own model service answers like any other provider, behind the same call.
- Testing without a network. A provider that returns fixed replies, or fails on demand, makes tests that run the same way each time. The Router lessons use it for that, starting with Router: model groups and deployments.
- A rule-based first answer. A bot that handles the known questions and passes the rest to a model.
response._hidden_params.get("response_cost") is None for it, so any cost tracking built on that field needs to treat a custom provider as unpriced instead of free.Related
- Previous: Multimodal: pictures and sound in the same call
- Next: Router: model groups and deployments
- See also: Tokens and cost for where cost comes from on a built-in provider
- Reference: Custom API server (custom format) in the LiteLLM docs
- Add a
"price"keyword toANSWERSand askWhat is the price?. - Register the same class under a second prefix,
help, and callhelp/v1. - Delete the
**kwargsfromcompletion()and read the error LiteLLM raises when it passes keyword arguments the method does not take.
Little by little, you're building something great.