LiteLLMLiteLLM 1.105 · Python 3.10+
Dashboard
0%

CustomLLM: writing a provider of your own

CustomLLM is a base class in LiteLLM that you subclass to add a model provider of your own, so any code you can write answers a completion() call like a built-in provider does.

Last updated: 12 Oct, 2026 · LiteLLM 1.105

Every provider so far, Groq and Gemini, came built in and needed a key. A custom provider needs neither. It is a Python class, so it can wrap an internal service, a rule-based bot or a test double. Because it needs no key and no network, this lesson's examples give the same output on every run.

How completion decides where a call goes. LiteLLM reads the prefix of the model string. A prefix such as groq or gemini goes to the built-in provider code and then the provider's API. A prefix registered in custom_provider_map, such as faq, goes to your own class. Both paths return the same ModelResponse object.

The prefix of the model string picks the provider, as Providers and keys: the model string showed. A custom provider adds one more prefix to that list.

Syntax: subclass CustomLLM, write a completion method that fills in model_response, and register the class under a prefix.

python
class Faq(CustomLLM):
    def completion(self, model, messages, model_response, **kwargs):
        model_response.choices[0].message.content = "reply text"
        return model_response

litellm.custom_provider_map = [{"provider": "faq", "custom_handler": Faq()}]
completion(model="faq/v1", messages=[{"role": "user", "content": "hi"}])

Writing the provider class

Answering from a dictionary

The class below is a small FAQ bot. answer() looks for a keyword in the last message and returns the matching reply. LiteLLM calls completion() with keyword arguments. The method takes **kwargs for the ones it does not use. model_response arrives as an empty reply object. Filling in content and model is the whole job.

python
class Faq(CustomLLM):
    ANSWERS = {"refund": "Refunds take 5 working days.", "ship": "Orders ship within 2 days."}

    def answer(self, messages):
        question = messages[-1]["content"].lower()
        for word, reply in self.ANSWERS.items():
            if word in question:
                return reply
        return "I do not know that one."

    def completion(self, model, messages, model_response, **kwargs):
        model_response.choices[0].message.content = self.answer(messages)
        model_response.model = model
        return model_response

Registering the provider

custom_provider_map pairs a prefix with an instance of the class. After this line, any model string that starts with faq/ is sent to Faq. The part after the slash arrives as the model argument.

python
litellm.custom_provider_map = [
    {"provider": "faq", "custom_handler": Faq()},   # "faq/anything" now goes to Faq
]

Calling it before it is registered

Without the registration, the prefix faq means nothing to LiteLLM.

ExampleAn unregistered prefix
import litellm
from litellm import completion

litellm.suppress_debug_info = True     # hides LiteLLM's help banner on errors

completion(model="faq/v1", messages=[{"role": "user", "content": "How long does a refund take?"}])

LiteLLM raised BadRequestError with LLM Provider NOT provided. It does not know the word before the slash, so it cannot pick a provider. Registering the class fixes it.

Calling the custom provider

This is the whole program: the class, the registration and three questions. Each answer comes from the dictionary, so a changed question changes the answer.

ExampleThree questions, three answers, no key
import litellm
from litellm import CustomLLM, completion

class Faq(CustomLLM):
    ANSWERS = {"refund": "Refunds take 5 working days.", "ship": "Orders ship within 2 days."}

    def answer(self, messages):
        question = messages[-1]["content"].lower()
        for word, reply in self.ANSWERS.items():
            if word in question:
                return reply
        return "I do not know that one."

    def completion(self, model, messages, model_response, **kwargs):
        model_response.choices[0].message.content = self.answer(messages)
        model_response.model = model
        return model_response

litellm.custom_provider_map = [
    {"provider": "faq", "custom_handler": Faq()},   # "faq/anything" now goes to Faq
]

for question in ["How long does a refund take?", "When will it ship?", "Do you sell tents?"]:
    response = completion(model="faq/v1", messages=[{"role": "user", "content": question}])
    print(response.choices[0].message.content, "|", response.model, "|", response.choices[0].finish_reason)
  • The first two questions match a keyword. refund and ship each return their own reply.
  • The third matches nothing. The class returns its fallback sentence, so the reply object is always filled in.
  • response.model is v1. It is the part after the slash, set by the line model_response.model = model.
  • finish_reason is stop. LiteLLM filled it in because the class did not set it.

Streaming from a custom provider

A call with stream=True goes to a separate method, streaming(). It is a generator that yields one small dictionary per chunk, with the text in text and is_finished set on the last one.

python
    def streaming(self, model, messages, **kwargs):
        words = self.answer(messages).split()
        for i, word in enumerate(words):
            last = i == len(words) - 1
            yield {"text": word + " ", "is_finished": last, "finish_reason": "stop" if last else "",
                   "index": 0, "tool_use": None,
                   "usage": {"completion_tokens": 1, "prompt_tokens": 0, "total_tokens": 1}}
ExampleStreaming the refund answer
for chunk in completion(model="faq/v1", stream=True,
                        messages=[{"role": "user", "content": "refund?"}]):
    print(repr(chunk.choices[0].delta.content), chunk.choices[0].finish_reason)

Each word arrives as its own chunk, and the last chunk reports stop. A class that defines only completion() has no streaming(), and a streaming call to it fails. LiteLLM's base class raises Not implemented yet! for any method you do not write.

Built-in provider vs CustomLLM

Built-in provider (groq/, gemini/)CustomLLM (your prefix)
Who writes the callLiteLLMYou
Needs an API keyYesOnly if your code needs one
StreamingIncludedWrite streaming()
Price and token countsFilled in from the model table and the replyNot filled in; usage reads zero unless you set it

Where you use a custom provider

  • Wrapping an internal API. Your own model service answers like any other provider, behind the same call.
  • Testing without a network. A provider that returns fixed replies, or fails on demand, makes tests that run the same way each time. The Router lessons use it for that, starting with Router: model groups and deployments.
  • A rule-based first answer. A bot that handles the known questions and passes the rest to a model.
Watch out. A custom provider reports no cost. response._hidden_params.get("response_cost") is None for it, so any cost tracking built on that field needs to treat a custom provider as unpriced instead of free.
Try it yourself
  • Add a "price" keyword to ANSWERS and ask What is the price?.
  • Register the same class under a second prefix, help, and call help/v1.
  • Delete the **kwargs from completion() and read the error LiteLLM raises when it passes keyword arguments the method does not take.

Little by little, you're building something great.