LiteLLMLiteLLM 1.105 · Python 3.10+
Dashboard
0%

Tool calling: when the model asks for data

Tool calling is a feature of chat models that lets the model ask your program to run a function and send back the result, so the answer can use data the model was never trained on.

Last updated: 12 Oct, 2026 · LiteLLM 1.105

Endpoints beyond chat listed what LiteLLM can call besides chat. This lesson stays with chat and adds tools. A model cannot look up an order or read a database. It can write out a request for one, and your program does the lookup. LiteLLM sends the same tool description to every provider. It hands the request back in one shape.

The tool calling loop between your program and a model. Step 1: the program sends the question and the tools list. Step 2: the model replies with finish_reason tool_calls and a call to lookup_order with order A17. Step 3: the program runs the function itself. Step 4: the program sends the history plus a tool message holding the result. Step 5: the model answers with finish_reason stop. If step 5 asks for another tool call, steps 3 and 4 repeat.

The model never runs anything. It answers a question with a request, your program runs the function, and the result goes back to the model as one more message. That exchange can happen more than once before the model gives its answer.

Syntax: pass tools to completion(), then read tool_calls on the reply message.

python
response = completion(
    model="groq/openai/gpt-oss-20b",
    messages=messages,       # the conversation so far
    tools=TOOLS,             # the functions the model may ask for
    max_tokens=1000,         # reasoning models spend tokens thinking first
)
message = response.choices[0].message
message.tool_calls           # None for an answer, a list for a request

Describing a tool to the model

The tool description

A tool is a dictionary in the OpenAI shape. The name is what the model writes back when it wants the tool. The description is the text the model reads to decide when the tool fits, so it should say what the tool does and give an example id. parameters is a JSON Schema that says which arguments exist and which are required.

python
TOOLS = [{
    "type": "function",
    "function": {
        "name": "lookup_order",                    # what the model writes when it wants this tool
        "description": "Look up one order by its id, such as A17.",
        "parameters": {                            # JSON Schema for the arguments
            "type": "object",
            "properties": {"order_id": {"type": "string"}},
            "required": ["order_id"],
        },
    },
}]

Reading the model's request

The question below is about one order. The model has no order data, so it should reply with a request instead of text. finish_reason says which of the two it was. function.arguments arrives as a JSON string, not a dictionary. max_tokens is 1000 because a gpt-oss model spends part of its budget reasoning before it writes anything. A small budget comes back with no content at all.

ExampleAPI keyThe model asks for lookup_order instead of answering
from litellm import completion

response = completion(
    model="groq/openai/gpt-oss-20b",
    messages=[{"role": "user", "content": "Where is my order A17?"}],
    tools=TOOLS,
    max_tokens=1000,
)
choice = response.choices[0]
call = choice.message.tool_calls[0]
print(choice.finish_reason)
print(choice.message.content)
print(call.function.name, call.function.arguments)
  • finish_reason is tool_calls, so this reply is a request, not an answer.
  • content is None. The model wrote no text, only the request.
  • function.arguments holds the order id the model pulled out of the question, as JSON text. json.loads turns it into a dictionary.

Sending the result back

One reply is not always enough. Ask about two orders and the model may request one, wait for its result, and then request the other. A program that answers after a single round would stop at a message with no text. The fix is a loop that runs while the reply still carries tool_calls.

Running the function

The function here is a dictionary lookup. In a real program it would query a database or call an API.

python
ORDERS = {"A17": "shipped on 3 March", "C40": "waiting for stock"}

def run_tool(call):
    order = json.loads(call.function.arguments)["order_id"]
    return ORDERS.get(order, "no such order")

Adding the request and the result to the history

Two messages go back: the model's own request, copied unchanged, and a tool message with the result. The tool message carries the tool_call_id of the request it answers, which is how the model matches results to requests.

python
messages.append(message.model_dump(exclude_none=True))   # the model's request
for call in message.tool_calls:
    messages.append({
        "role": "tool",
        "tool_call_id": call.id,       # which request this answers
        "content": run_tool(call),
    })

Looping until the model answers

The loop calls the model, stops when the reply has no tool calls, and otherwise runs the tools and goes round again. A turn limit keeps a confused model from looping forever.

python
for turn in range(4):                          # a limit, not an expectation
    response = completion(model=model, messages=messages, tools=TOOLS, max_tokens=1000)
    message = response.choices[0].message
    if not message.tool_calls:                 # no request: this is the answer
        break
    # ... add the request and the results, as shown above

Two models answering the same two-order question

The same loop runs against gpt-oss-20b on Groq and gemini-2.5-flash-lite. The system message keeps each answer to one line per order, so the model reports only what the tool returned. Each line printed before the answer shows one turn.

ExampleAPI keyOne loop, two providers; the wording of the answers changes run to run
import json
from litellm import completion

ORDERS = {"A17": "shipped on 3 March", "C40": "waiting for stock"}
TOOLS = [{"type": "function", "function": {
    "name": "lookup_order",
    "description": "Look up one order by its id, such as A17.",
    "parameters": {"type": "object",
                   "properties": {"order_id": {"type": "string"}},
                   "required": ["order_id"]},
}}]

for model in ["groq/openai/gpt-oss-20b", "gemini/gemini-2.5-flash-lite"]:
    messages = [
        {"role": "system", "content": "Answer in one short line per order, using only the tool results."},
        {"role": "user", "content": "Where are my orders A17 and C40?"},
    ]
    for turn in range(4):
        response = completion(model=model, messages=messages, tools=TOOLS, max_tokens=1000)
        message = response.choices[0].message
        asked = [call.function.arguments for call in message.tool_calls or []]
        print(model, "turn", turn, response.choices[0].finish_reason, asked)
        if not message.tool_calls:
            break
        messages.append(message.model_dump(exclude_none=True))
        for call in message.tool_calls:
            order = json.loads(call.function.arguments)["order_id"]
            messages.append({"role": "tool", "tool_call_id": call.id,
                             "content": ORDERS.get(order, "no such order")})
    print(message.content)
  • gpt-oss-20b asked for one order per turn. It requested A17, got the result, then requested C40, so the answer came on the third turn.
  • gemini-2.5-flash-lite put both requests in one reply. Its first turn carries two entries in tool_calls, and the loop answers both before going round again.
  • The loop did not change. It handles one request or several, because it always works through message.tool_calls as a list.

One call per reply vs several

One call per replySeveral calls in one reply
Seen ongpt-oss-20bgemini-2.5-flash-lite
Turns for two ordersThree: two requests, then the answerTwo: one reply with both requests, then the answer
What your loop needsTo go round again after each resultTo answer every entry in the list before going round
Safe codeA loop over message.tool_callsThe same loop

Where you use tool calling

  • Looking up live data. Order status, stock levels or a customer record that no model was trained on.
  • Doing arithmetic or date maths. The model asks a calculator function and quotes its result.
  • Taking an action. Creating a ticket or sending a message, with your program deciding whether to allow it before it runs.
Watch out. A reply that carries tool_calls is not an answer, and its content is empty. Code that prints message.content after one call shows None for any question that needs more than one lookup. Loop until tool_calls is empty, and cap the number of turns.
Try it yourself
  • Ask about one order, Where is my order C40?, and count the turns each model needs.
  • Change range(4) to range(2) and run gpt-oss-20b again. It ends without an answer and prints None.
  • Ask about an order that is not in ORDERS, such as Z99, and read how the model reports the no such order result.

Slow is fine. Stopping is the only problem.