LiteLLMLiteLLM 1.105 · Python 3.10+
Dashboard
0%

MCP tools through the gateway

An MCP gateway is the part of the LiteLLM proxy that lists and calls the tools of several MCP servers behind one endpoint, /mcp/, protected by the gateway's own keys.

Last updated: 12 Oct, 2026 · LiteLLM 1.105

Observability: watching every call watched model calls. Models are not the only thing an application asks the gateway for. MCP, the Model Context Protocol, is a way for a server to publish tools so that any client can list them and call them. Put the MCP servers behind the gateway and the argument that applied to models applies to tools: applications hold one key, and the gateway holds the connections. The MCP gateway page calls this a fixed endpoint for all MCP tools. This lesson builds a small tool server and calls its tool through a real gateway.

Syntax: servers go under a top-level mcp_servers key in the config. The page shows a server with a url for HTTP, or with transport: "stdio", a command and args for a process the gateway starts. transport defaults to http in config.yaml, and auth_type with auth_value says how to authenticate to the server.

yaml
mcp_servers:
  shop:
    url: "http://127.0.0.1:8931/mcp"
    transport: "http"
  # a process the gateway starts, from the docs:
  # circleci:
  #   transport: "stdio"
  #   command: "npx"
  #   args: ["-y", "@circleci/mcp-server-circleci"]

Writing a tool server

The tool server is a Python file that defines one tool, lookup_order, with the mcp package that litellm[proxy] installs. This course's environment has mcp 2.3.0, whose server class is MCPServer. The class name can differ between releases of the package, so check it against the version you install.

python
from mcp.server import MCPServer

mcp = MCPServer("shop")

ORDERS = {"A17": "shipped on 9 Oct", "B02": "waiting for payment"}

@mcp.tool()
def lookup_order(order_id: str) -> str:
    """Return the status of an order."""
    return ORDERS.get(order_id, "no such order")

if __name__ == "__main__":
    mcp.run("streamable-http", port=8931)

Calling the gateway when the tool server is down

The gateway speaks MCP itself, at /mcp/, as JSON-RPC over HTTP. A client sends its gateway key in the x-litellm-api-key header, with the Bearer prefix, and asks for tools/list. The first run starts only the gateway, with the config above and nothing listening on port 8931.

ExampleListing tools while the shop server is not running
import json
import subprocess
import time
import urllib.request
from pathlib import Path

import httpx


def start(port=4400, env=None):
    proc = subprocess.Popen(
        ["litellm", "--config", "config.yaml", "--port", str(port)],
        stdout=open("gateway.log", "w"), stderr=subprocess.STDOUT, env=env,
    )
    while True:
        try:
            urllib.request.urlopen(f"http://127.0.0.1:{port}/health/liveliness")
            return proc
        except OSError:
            time.sleep(1)


Path("config.yaml").write_text("""model_list:
  - model_name: careful
    litellm_params:
      model: groq/openai/gpt-oss-120b
      api_key: os.environ/GROQ_API_KEY

mcp_servers:
  shop:
    url: "http://127.0.0.1:8931/mcp"
    transport: "http"

general_settings:
  master_key: sk-gateway-1234
""")

def rpc(method, params=None):
    body = {"jsonrpc": "2.0", "id": 1, "method": method}
    if params:
        body["params"] = params
    r = httpx.post(
        "http://127.0.0.1:4400/mcp/", json=body, timeout=60,
        headers={"x-litellm-api-key": "Bearer sk-gateway-1234",
                 "Accept": "application/json, text/event-stream"},
    )
    data = next(l for l in r.text.splitlines() if l.startswith("data:"))
    return json.loads(data[5:])["result"]


proc = start()
try:
    result = rpc("tools/list")
    print(result["tools"])
    print(result["_meta"])
finally:
    proc.terminate()
    proc.wait()
  • The list is empty. The gateway has no tools to offer, and it did not raise an error.
  • The reason is in the metadata. The result carries server_outcomes, and the shop server's status is unreachable. A client that only reads tools would conclude the shop has no tools. Read the outcome too.

Listing and calling a tool through the gateway

Now the tool server runs. The example starts it as a child process, waits for its port, starts the gateway, lists the tools, and calls the tool for a real order and for one that does not exist. The finally block stops the tool server as well as the gateway.

ExampleA tool listed and called through the gateway
import json
import socket
import subprocess
import sys
import time
import urllib.request
from pathlib import Path

import httpx


def start(port=4400, env=None):
    proc = subprocess.Popen(
        ["litellm", "--config", "config.yaml", "--port", str(port)],
        stdout=open("gateway.log", "w"), stderr=subprocess.STDOUT, env=env,
    )
    while True:
        try:
            urllib.request.urlopen(f"http://127.0.0.1:{port}/health/liveliness")
            return proc
        except OSError:
            time.sleep(1)


Path("shop_tools.py").write_text(r'''from mcp.server import MCPServer

mcp = MCPServer("shop")

ORDERS = {"A17": "shipped on 9 Oct", "B02": "waiting for payment"}


@mcp.tool()
def lookup_order(order_id: str) -> str:
    """Return the status of an order."""
    return ORDERS.get(order_id, "no such order")


if __name__ == "__main__":
    mcp.run("streamable-http", port=8931)
''')
Path("config.yaml").write_text("""model_list:
  - model_name: careful
    litellm_params:
      model: groq/openai/gpt-oss-120b
      api_key: os.environ/GROQ_API_KEY

mcp_servers:
  shop:
    url: "http://127.0.0.1:8931/mcp"
    transport: "http"

general_settings:
  master_key: sk-gateway-1234
""")

def rpc(method, params=None):
    body = {"jsonrpc": "2.0", "id": 1, "method": method}
    if params:
        body["params"] = params
    r = httpx.post(
        "http://127.0.0.1:4400/mcp/", json=body, timeout=60,
        headers={"x-litellm-api-key": "Bearer sk-gateway-1234",
                 "Accept": "application/json, text/event-stream"},
    )
    data = next(l for l in r.text.splitlines() if l.startswith("data:"))
    return json.loads(data[5:])["result"]


proc = start()
try:
    tools = subprocess.Popen([sys.executable, "shop_tools.py"], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
    for _ in range(40):
        try:
            socket.create_connection(("127.0.0.1", 8931), timeout=1).close()
            break
        except OSError:
            time.sleep(0.5)
    listed = rpc("tools/list")
    print([t["name"] for t in listed["tools"]], listed["_meta"]["litellm.ai/server_outcomes"])
    for order in ["A17", "Z99"]:
        result = rpc("tools/call", {"name": "shop-lookup_order", "arguments": {"order_id": order}})
        print(order, result["content"][0]["text"], result["isError"])
finally:
    proc.terminate()
    proc.wait()
    tools.terminate()
    tools.wait()
  • The tool name has the server's name in front. The gateway lists shop-lookup_order, not lookup_order. The prefix tells apart servers that define a tool with the same name, and a call must use the prefixed name. The MCP page says tools are prefixed with the server name but uses a hyphen in one example and an underscore in another, so trust the listing your own gateway returns, as this run did.
  • The server is now ok. The outcome reports one tool.
  • The tool ran in the other process. The answer shipped on 9 Oct came from the dictionary in shop_tools.py. The caller never connected to port 8931.
  • An unknown order is a normal answer. The tool returned its own text, no such order, and isError stays false because the tool did not fail.

What the documentation adds

  • Per-server paths. The page shows a path for one server in the form http://localhost:4000/github_mcp/mcp, beside the combined /mcp/.
  • Stdio servers need an environment variable. The page says LITELLM_ENABLE_MCP_STDIO=true must be set in the proxy's process environment, and that setting it in the config has no effect.
  • Storing servers in the database. The page says saving MCP servers to the database needs store_model_in_db. Servers listed in config.yaml, as here, ran without a database. Database-backed servers are not run here.

Tools through the gateway vs direct

Direct to the MCP serverThrough the gateway
What the app holdsThe server's address and credentialsOne gateway key
Tool namesAs the server defines themPrefixed with the server's name
Many serversOne connection per serverOne endpoint, /mcp/

Where an MCP gateway fits

  • Agents that share tools. Several agents list the same tools from one address.
  • Keeping tool credentials out of applications. The gateway holds the server's auth_value.
  • One place to put a rule. Access to tools is decided at the gateway and not in each agent.
Watch out. The gateway's MCP support and the mcp package move together. The server file here uses MCPServer from mcp 2.3.0 and the gateway is 1.105.0. Pin both, and re-run this example after upgrading either.
Try it yourself
  • Add a second tool to shop_tools.py, restart both processes and list the tools again.
  • Call lookup_order without the shop- prefix and read the answer.
  • Change the key in the x-litellm-api-key header and read what the gateway answers.

Little by little, you're building something great.