MCP tools through the gateway
An MCP gateway is the part of the LiteLLM proxy that lists and calls the tools of several MCP servers behind one endpoint, /mcp/, protected by the gateway's own keys.
Last updated: 12 Oct, 2026 · LiteLLM 1.105
Observability: watching every call watched model calls. Models are not the only thing an application asks the gateway for. MCP, the Model Context Protocol, is a way for a server to publish tools so that any client can list them and call them. Put the MCP servers behind the gateway and the argument that applied to models applies to tools: applications hold one key, and the gateway holds the connections. The MCP gateway page calls this a fixed endpoint for all MCP tools. This lesson builds a small tool server and calls its tool through a real gateway.
Syntax: servers go under a top-level mcp_servers key in the config. The page shows a server with a url for HTTP, or with transport: "stdio", a command and args for a process the gateway starts. transport defaults to http in config.yaml, and auth_type with auth_value says how to authenticate to the server.
mcp_servers:
shop:
url: "http://127.0.0.1:8931/mcp"
transport: "http"
# a process the gateway starts, from the docs:
# circleci:
# transport: "stdio"
# command: "npx"
# args: ["-y", "@circleci/mcp-server-circleci"]Writing a tool server
The tool server is a Python file that defines one tool, lookup_order, with the mcp package that litellm[proxy] installs. This course's environment has mcp 2.3.0, whose server class is MCPServer. The class name can differ between releases of the package, so check it against the version you install.
from mcp.server import MCPServer
mcp = MCPServer("shop")
ORDERS = {"A17": "shipped on 9 Oct", "B02": "waiting for payment"}
@mcp.tool()
def lookup_order(order_id: str) -> str:
"""Return the status of an order."""
return ORDERS.get(order_id, "no such order")
if __name__ == "__main__":
mcp.run("streamable-http", port=8931)Calling the gateway when the tool server is down
The gateway speaks MCP itself, at /mcp/, as JSON-RPC over HTTP. A client sends its gateway key in the x-litellm-api-key header, with the Bearer prefix, and asks for tools/list. The first run starts only the gateway, with the config above and nothing listening on port 8931.
import json
import subprocess
import time
import urllib.request
from pathlib import Path
import httpx
def start(port=4400, env=None):
proc = subprocess.Popen(
["litellm", "--config", "config.yaml", "--port", str(port)],
stdout=open("gateway.log", "w"), stderr=subprocess.STDOUT, env=env,
)
while True:
try:
urllib.request.urlopen(f"http://127.0.0.1:{port}/health/liveliness")
return proc
except OSError:
time.sleep(1)
Path("config.yaml").write_text("""model_list:
- model_name: careful
litellm_params:
model: groq/openai/gpt-oss-120b
api_key: os.environ/GROQ_API_KEY
mcp_servers:
shop:
url: "http://127.0.0.1:8931/mcp"
transport: "http"
general_settings:
master_key: sk-gateway-1234
""")
def rpc(method, params=None):
body = {"jsonrpc": "2.0", "id": 1, "method": method}
if params:
body["params"] = params
r = httpx.post(
"http://127.0.0.1:4400/mcp/", json=body, timeout=60,
headers={"x-litellm-api-key": "Bearer sk-gateway-1234",
"Accept": "application/json, text/event-stream"},
)
data = next(l for l in r.text.splitlines() if l.startswith("data:"))
return json.loads(data[5:])["result"]
proc = start()
try:
result = rpc("tools/list")
print(result["tools"])
print(result["_meta"])
finally:
proc.terminate()
proc.wait()[]
{'litellm.ai/server_outcomes': {'shop': {'status': 'unreachable'}}}- The list is empty. The gateway has no tools to offer, and it did not raise an error.
- The reason is in the metadata. The result carries
server_outcomes, and theshopserver's status isunreachable. A client that only readstoolswould conclude the shop has no tools. Read the outcome too.
Listing and calling a tool through the gateway
Now the tool server runs. The example starts it as a child process, waits for its port, starts the gateway, lists the tools, and calls the tool for a real order and for one that does not exist. The finally block stops the tool server as well as the gateway.
import json
import socket
import subprocess
import sys
import time
import urllib.request
from pathlib import Path
import httpx
def start(port=4400, env=None):
proc = subprocess.Popen(
["litellm", "--config", "config.yaml", "--port", str(port)],
stdout=open("gateway.log", "w"), stderr=subprocess.STDOUT, env=env,
)
while True:
try:
urllib.request.urlopen(f"http://127.0.0.1:{port}/health/liveliness")
return proc
except OSError:
time.sleep(1)
Path("shop_tools.py").write_text(r'''from mcp.server import MCPServer
mcp = MCPServer("shop")
ORDERS = {"A17": "shipped on 9 Oct", "B02": "waiting for payment"}
@mcp.tool()
def lookup_order(order_id: str) -> str:
"""Return the status of an order."""
return ORDERS.get(order_id, "no such order")
if __name__ == "__main__":
mcp.run("streamable-http", port=8931)
''')
Path("config.yaml").write_text("""model_list:
- model_name: careful
litellm_params:
model: groq/openai/gpt-oss-120b
api_key: os.environ/GROQ_API_KEY
mcp_servers:
shop:
url: "http://127.0.0.1:8931/mcp"
transport: "http"
general_settings:
master_key: sk-gateway-1234
""")
def rpc(method, params=None):
body = {"jsonrpc": "2.0", "id": 1, "method": method}
if params:
body["params"] = params
r = httpx.post(
"http://127.0.0.1:4400/mcp/", json=body, timeout=60,
headers={"x-litellm-api-key": "Bearer sk-gateway-1234",
"Accept": "application/json, text/event-stream"},
)
data = next(l for l in r.text.splitlines() if l.startswith("data:"))
return json.loads(data[5:])["result"]
proc = start()
try:
tools = subprocess.Popen([sys.executable, "shop_tools.py"], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
for _ in range(40):
try:
socket.create_connection(("127.0.0.1", 8931), timeout=1).close()
break
except OSError:
time.sleep(0.5)
listed = rpc("tools/list")
print([t["name"] for t in listed["tools"]], listed["_meta"]["litellm.ai/server_outcomes"])
for order in ["A17", "Z99"]:
result = rpc("tools/call", {"name": "shop-lookup_order", "arguments": {"order_id": order}})
print(order, result["content"][0]["text"], result["isError"])
finally:
proc.terminate()
proc.wait()
tools.terminate()
tools.wait()['shop-lookup_order'] {'shop': {'status': 'ok', 'tool_count': 1}}
A17 shipped on 9 Oct False
Z99 no such order False- The tool name has the server's name in front. The gateway lists
shop-lookup_order, notlookup_order. The prefix tells apart servers that define a tool with the same name, and a call must use the prefixed name. The MCP page says tools are prefixed with the server name but uses a hyphen in one example and an underscore in another, so trust the listing your own gateway returns, as this run did. - The server is now
ok. The outcome reports one tool. - The tool ran in the other process. The answer
shipped on 9 Octcame from the dictionary inshop_tools.py. The caller never connected to port 8931. - An unknown order is a normal answer. The tool returned its own text,
no such order, andisErrorstays false because the tool did not fail.
What the documentation adds
- Per-server paths. The page shows a path for one server in the form
http://localhost:4000/github_mcp/mcp, beside the combined/mcp/. - Stdio servers need an environment variable. The page says
LITELLM_ENABLE_MCP_STDIO=truemust be set in the proxy's process environment, and that setting it in the config has no effect. - Storing servers in the database. The page says saving MCP servers to the database needs
store_model_in_db. Servers listed inconfig.yaml, as here, ran without a database. Database-backed servers are not run here.
Tools through the gateway vs direct
| Direct to the MCP server | Through the gateway | |
|---|---|---|
| What the app holds | The server's address and credentials | One gateway key |
| Tool names | As the server defines them | Prefixed with the server's name |
| Many servers | One connection per server | One endpoint, /mcp/ |
Where an MCP gateway fits
- Agents that share tools. Several agents list the same tools from one address.
- Keeping tool credentials out of applications. The gateway holds the server's
auth_value. - One place to put a rule. Access to tools is decided at the gateway and not in each agent.
mcp package move together. The server file here uses MCPServer from mcp 2.3.0 and the gateway is 1.105.0. Pin both, and re-run this example after upgrading either.Related
- Previous: Observability: watching every call
- Next: Fallbacks and caching in the gateway config
- Reference: MCP gateway in the LiteLLM docs
- Add a second tool to
shop_tools.py, restart both processes and list the tools again. - Call
lookup_orderwithout theshop-prefix and read the answer. - Change the key in the
x-litellm-api-keyheader and read what the gateway answers.
Little by little, you're building something great.