Secret managers

A secret manager is a service that stores secrets such as API keys, so that the gateway reads them from there instead of from environment variables you manage yourself.

Last updated: 12 Oct, 2026 · LiteLLM 1.105

Until now every provider key sat in an environment variable, and the config pointed at it with os.environ/GROQ_API_KEY. That works, and the key is then only as safe as the place that sets the variable: a deployment file, a shell profile, a container definition. A secret manager keeps keys in one guarded place. The secret managers overview says LiteLLM can read secrets such as OPENAI_API_KEY, and write secrets such as virtual keys, from Azure Key Vault, Google Secret Manager, Hashicorp Vault, CyberArk Conjur and AWS Secret Manager.

Syntax: one setting names the manager. The model entries keep using os.environ/NAME, and the gateway looks the name up in the manager. The Azure page comments its example line as reading from the key vault.

general_settings:
  key_management_system: "hashicorp_vault"   # which manager to ask

The names this version accepts

The installed 1.105.0 library lists these values for key_management_system: aws_secret_manager, aws_kms, azure_key_vault, google_secret_manager, google_kms, hashicorp_vault, cyberark, local and custom.

The settings around it

The overview lists these settings under key_management_settings. access_mode defaults to read-only. hosted_keys limits the lookup to named keys.

general_settings:
  key_management_system: "aws_secret_manager"
  key_management_settings:
    store_virtual_keys: true            # also write virtual keys to the manager
    prefix_for_stored_virtual_keys: "litellm/"
    access_mode: "write_only"           # read_only, write_only or read_and_write
    hosted_keys: ["litellm_master_key"] # only look these names up

Writing the helper that starts a gateway

ops.py starts the litellm command, waits until it answers, and stops it in a finally block. call() sends a request and returns the status, body and headers. chat() sends one chat request.

from pathlib import Path
import sys

Path("ops.py").write_text('''import json
import os
import subprocess
import sys
import time
import urllib.error
import urllib.request
from contextlib import contextmanager

KEY = "sk-gateway-1234"          # the master key written in every config.yaml


@contextmanager
def running(port, env=None):
    """Start the gateway on `port` from config.yaml, wait until it answers, stop it afterwards."""
    litellm = os.path.join(os.path.dirname(sys.executable), "litellm")
    cmd = [litellm, "--config", "config.yaml", "--port", str(port)]
    proc = subprocess.Popen(cmd, env=env, stdout=open("gateway.log", "w"), stderr=subprocess.STDOUT)
    try:
        for _ in range(90):
            if proc.poll() is not None:
                raise RuntimeError("the gateway stopped; read gateway.log")
            try:
                urllib.request.urlopen(f"http://127.0.0.1:{port}/health/liveliness")
                break
            except OSError:
                time.sleep(1)
        yield f"http://127.0.0.1:{port}"
    finally:
        proc.terminate()
        proc.wait(10)


def call(url, path, body=None, key=KEY, headers=None):
    """Send one request. Returns (status, text, headers) and does not raise on 4xx or 5xx."""
    head = dict(headers or {})
    if key:
        head["Authorization"] = f"Bearer {key}"
    if body is not None:
        head["Content-Type"] = "application/json"
    data = json.dumps(body).encode() if body is not None else None
    try:
        reply = urllib.request.urlopen(urllib.request.Request(url + path, data=data, headers=head), timeout=60)
    except urllib.error.HTTPError as error:
        reply = error
    return reply.getcode(), reply.read().decode(), {k.lower(): v for k, v in reply.headers.items()}


def chat(url, model, text="Reply with one word: ok", key=KEY, headers=None, **extra):
    """One chat request. Extra keyword arguments are added to the JSON body."""
    body = {"model": model, "messages": [{"role": "user", "content": text}], "max_tokens": 60, **extra}
    return call(url, "/chat/completions", body, key, headers)
''')
sys.path.insert(0, ".")        # lets the programs below run: from ops import running, call, chat
print("wrote ops.py")

wrote ops.py

Writing a secret manager of your own

The custom secret manager page shows how to connect any system. A class extends CustomSecretManager and implements two methods, async_read_secret and sync_read_secret, each taking a secret name and returning its value or None. This is the same contract the real integrations fill, so it is a way to run the feature here without an AWS or Vault account.

The stand-in vault

DeskVault is a dictionary, not a vault. It holds one secret, DESK_GROQ_KEY, taken from an environment variable that only the gateway process receives. Each read prints a line to the gateway's log, so the example can show that the gateway asked.

Pointing the config at it

key_management_system: custom turns the feature on. custom_secret_manager gives the module and class. The model entry names DESK_GROQ_KEY, which exists in no environment variable of the gateway, so the vault is the only place the key can come from.

general_settings:
  key_management_system: custom
  key_management_settings:
    custom_secret_manager: my_secret_manager.DeskVault
    hosted_keys: ["DESK_GROQ_KEY"]

Starting a gateway that reads its key from a vault

The program starts the gateway three times. The first config names the vault and limits the lookup with hosted_keys. The second leaves hosted_keys out. The third has no vault. In every run the gateway's environment has no GROQ_API_KEY. After each run the program counts the distinct names the vault was asked for in the gateway log, and prints the first two.

import json
import os
import re
from pathlib import Path
from ops import running, chat

Path("my_secret_manager.py").write_text('''import os
from typing import Optional, Union

import httpx
from litellm.integrations.custom_secret_manager import CustomSecretManager


class DeskVault(CustomSecretManager):
    """A stand-in vault: a dictionary that hands out secrets by name."""

    def __init__(self):
        super().__init__(secret_manager_name="desk_vault")
        self.secrets = {"DESK_GROQ_KEY": os.environ["DESK_VAULT_SEED"]}

    async def async_read_secret(self, secret_name: str, optional_params: Optional[dict] = None,
                                timeout: Optional[Union[float, httpx.Timeout]] = None) -> Optional[str]:
        print("VAULT READ", secret_name, flush=True)
        return self.secrets.get(secret_name)

    def sync_read_secret(self, secret_name: str, optional_params: Optional[dict] = None,
                         timeout: Optional[Union[float, httpx.Timeout]] = None) -> Optional[str]:
        print("VAULT READ", secret_name, flush=True)
        return self.secrets.get(secret_name)
''')

TEMPLATE = """model_list:
  - model_name: fast
    litellm_params:
      model: groq/openai/gpt-oss-20b
      api_key: os.environ/DESK_GROQ_KEY
general_settings:
  master_key: sk-gateway-1234
{vault}"""
VAULT_LINES = """  key_management_system: custom
  key_management_settings:
    custom_secret_manager: my_secret_manager.DeskVault
    hosted_keys: ["DESK_GROQ_KEY"]
"""

# the gateway gets the key only through the vault: no GROQ_API_KEY in its environment
env = {k: v for k, v in os.environ.items() if k != "GROQ_API_KEY"}
env["DESK_VAULT_SEED"] = os.environ["GROQ_API_KEY"]

RUNS = [
    ("vault, hosted_keys", VAULT_LINES),
    ("vault, no hosted_keys", VAULT_LINES.replace('    hosted_keys: ["DESK_GROQ_KEY"]\n', "")),
    ("no vault", ""),
]
for number, (label, vault) in enumerate(RUNS):
    Path("config.yaml").write_text(TEMPLATE.format(vault=vault))
    with running(4751 + number, env=env) as url:
        status, text, _ = chat(url, "fast")
    asked = sorted({l.split()[2] for l in open("gateway.log") if l.startswith("VAULT READ")})
    note = "" if status == 200 else re.search(r'"message":"(.*?)"', json.loads(text)["error"]["message"]).group(1)
    print(f"{label:22} {status}  vault asked for {len(asked)}: {asked[:2]} {note}")

vault, hosted_keys     200  vault asked for 1: ['DESK_GROQ_KEY'] 
vault, no hosted_keys  200  vault asked for 79: ['CONFIG_FILE_PATH', 'DATABASE_URL'] 
no vault               401  vault asked for 0: [] Invalid API Key

Reading what the vault supplied

• With the vault and hosted_keys, the call answered 200. The only source of the Groq key was the vault, and the vault was asked for one name, DESK_GROQ_KEY.
• Without hosted_keys, the call also answered 200, and the vault was asked for dozens of names. The first two show the pattern: the gateway looked up its own settings, such as DATABASE_URL, and Redis and cloud settings. With a real manager each lookup is a request to that service. The custom manager page lists hosted_keys as the list of keys to check in the manager, with all keys as the default.
• Without the vault, the call failed with a rejected key. The name DESK_GROQ_KEY was unset, so the gateway had no key to send. The log has no vault lines.
• The reads happen while the gateway loads its config. The log lines were there before any request was sent. The docs mention a refresh interval for the real managers.

Connecting a real secret manager

Each real manager needs its own account and credentials, so none is run here. These are the settings the docs pages name.

The AWS page also shows primary_secret_name, which reads several keys from one JSON secret. The AWS KMS and Google KMS pages are a different use: they keep an encrypted copy of a key, such as the master key, in the environment and decrypt it with the cloud's key service.

Environment variables vs a secret manager

Where you use a secret manager

• One place for provider keys. Several gateway copies read the same secrets.
• Rotation. A new key is stored once in the manager instead of in every deployment file.
• Access control. The manager decides which credentials may read which secret, which an environment variable cannot.

Related

• Previous: Tag routing
• Next: The auto router
• See also: Gateway keys for the master key, which the AWS page shows being read from the manager
• Reference: Secret Managers Overview and Custom Secret Manager in the LiteLLM docs