How do you handle multi-tenancy and noisy neighbours in a shared LLM platform?

30-second answerSay your answer out loud first, then reveal.

Isolate data (tenant-scoped indexes, caches, logs and adapters) and resources (per-tenant quotas on requests and tokens, concurrency limits, fair-share or weighted queues, priority tiers). Protect shared capacity from heavy users: token-aware rate limiting at the gateway, separate pools or dedicated capacity for large customers, and admission control during overload. Attribute costs per tenant for billing and capacity planning.

Mechanisms

Problem	Solution
One tenant's batch job saturates GPUs	Separate batch and interactive queues; quotas; schedule batch off-peak
Large prompts from one tenant blow up latency	Per-tenant max tokens; long-context pool
Provider rate limits shared across tenants	Token-bucket per tenant within the global budget; fair queuing
Premium SLAs	Priority classes; reserved capacity
Data isolation	Tenant ID enforced in retrieval filters, cache keys, adapter routing, logs
Cost recovery	Per-tenant metering; usage-based pricing or caps

Fair queuing example. Weighted fair queuing by tenant tier. Each tenant gets a share of capacity proportional to its plan weight when the system is contended, and can burst when it's idle.

Monitoring: per-tenant latency, errors, usage vs quota, and a "top consumers" dashboard to spot abuse or unexpected load early.

PreviousWhat security practices are specific to operating LLM systems (supply chain and runtime)?

Slow is fine. Stopping is the only problem.

How do you handle multi-tenancy and noisy neighbours in a shared LLM platform?

Mechanisms

Related