Design a multi-tenant platform where customers can deploy their own agents (code execution, tools, memory).

30-second answerSay your answer out loud first, then reveal.

The core concerns are isolation, security, fairness and observability. Run each agent's code execution in a strong sandbox (microVMs like Firecracker or gVisor containers) with no default network access. Isolate tenant data (namespaced memory and vector stores, row-level security), keep per-tenant secrets in a vault, enforce quotas and rate limits per tenant, use durable execution for long runs, and provide per-tenant tracing, cost metering and kill switches.

A multi-tenant agent platform: the control plane API feeds a per-tenant fair run queue and kill switch, the queue feeds durable orchestrator workers, and the workers use a model gateway, a sandbox pool whose traffic leaves through a per-tenant egress proxy, a tool gateway, tenant-namespaced memory, and tracing and billing.

Key areas

Compute isolation: untrusted, model-generated code must never run on shared hosts unsandboxed. Use microVMs or gVisor per session; CPU/memory/time limits; ephemeral filesystems; destroy after the run.
Network: deny by default; egress through a proxy with per-tenant allow-lists. This blocks data exfiltration and abuse such as crypto-mining or attacks from your IPs.
Data isolation: tenant ID enforced at the storage layer (separate indexes or row-level security), not just in app code. Test for cross-tenant leakage.
Secrets: a vault, injected at the tool gateway; never in prompts, logs or sandbox env vars unless required.
Fairness and quotas: per-tenant concurrency limits, token budgets, rate limits at the model gateway, priority queues.
Reliability: durable execution for long runs; retries; regional failover.
Observability and billing: per-tenant traces, cost metering (tokens, compute seconds, tool calls), anomaly alerts.
Governance: audit logs, data retention controls, abuse detection, an emergency kill switch per tenant or agent.

Trade-offs to discuss: microVM cold-start latency vs isolation strength (warm pools help); shared vs dedicated vector indexes (cost vs isolation); letting tenants bring their own model keys.

PreviousYour agent reports "Done! The record has been updated," but the record was never updated. How do you detect and prevent this?

Slow is fine. Stopping is the only problem.

Design a multi-tenant platform where customers can deploy their own agents (code execution, tools, memory).

Key areas

Related