How do you serve many fine-tuned variants (e.g. per-customer LoRA adapters) efficiently?
30-second answerSay your answer out loud first, then reveal.
Architecture
Request {tenant: "acme", adapter: "acme-support@v3"} → gateway → inference pool
Inference pool: base model (e.g. 8B) + adapter cache in GPU/CPU memory
- Hot adapters resident in GPU memory
- Warm adapters in CPU memory; cold ones loaded from the registry on demandBenefits: hundreds of variants on a few GPUs; fast onboarding of new adapters; cost-efficient per-tenant customisation.
Considerations
| Concern | Approach |
|---|---|
| Latency of cold adapter loads | Pre-load popular adapters; async loading; small adapter sizes |
| Batching across adapters | Engines batch requests for different adapters together (with some overhead) |
| Base model upgrades | Adapters are tied to a base version, so retrain or revalidate when the base changes |
| Isolation | Tenant A must never get tenant B's adapter; enforce mapping server-side |
| Quality monitoring | Evals per adapter; detect regressions after retraining |
| Rank limits | Engines often cap max LoRA rank; plan adapter configs accordingly |
Related
You understood something today that you didn't yesterday.