The customer requires everything to run inside their own VPC or on-prem, with data in Oracle databases and a mainframe. Design the deployment.
30-second answerSay your answer out loud first, then reveal.
Considerations
- Models: pick open-weight models that fit their GPUs and meet the eval thresholds; quantize if needed; plan capacity (Q36 of the LLM guide); update models via the internal registry.
- Data access: prefer replicas or change-data-capture over hitting production Oracle; mainframe through existing integration layers (MQ, CICS web services, nightly extracts) owned by their team.
- Air-gapped operations: all images, packages and weights mirrored internally; signed releases; no telemetry leaving the network unless approved.
- Security: network policies, secrets in their vault, encryption, integration with their IAM and audit tools.
- Operations: runbooks, health checks, backups, upgrade procedures performed by or with their ops team; support access via their approved remote access process.
- Evals inside the perimeter: the eval harness runs locally; results shared as aggregate reports.
Trade-offs: lower model quality vs frontier APIs (quantify it with evals), slower update cycles, higher operational burden. Make these explicit to the sponsor.
Related
Slow is fine. Stopping is the only problem.