How do you reduce an agent's latency?

Where time goes: total latency ≈ Σ (LLM time-to-first-token + generation time) + Σ tool latency, across sequential steps. Long outputs and many steps are the usual culprits.

Techniques

1. Fewer steps: combine tools so one call does what three did; hard-code predictable steps; give the agent all the context it needs upfront so it doesn't need discovery calls.
2. Parallelism: parallel tool calls in one turn, parallel sub-agents, and speculative execution (start likely tool calls early).
3. Model routing: a small, fast model for classification, extraction and simple tool picks; the large model only where reasoning is needed.
4. Fewer output tokens: generation is the slow part. Ask for concise reasoning and structured short outputs; limit verbose chain-of-thought where it doesn't help.
5. Prompt caching: stable prefixes (system prompt, tool definitions) reduce time-to-first-token on long prompts.
6. Streaming / progress UX: stream the final answer, and show progress ("Searching orders...") to cut perceived latency.
7. Faster tools: caching, indexes, timeouts, async I/O.
8. Async mode: for long tasks, return immediately and notify the user when done.

Related

• Previous: Q26. Your customer-support agent issued a refund that violated policy. What happened, and how do you prevent it?
• Next: Q28. How do you reduce an agent's cost?