When do you use synchronous, streaming, or asynchronous/batch processing for LLM features?

30-second answerSay your answer out loud first, then reveal.

Use streaming for interactive generation (chat, writing help), so users see tokens immediately even if the full answer takes seconds. Use synchronous non-streaming calls for short, structured outputs consumed by code (classification, extraction) under a few seconds. Use async jobs with queues for long-running tasks (deep research, large document processing), with progress updates and notifications. Use batch for large offline volumes where latency doesn't matter (cheaper, often around 50% off with batch APIs).

Mode	Latency need	Example	Notes
Streaming	Interactive, perceived speed	Chat, drafting emails	SSE/WebSockets; handle cancellation
Sync request/response	Short, needed before next step	Intent classification, JSON extraction in a workflow	Set timeouts; small fast models
Async job	Seconds to hours	Report generation, video summarisation, agent tasks	Queue + workers + status API + webhook/notification
Batch	Hours acceptable	Tag 10M products, nightly summaries, embeddings backfill	Batch APIs, spot GPUs, max throughput

Async pattern

text

POST /jobs → 202 Accepted {job_id}
Worker pulls from queue → processes (with checkpoints) → stores result
Client polls GET /jobs/{id} or receives webhook / push notification

Design considerations

Timeouts: gateways and load balancers often cut connections at 30–60s, so long tasks must be async.
Idempotency: retries of job submission must not duplicate work.
Progress UX: stream intermediate steps ("Reading 12 documents...").
Priority queues: interactive traffic must not be starved by batch jobs sharing the same model capacity.

PreviousHow do you do back-of-the-envelope estimation for an LLM system?

Every expert started right here.

When do you use synchronous, streaming, or asynchronous/batch processing for LLM features?

Async pattern

Design considerations

Related