Curious builder0 XP earned · 300 to level 2

0 daysFinish a lesson to begin

Badge collection0 of 6 unlocked

51 small wins to finish your pathNext question →

Q27IntermediateSystem design

Design a batch pipeline to classify and tag 50 million product reviews with an LLM.

30-second answerSay your answer out loud first, then reveal.

Stage the data in object storage, split it into shards, and process with a distributed job (queue + workers or Spark/Ray) that calls the LLM via batch APIs or self-hosted high-throughput inference. Use a strict output schema, validation and retries, idempotent checkpointing per shard, and a results table. First run a pilot on a labelled sample to choose the cheapest model meeting the quality bar. Consider distilling to a small classifier if this is a recurring job.

Batch pipeline: 50M reviews are sharded into files, queued, processed by workers that dedupe, batch and call an LLM batch API or vLLM cluster, then validated, with results written to a table and a checkpoint per shard; a pilot on 2K labelled reviews picks the model and prompt.

Sizing example

text

50M reviews × ~150 tokens input + ~100 tokens prompt overhead = 12.5B input tokens
Output ~30 tokens (JSON labels) = 1.5B output tokens

Batch API discounts and a small model make this affordable. A frontier model may be unnecessary for sentiment and topic tags.

Design points

Pilot first: compare 2–3 models on 1–2K human-labelled reviews; pick by accuracy per dollar.
Throughput: batch APIs (24h turnaround, cheaper) or self-hosted vLLM with large batches. Respect rate limits.
Efficiency: deduplicate identical reviews; pack several short reviews per prompt (with IDs) when quality holds; keep outputs tiny.
Robustness: schema validation, retries with backoff, a dead-letter queue for repeated failures, idempotent writes by review_id.
Quality monitoring: sample outputs for human audit; track the label distribution against expectations.
Recurring jobs: if new reviews arrive daily, distil LLM labels into a fine-tuned small model for cheap ongoing inference, and use the LLM only for low-confidence cases.

PreviousA month after launch, the LLM bill is 5x the forecast. How do you investigate and fix it?

You understood something today that you didn't yesterday.

Design a batch pipeline to classify and tag 50 million product reviews with an LLM.

Sizing example

Design points

Related