Generative AI and Agentic AILangChain 1.4 · LangGraph 1.2 · Python 3.12 or 3.13
Dashboard
0%

Streaming and batch

Streaming returns a model's reply in chunks as it is generated, and batching sends several independent requests at the same time and returns one reply for each.

Last updated: 08 Oct, 2026 · LangChain 1.4 · LangGraph 1.2

Every call so far waited for the whole reply. That is fine for a short answer and slow to watch for a long one, and a program with many questions loses time asking them one after another. stream and batch are the two calls that fix that.

invoke waits until the whole reply is ready and returns it at once. stream returns chunks one after another, so the first text appears early. batch sends several questions to the model at the same time and returns one answer for each.

Streaming a reply with stream

Why stream, and how stream works · from the Complete Agentic AI Course video · 50:12 to 54:28

The video starts from a long request, a 200-word paragraph on artificial intelligence. With invoke the program waits until the reply is complete, and the video's point is that showing the output progressively improves the experience for long replies. stream returns an iterator that yields chunks of the reply, so a for loop receives each chunk as the model produces it. The video prints chunk.text with end set to a separator and flush=True so that the chunk boundaries are visible.

python
for chunk in model.stream("Write a 100 word paragraph about parrots."):
    if chunk.text:                       # skip chunks that carry no text
        print(chunk.text, end="|", flush=True)

The model here reasons before it answers, and its reasoning arrives as chunks with empty text, which is why the loop skips them. reasoning_effort="low", a setting of this model, keeps that phase short.

ExampleAPI keyRun on Groq
from langchain.chat_models import init_chat_model

model = init_chat_model("groq:openai/gpt-oss-120b", temperature=0, reasoning_effort="low")

for chunk in model.stream("Write a 100 word paragraph about parrots."):
    if chunk.text:
        print(chunk.text, end="|", flush=True)

Each | marks the end of a chunk. A chunk is a few characters or a short word, and the paragraph arrives as pieces such as Par, rots, are and vibrant. Only chunks that carried text were printed.

Streaming does not make the model finish sooner. It lets the first text appear sooner. The next example times both calls on a longer reply: when the first text arrives from stream, and when invoke returns the whole reply.

ExampleAPI keyRun on Groq
import time

from langchain.chat_models import init_chat_model

model = init_chat_model("groq:openai/gpt-oss-120b", temperature=0, reasoning_effort="low")
question = "Write a 250 word paragraph about parrots."

start = time.time()
first = None
for chunk in model.stream(question):
    if chunk.text and first is None:
        first = time.time() - start
print(f"stream: first text after {first:.2f}s, finished after {time.time() - start:.2f}s")

start = time.time()
reply = model.invoke(question)
print(f"invoke: {len(reply.text.split())} words, all there after {time.time() - start:.2f}s")

In this run stream delivered the first text after 0.71 s and finished after 1.55 s, and invoke returned all 293 words after 1.41 s. The two finish at about the same time, because streaming does not make the model faster. What changes is when the reader has something to read: about 0.7 s into the wait, where invoke makes the reader sit through the full 1.4 s.

How long the first text takes depends on how much the model reasons first. The same stream at two settings shows it:

ExampleAPI keyRun on Groq
import time

from langchain.chat_models import init_chat_model

for effort in ["low", "medium"]:
    model = init_chat_model("groq:openai/gpt-oss-120b", temperature=0, reasoning_effort=effort)
    start, first, chunks, with_text = time.time(), None, 0, 0
    for chunk in model.stream("Write a 120 word paragraph about parrots."):
        chunks += 1
        if chunk.text:
            with_text += 1
            first = first or time.time() - start
    print(f"{effort:>6}: {chunks} chunks, {with_text} with text, first text after {first:.2f}s")

At low, 245 chunks arrived and the first text appeared after 0.61 s. At medium, 1182 chunks arrived but only 174 carried text: about a thousand chunks were reasoning, and the first text came after 2.69 s. The reasoning chunks have empty text, which is why the loop checks chunk.text.

Keeping the whole reply from a stream

Chunks add up: adding one chunk to the next gives a chunk that holds the text so far, and the sum of all of them is the complete message, with the token counts the provider sent at the end. That is how a program shows text as it arrives and still stores the whole reply.

ExampleAPI keyRun on Groq
from langchain.chat_models import init_chat_model

model = init_chat_model("groq:openai/gpt-oss-120b", temperature=0)

full = None
for chunk in model.stream("Say hi in three words."):
    full = chunk if full is None else full + chunk

print(type(full).__name__)
print(full.text)
print(full.usage_metadata)

Adding chunks gave an AIMessageChunk whose text is the whole reply, Hey there, friend!, and whose usage_metadata holds the token counts for the call: 77 in and 89 out, of which 75 were reasoning. A program can print chunks as they arrive and still keep one message with the complete text and the counts.

Sending several questions with batch

Batch and max_concurrency · from the Complete Agentic AI Course video · 56:15 to 57:15

The video's second call is batch. It takes a list of inputs, here three questions, sends them to the model in parallel and returns a list with one reply for each, in the same order. A config argument with max_concurrency sets how many calls run at the same time: with ten questions and a limit of five, the model gets five and then five.

python
replies = model.batch(questions)
replies = model.batch(questions, config={"max_concurrency": 2})   # at most two calls at a time

A real model call takes a different time on every run, and a rate limit can make it longer, so the first example uses a runnable that sleeps for one second in place of a model call. Every runnable has batch, and the timing shows what it does.

ExamplePure Python, Python 3.12
import time

from langchain_core.runnables import RunnableLambda

def slow_square(x):
    time.sleep(1)          # one second, standing in for a model call
    return x * x

step = RunnableLambda(slow_square)

start = time.time()
print([step.invoke(n) for n in [1, 2, 3]], round(time.time() - start), "s one by one")

start = time.time()
print(step.batch([1, 2, 3]), round(time.time() - start), "s with batch")

start = time.time()
print(step.batch([1, 2, 3], config={"max_concurrency": 2}), round(time.time() - start), "s with max_concurrency=2")

Three one-second calls took 3 s one after another and 1 s with batch, because batch ran them at the same time. With max_concurrency set to 2, the third call had to wait for a free slot, and the batch took 2 s.

ExampleAPI keyRun on Groq
from langchain.chat_models import init_chat_model

model = init_chat_model("groq:openai/gpt-oss-120b", temperature=0, reasoning_effort="low")
questions = [
    "Why do parrots have colorful feathers? One sentence.",
    "How do airplanes fly? One sentence.",
    "What is quantum computing? One sentence.",
]

replies = model.batch(questions)
for question, reply in zip(questions, replies):
    print(question)
    print("  ->", reply.text)

batch returned a list with one reply per question, in the order of the questions, so zip pairs each question with its answer.

invoke vs stream vs batch

invokestreambatch
InputOne promptOne promptA list of prompts
ReturnsOne message, when completeChunks, as they are generatedA list of messages, one per prompt
WaitingFor the whole replyFor the first chunk onlyFor the slowest reply
Use it forShort answers and steps inside a programChat screens and long repliesMany independent questions

Where you use streaming and batch

  • A chat screen: show text as it arrives, so the user is not looking at a blank box.
  • A review of many items: classify or summarise a list of tickets with one batch call.
  • A test run: send a set of questions through a prompt and compare the replies.
Watch out. Providers limit how many requests and tokens you may use per minute, and batch sends its requests at the same moment, so a long list can reach the limit and fail with a rate-limit error. Set max_concurrency to a small number, as in the video, and keep batches short on a free key. Calls that depend on each other, such as a second question that uses the first answer, cannot be batched.
Try it yourself
  • Change end="|" to end="" in the stream loop and read the paragraph as it prints.
  • Add config={"max_concurrency": 1} to the batch call and compare the time with the plain batch.
  • Add a fourth question to the batch and print the replies in order.

Slow is fine. Stopping is the only problem.