invoke, batch and stream

A chat model can be called three ways: invoke for one reply, batch for many independent requests at once, and stream for a reply that arrives in pieces.

Lesson 3's ShopModel supplied only _generate. All three calling methods come from BaseChatModel and work on it unchanged.

What invoke returns

from shop_model import ShopModel

reply = ShopModel().invoke("Where is my order A17?")

print(reply.type)
print(reply.text)
print(reply.usage_metadata)

The reply is an AIMessage. usage_metadata holds token counts when a provider reports them; your model counts no tokens, so it is None. A hosted model fills it in, which is how you track what a conversation costs.

Many requests at once

from shop_model import ShopModel

replies = ShopModel().batch(["Hello", "Where is B22?", "Is C40 in stock?"])

for reply in replies:
    print(reply.text)

batch takes a list of inputs and returns the replies in the same order. The calls run in parallel on your side, which saves time with a hosted model. This is separate from the batch APIs some providers sell at a discount.

A reply in pieces

from shop_model import ShopModel

for chunk in ShopModel().stream("Hello"):
    print(type(chunk).__name__, repr(chunk.text))

A hosted model streams many small AIMessageChunk pieces, which is what lets a chat window show text as it is written. Your model supplies only _generate, so stream falls back to calling it once and yields the finished AIMessage as a single piece. Code that loops over a stream works with both.