Keeping the conversation short

Every message in a thread goes to the model on every call, so a long conversation costs more each turn. SummarizationMiddleware replaces older messages with a summary.

In lesson 13 each question added four messages to Ravi's thread, and all of them were sent with the next question. Summarizing needs a model, so this one lists the orders it finds in the text it is given.

Examplesummary_model.py

import re

from langchain.chat_models import BaseChatModel
from langchain.messages import AIMessage
from langchain_core.outputs import ChatGeneration, ChatResult


class SummaryModel(BaseChatModel):
    @property
    def _llm_type(self):
        return "summary"

    def _generate(self, messages, stop=None, run_manager=None, **kwargs):
        found = re.findall(r"\b([A-Z]\d+) (shipped on \d+ \w+|waiting for stock)", messages[-1].text)
        summary = "; ".join(f"{order} {status}" for order, status in dict(found).items())
        message = AIMessage(f"Orders discussed: {summary}.")
        return ChatResult(generations=[ChatGeneration(message=message)])

Exampleagent.py

from langchain.agents import create_agent
from langchain.agents.middleware import SummarizationMiddleware
from langgraph.checkpoint.memory import InMemorySaver
from shop_model import ShopModel
from summary_model import SummaryModel
from tools import lookup_order

summarize = SummarizationMiddleware(SummaryModel(), trigger=("messages", 6), keep=("messages", 2))
agent = create_agent(ShopModel(), tools=[lookup_order], middleware=[summarize],
                     checkpointer=InMemorySaver())

Before each model call, the middleware counts the messages. Once there are at least six, the trigger, it asks SummaryModel to summarize all but the last two, the keep, and puts the summary in their place.

Example

thread = {"configurable": {"thread_id": "ravi-1"}}
for text in ["Where is A17?", "And C40?", "And B22?"]:
    result = agent.invoke({"messages": [{"role": "user", "content": text}]}, thread)
    print(text, "->", len(result["messages"]), "messages")

The thread stays at four messages instead of growing to twelve. When the cut would separate a tool call from its result, the middleware moves it so the two stay together.

Example

thread = {"configurable": {"thread_id": "ravi-1"}}
for text in ["Where is A17?", "And C40?", "And B22?"]:
    result = agent.invoke({"messages": [{"role": "user", "content": text}]}, thread)
    print(text, "->", len(result["messages"]), "messages")

print(result["messages"][0].text)

The first message is now a human message holding the summary, which covers A17 and C40. The newest exchange about B22 is kept as it was.

Without a trigger

Exampleagent.py, the same agent with no trigger

summarize = SummarizationMiddleware(SummaryModel())

Example

thread = {"configurable": {"thread_id": "ravi-1"}}
for text in ["Where is A17?", "And C40?", "And B22?"]:
    result = agent.invoke({"messages": [{"role": "user", "content": text}]}, thread)
    print(text, "->", len(result["messages"]), "messages")

With no trigger, the middleware never summarizes, and the thread grows by four each time. A trigger can also be a token count, ('tokens', 4000); a dictionary of conditions must all hold, and a list fires when any one does.

Try it yourself

Set keep=("messages", 4) and check what the summary covers.
Use trigger=("messages", 10) and ask five questions.
Ask about B22 first and read the summary: why is B22 missing from it?

PreviousRetries and a fallback model

You understood something today that you didn't yesterday.