Testing an agent

An agent's tools, routing and refusals can be tested without any model: a scripted fake plays the model's part, so every test runs the same way each time.

A test should check your code, not a model's mood. LangChain's documentation recommends GenericFakeChatModel, which returns the replies you give it, one per call, including tool calls.

Example

from langchain.agents import create_agent
from langchain_core.language_models.fake_chat_models import GenericFakeChatModel
from tools import lookup_order

model = GenericFakeChatModel(messages=iter(["done"]))
agent = create_agent(model, tools=[lookup_order])
agent.invoke({"messages": [{"role": "user", "content": "B22?"}]})

It cannot be given tools: its bind_tools raises NotImplementedError, so an agent with any tool fails on its first call. The documentation's example passes tools=[], which is why it works there. Three lines fix it.

Examplescripted.py

from langchain_core.language_models.fake_chat_models import GenericFakeChatModel


class ScriptedModel(GenericFakeChatModel):
    def bind_tools(self, tools, **kwargs):
        return self

Example

from langchain.messages import AIMessage, ToolCall
from scripted import ScriptedModel

call = ToolCall(name="lookup_order", args={"order_id": "B22"}, id="call_1")
model = ScriptedModel(messages=iter([AIMessage("", tool_calls=[call]), "done"]))
result = create_agent(model, tools=[lookup_order]).invoke({"messages": [{"role": "user", "content": "B22?"}]})

for message in result["messages"]:
    print(f"{message.type:<5} {message.text or message.tool_calls[0]['args']}")

The model's two replies were fixed in advance, so this run is the same every time. The tool itself ran, and its result is what the test checks.

Tests with pytest

Exampletest_desk.py

from langchain.agents import create_agent
from langchain.messages import AIMessage, ToolCall
from router import answer
from scripted import ScriptedModel
from tools import lookup_order


def test_unknown_order_is_reported():
    call = ToolCall(name="lookup_order", args={"order_id": "B22"}, id="call_1")
    model = ScriptedModel(messages=iter([AIMessage("", tool_calls=[call]), "done"]))
    agent = create_agent(model, tools=[lookup_order])
    result = agent.invoke({"messages": [{"role": "user", "content": "Where is B22?"}]})
    assert result["messages"][2].text == "B22 is not an order we have."


def test_order_questions_go_to_the_orders_agent():
    assert answer("Where is A17?") == ("orders", "A17 shipped on 3 March.")


def test_uncovered_questions_are_refused():
    name, reply = answer("Can I pay with bitcoin?")
    assert name == "policies" and reply.startswith("Our policies do not cover that")

The first test uses the scripted model to check the tool's answer for an unknown order. The other two check lesson 31's router and lesson 28's refusal with the stand-in models, which decide the same way on every run. None of them needs a key or a network, so they can run on every change.

Example

pytest -q -p no:warnings test_desk.py

Try it yourself

Change the expected text in the first test and read how pytest reports the failure.
Add a test that answer("Is shipping free?") goes to the policies agent.
Script a model that asks for lookup_order twice and assert there are two tool messages.

PreviousRouting between specialists

You understood something today that you didn't yesterday.