Generative AI and Agentic AILangChain 1.4 · LangGraph 1.2 · Python 3.12 or 3.13
Dashboard
0%

Verifiers and stop conditions

A verifier is the part of a loop that checks each result and returns pass or fail with feedback, and a stop condition is the rule that ends the loop when pass never comes.

Last updated: 08 Oct, 2026 · LangChain 1.4

Loop engineering showed a loop that checks a reply with a few lines of Python and tries again. Those few lines were a verifier, and the three attempts were a stop condition. This lesson takes the two parts one at a time, because a loop with a weak verifier returns wrong answers with confidence, and a loop with no stop condition never returns.

The verifier and the interruption rule · from the Loop and Harness Engineering Crash Course video · 26:52 to 29:42

The verifier and the interruption rule

The video opens the documentation of a coding agent to show its loop: gather context, take action, verify. When the verify step finds a bug, the feedback goes to the agent, the agent acts again, and the check runs again. The loop runs until the output is good.

A loop in Python needs a termination condition, and so does this one. Without one, an agent given a prompt would run for ever. The video names three ways to interrupt it: a human in the loop, a condition, or a maximum number of iterations. That gives an agentic loop two main components: the verifier, which decides whether a result is good, and the interruption rule, which ends the loop. A human is one option for the interruption rule and not a requirement, since a condition or an iteration limit works on its own.

An agent acts and a verifier checks. On pass the answer is returned. On fail a stop rule asks whether attempts and budget are left: if yes the feedback goes back to the agent, if no the loop stops and returns the last draft marked as not passed.

Choosing a verifier

A reviewer agent as the verifier · from the Loop and Harness Engineering Crash Course video · 33:49 to 34:58

The video's example is a blog-writing pipeline: one agent researches the topic, another writes, and a reviewer checks the blog. The reviewer is the verifier. If the blog is fine it goes to the final response, and otherwise the feedback goes back to the writer. The video adds that one agent or many is a design choice.

There are three common kinds of verifier, from cheapest to most flexible:

  • A check in code that runs the result. Run the tests, or call the function and compare its numbers with the expected ones. It is exact, free and repeatable, and it only covers what a program can measure.
  • A rule check in Python. Look for required words, a format or a length in the text. It is the same idea as check in the previous lesson.
  • A model as judge. A second model reads the result against written rules and returns a verdict and feedback. It can judge tone and clarity, and it costs a call. The judge should not be the model that wrote the draft, because a model reviewing its own text shares its own blind spots.

A check in code is the easiest to trust, so the next example uses one. A model writes a function, apply_coupon, and the verifier does not read it: it runs it on three cases and compares the results.

Loading the model's function

python
CASES = [((100, "SAVE20"), 80.0), ((49.99, "SAVE20"), 39.99), ((100, "BOGUS"), 100)]

def load(source):
    namespace = {}
    exec(source, namespace)          # runs the model's code: use a sandbox for this in a real system
    return namespace["apply_coupon"]

Comparing results with the expected ones

verify returns a list of problems in the same form as before: empty means pass. Each problem says what the call returned and what it should have returned, which is the feedback a model needs.

python
def verify(source):
    try:
        apply_coupon = load(source)
    except Exception as error:
        return [f"apply_coupon is not defined: {error}"]
    problems = []
    for args, expected in CASES:
        got = apply_coupon(*args)
        if got != expected:
            problems.append(f"apply_coupon{args} returned {got}, expected {expected}")
    return problems

Running the verifier on a bug and on the model's code

First a function with a 2% discount in place of 20%, written by hand, to show what a failing verdict looks like. Then the function the model writes.

ExampleAPI keyLive run on Groq, openai/gpt-oss-20b
from langchain.chat_models import init_chat_model

buggy = "def apply_coupon(price, code):\n    return round(price * 0.98, 2) if code == 'SAVE20' else price\n"
print("a 2% discount:", verify(buggy))

llm = init_chat_model("groq:openai/gpt-oss-20b", temperature=0)
reply = llm.invoke("Write a Python function apply_coupon(price, code). The code SAVE20 takes 20% off. "
                   "Any other code changes nothing. Round the result to 2 decimals. Return only the code, no fences.")
print(reply.content)
print("the model's code:", verify(reply.content) or "passed")

What the verdicts say

  • The 2% function fails two of three cases. For a price of 100 it returned 98.0 where 80.0 was expected, and for 49.99 it returned 48.99 where 39.99 was expected. The unknown-code case passes for it, so it does not appear in the list.
  • The model's function passes all three. The verifier never read the code. It ran the function and compared numbers, which cannot be argued with, and that is why a check in code is the easiest verifier to trust.

Stopping a loop

Why a loop needs a stopping rule · from the Loop and Harness Engineering Crash Course video · 45:12 to 46:21

The video's words are the rule to keep: a loop must have a stopping rule, or the agent keeps spending tokens and time without reaching a better result. Most loops stop for one of four reasons, checked in this order after every attempt.

python
def run_loop(task, check, max_attempts=4, token_budget=2500):
    ...                       # ask the model, add up the tokens, call check(draft)
    # 1. no problems          -> stop: passed
    # 2. tokens >= budget     -> stop: token budget used up
    # 3. same problems twice  -> stop: no progress
    # 4. attempts used up     -> stop: max attempts reached

The token budget guards cost, the same-problems check guards against a model that keeps making the same mistake, and the attempt limit guards against everything else. The loop below runs three times: with a check an answer can pass, with a check that no answer can pass because it asks for five words in four, and with that same impossible check and a small token budget.

ExampleAPI keyLive run on Groq, openai/gpt-oss-20b
import re
from langchain.chat_models import init_chat_model

llm = init_chat_model("groq:openai/gpt-oss-20b", temperature=0)

def run_loop(task, check, max_attempts=4, token_budget=2500):
    messages, tokens, last_problems = [("user", task)], 0, None
    for attempt in range(1, max_attempts + 1):
        reply = llm.invoke(messages)
        draft, tokens = reply.content, tokens + reply.usage_metadata["total_tokens"]
        problems = check(draft)
        print(f"  attempt {attempt}: {problems if problems else 'passed'} (tokens so far {tokens})")
        if not problems:
            return draft, True, "passed"
        if tokens >= token_budget:
            return draft, False, "token budget used up"
        if problems == last_problems:
            return draft, False, "no progress: same problems twice"
        last_problems = problems
        messages += [("assistant", draft), ("user", "Fix these and return only the new text: " + "; ".join(problems))]
    return draft, False, "max attempts reached"

def possible(text):
    ok = len(text.split()) <= 12 and "ceramic" in text.lower()
    return [] if ok else ["use at most 12 words and include 'ceramic'"]

def impossible(text):
    words = re.findall(r"[a-z]+", text.lower())
    problems = [f"include the word {w}" for w in ["ceramic", "dishwasher", "microwave", "handle", "lid"] if w not in words]
    if len(words) > 4:
        problems.append("use at most 4 words")
    return problems

TASK = "Write a one-line product description for a coffee mug."
for name, check, options in [("a check that can pass", possible, {}),
                             ("a check no answer can pass", impossible, {}),
                             ("the same check with a small budget", impossible, {"token_budget": 600})]:
    print(name)
    draft, passed, why = run_loop(TASK, check, **options)
    print(f"  stopped: {why}; passed={passed}; last draft: {draft!r}")

Why each loop stopped

  • A check that can pass. Attempt 1 failed it, the problem went back, and attempt 2 passed after 517 tokens in total, so the loop stopped with passed=True.
  • A check no answer can pass. Four words cannot hold five required words, so the verifier kept failing. Attempts 3 and 4 returned the same problem, "use at most 4 words", and the loop stopped on "no progress: same problems twice" at 2389 tokens, returning the last draft with passed=False. Attempt 4 was also the last one allowed, and the no-progress rule is checked first.
  • The same check with a budget of 600 tokens. The loop stopped after attempt 2, when 621 tokens had been used, with the reason "token budget used up". The budget ended the loop before a third attempt was paid for.

Stop condition vs what it protects

Stop conditionWhat it protects againstWhat the loop returns
Check passedNothing: the goal is metThe verified result
Maximum attemptsA loop that never endsThe last draft, marked not passed
Token or cost budgetA bill that grows with every retryThe last draft, marked not passed
No progressA model repeating the same mistakeThe last draft, marked not passed
Human approvalA risky action running without a personNothing until the person decides

Where you use verifiers and stop conditions

  • Code agents. The tests are the verifier, and the maximum number of repair attempts is the stop condition.
  • Data pipelines. A schema check or a total that must match another system verifies each batch.
  • Writing agents. A word count in code plus a judge model for tone, with a limit on revisions.
Watch out. A loop that ends on a stop rule has not produced a verified result. Return the last draft together with passed=False and the reason, and never show it as checked. Also, exec runs whatever code the model writes with your permissions: run model-written code in a sandbox, never in the process that holds your keys.
Try it yourself
  • Add a fourth case to CASES, ((200, "SAVE20"), 160.0), and confirm the verdict does not change.
  • Change the 2% bug to price * 0.8 and read how verify reports it.
  • Raise token_budget in the third run to 4000 and see which stop reason replaces the budget.

You understood something today that you didn't yesterday.