Verifiers and stop conditions
A verifier is the part of a loop that checks each result and returns pass or fail with feedback, and a stop condition is the rule that ends the loop when pass never comes.
Last updated: 08 Oct, 2026 · LangChain 1.4
Loop engineering showed a loop that checks a reply with a few lines of Python and tries again. Those few lines were a verifier, and the three attempts were a stop condition. This lesson takes the two parts one at a time, because a loop with a weak verifier returns wrong answers with confidence, and a loop with no stop condition never returns.
The verifier and the interruption rule
The video opens the documentation of a coding agent to show its loop: gather context, take action, verify. When the verify step finds a bug, the feedback goes to the agent, the agent acts again, and the check runs again. The loop runs until the output is good.
A loop in Python needs a termination condition, and so does this one. Without one, an agent given a prompt would run for ever. The video names three ways to interrupt it: a human in the loop, a condition, or a maximum number of iterations. That gives an agentic loop two main components: the verifier, which decides whether a result is good, and the interruption rule, which ends the loop. A human is one option for the interruption rule and not a requirement, since a condition or an iteration limit works on its own.
Choosing a verifier
The video's example is a blog-writing pipeline: one agent researches the topic, another writes, and a reviewer checks the blog. The reviewer is the verifier. If the blog is fine it goes to the final response, and otherwise the feedback goes back to the writer. The video adds that one agent or many is a design choice.
There are three common kinds of verifier, from cheapest to most flexible:
- A check in code that runs the result. Run the tests, or call the function and compare its numbers with the expected ones. It is exact, free and repeatable, and it only covers what a program can measure.
- A rule check in Python. Look for required words, a format or a length in the text. It is the same idea as
checkin the previous lesson. - A model as judge. A second model reads the result against written rules and returns a verdict and feedback. It can judge tone and clarity, and it costs a call. The judge should not be the model that wrote the draft, because a model reviewing its own text shares its own blind spots.
A check in code is the easiest to trust, so the next example uses one. A model writes a function, apply_coupon, and the verifier does not read it: it runs it on three cases and compares the results.
Loading the model's function
CASES = [((100, "SAVE20"), 80.0), ((49.99, "SAVE20"), 39.99), ((100, "BOGUS"), 100)]
def load(source):
namespace = {}
exec(source, namespace) # runs the model's code: use a sandbox for this in a real system
return namespace["apply_coupon"]Comparing results with the expected ones
verify returns a list of problems in the same form as before: empty means pass. Each problem says what the call returned and what it should have returned, which is the feedback a model needs.
def verify(source):
try:
apply_coupon = load(source)
except Exception as error:
return [f"apply_coupon is not defined: {error}"]
problems = []
for args, expected in CASES:
got = apply_coupon(*args)
if got != expected:
problems.append(f"apply_coupon{args} returned {got}, expected {expected}")
return problemsRunning the verifier on a bug and on the model's code
First a function with a 2% discount in place of 20%, written by hand, to show what a failing verdict looks like. Then the function the model writes.
from langchain.chat_models import init_chat_model
buggy = "def apply_coupon(price, code):\n return round(price * 0.98, 2) if code == 'SAVE20' else price\n"
print("a 2% discount:", verify(buggy))
llm = init_chat_model("groq:openai/gpt-oss-20b", temperature=0)
reply = llm.invoke("Write a Python function apply_coupon(price, code). The code SAVE20 takes 20% off. "
"Any other code changes nothing. Round the result to 2 decimals. Return only the code, no fences.")
print(reply.content)
print("the model's code:", verify(reply.content) or "passed")a 2% discount: ["apply_coupon(100, 'SAVE20') returned 98.0, expected 80.0", "apply_coupon(49.99, 'SAVE20') returned 48.99, expected 39.99"]
def apply_coupon(price, code):
if code == "SAVE20":
discounted = price * 0.8
else:
discounted = price
return round(discounted, 2)
the model's code: passedWhat the verdicts say
- The 2% function fails two of three cases. For a price of 100 it returned 98.0 where 80.0 was expected, and for 49.99 it returned 48.99 where 39.99 was expected. The unknown-code case passes for it, so it does not appear in the list.
- The model's function passes all three. The verifier never read the code. It ran the function and compared numbers, which cannot be argued with, and that is why a check in code is the easiest verifier to trust.
Stopping a loop
The video's words are the rule to keep: a loop must have a stopping rule, or the agent keeps spending tokens and time without reaching a better result. Most loops stop for one of four reasons, checked in this order after every attempt.
def run_loop(task, check, max_attempts=4, token_budget=2500):
... # ask the model, add up the tokens, call check(draft)
# 1. no problems -> stop: passed
# 2. tokens >= budget -> stop: token budget used up
# 3. same problems twice -> stop: no progress
# 4. attempts used up -> stop: max attempts reachedThe token budget guards cost, the same-problems check guards against a model that keeps making the same mistake, and the attempt limit guards against everything else. The loop below runs three times: with a check an answer can pass, with a check that no answer can pass because it asks for five words in four, and with that same impossible check and a small token budget.
import re
from langchain.chat_models import init_chat_model
llm = init_chat_model("groq:openai/gpt-oss-20b", temperature=0)
def run_loop(task, check, max_attempts=4, token_budget=2500):
messages, tokens, last_problems = [("user", task)], 0, None
for attempt in range(1, max_attempts + 1):
reply = llm.invoke(messages)
draft, tokens = reply.content, tokens + reply.usage_metadata["total_tokens"]
problems = check(draft)
print(f" attempt {attempt}: {problems if problems else 'passed'} (tokens so far {tokens})")
if not problems:
return draft, True, "passed"
if tokens >= token_budget:
return draft, False, "token budget used up"
if problems == last_problems:
return draft, False, "no progress: same problems twice"
last_problems = problems
messages += [("assistant", draft), ("user", "Fix these and return only the new text: " + "; ".join(problems))]
return draft, False, "max attempts reached"
def possible(text):
ok = len(text.split()) <= 12 and "ceramic" in text.lower()
return [] if ok else ["use at most 12 words and include 'ceramic'"]
def impossible(text):
words = re.findall(r"[a-z]+", text.lower())
problems = [f"include the word {w}" for w in ["ceramic", "dishwasher", "microwave", "handle", "lid"] if w not in words]
if len(words) > 4:
problems.append("use at most 4 words")
return problems
TASK = "Write a one-line product description for a coffee mug."
for name, check, options in [("a check that can pass", possible, {}),
("a check no answer can pass", impossible, {}),
("the same check with a small budget", impossible, {"token_budget": 600})]:
print(name)
draft, passed, why = run_loop(TASK, check, **options)
print(f" stopped: {why}; passed={passed}; last draft: {draft!r}")a check that can pass attempt 1: ["use at most 12 words and include 'ceramic'"] (tokens so far 194) attempt 2: passed (tokens so far 517) stopped: passed; passed=True; last draft: 'Enjoy every sip with our ceramic mug, crafted for comfort and style.' a check no answer can pass attempt 1: ['include the word dishwasher', 'include the word microwave', 'include the word handle', 'include the word lid', 'use at most 4 words'] (tokens so far 194) attempt 2: ['include the word ceramic'] (tokens so far 671) attempt 3: ['use at most 4 words'] (tokens so far 1519) attempt 4: ['use at most 4 words'] (tokens so far 2389) stopped: no progress: same problems twice; passed=False; last draft: 'dishwasher-microwave-ceramic-handle-lid' the same check with a small budget attempt 1: ['include the word dishwasher', 'include the word microwave', 'include the word handle', 'include the word lid', 'use at most 4 words'] (tokens so far 237) attempt 2: ['include the word ceramic'] (tokens so far 621) stopped: token budget used up; passed=False; last draft: 'Dishwasher microwave handle lid'
Why each loop stopped
- A check that can pass. Attempt 1 failed it, the problem went back, and attempt 2 passed after 517 tokens in total, so the loop stopped with
passed=True. - A check no answer can pass. Four words cannot hold five required words, so the verifier kept failing. Attempts 3 and 4 returned the same problem, "use at most 4 words", and the loop stopped on "no progress: same problems twice" at 2389 tokens, returning the last draft with
passed=False. Attempt 4 was also the last one allowed, and the no-progress rule is checked first. - The same check with a budget of 600 tokens. The loop stopped after attempt 2, when 621 tokens had been used, with the reason "token budget used up". The budget ended the loop before a third attempt was paid for.
Stop condition vs what it protects
| Stop condition | What it protects against | What the loop returns |
|---|---|---|
| Check passed | Nothing: the goal is met | The verified result |
| Maximum attempts | A loop that never ends | The last draft, marked not passed |
| Token or cost budget | A bill that grows with every retry | The last draft, marked not passed |
| No progress | A model repeating the same mistake | The last draft, marked not passed |
| Human approval | A risky action running without a person | Nothing until the person decides |
Where you use verifiers and stop conditions
- Code agents. The tests are the verifier, and the maximum number of repair attempts is the stop condition.
- Data pipelines. A schema check or a total that must match another system verifies each batch.
- Writing agents. A word count in code plus a judge model for tone, with a limit on revisions.
passed=False and the reason, and never show it as checked. Also, exec runs whatever code the model writes with your permissions: run model-written code in a sandbox, never in the process that holds your keys.Related
- Previous: Loop engineering
- Next: Anatomy of a good loop
- See also: Human in the loop
- Add a fourth case to
CASES,((200, "SAVE20"), 160.0), and confirm the verdict does not change. - Change the 2% bug to
price * 0.8and read howverifyreports it. - Raise
token_budgetin the third run to4000and see which stop reason replaces the budget.
You understood something today that you didn't yesterday.