Circuit breakers and an error budget

A circuit breaker wraps a tool call, counts its failures, and after a threshold stops calling a failing dependency for a while, so one broken service does not have the agent retry it on every message.

Last updated: 14 Oct, 2026 · Agent Governance Toolkit 4.1.0

The Sandbox policy, without running a sandbox lesson limited how an action runs. Reliability is the next concern: what happens when a tool keeps failing. The toolkit ships a circuit breaker in agent_os.circuit_breaker and an error budget in agent_sre, and both have a signature or two that catch you out, so every one here was checked against the install before it was written.

Describing when to trip

Syntax: CircuitBreakerConfig sets the number of failures that opens the breaker and the delay before it tries again. The recovery timeout is kept small here so the example can show the breaker healing without a long wait.

from agent_os.circuit_breaker import CircuitBreaker, CircuitBreakerConfig, CircuitOpenError

# open after 3 failures; retry after the recovery timeout (kept small for the demo)
config = CircuitBreakerConfig(failure_threshold=3, recovery_timeout_seconds=0.3)

The retry delay's name is not the obvious one

The first trap is the parameter name. The obvious recovery_timeout does not exist and the constructor rejects it; the real name is recovery_timeout_seconds. Guessing the short name raises a TypeError rather than being quietly ignored.

from agent_os.circuit_breaker import CircuitBreakerConfig

# the obvious name does not exist; the real one is recovery_timeout_seconds
CircuitBreakerConfig(failure_threshold=3, recovery_timeout=5)

Traceback (most recent call last):
  File "main.py", line 4, in <module>
    CircuitBreakerConfig(failure_threshold=3, recovery_timeout=5)
TypeError: CircuitBreakerConfig.__init__() got an unexpected keyword argument 'recovery_timeout'

With the right name the config builds and holds the two values, ready to hand to a breaker.

config = CircuitBreakerConfig(failure_threshold=3, recovery_timeout_seconds=0.3)
print("threshold      :", config.failure_threshold)
print("recovery seconds:", config.recovery_timeout_seconds)

threshold      : 3
recovery seconds: 0.3

A breaker around the shop agent's tool call

Wrap the tool in a breaker and call it through breaker.call. While a dependency is failing, each real failure counts; on the third the breaker opens, and from then on it refuses the call without touching the tool at all.

breaker = CircuitBreaker("order_status", config=config)
calls = {"n": 0}

def order_status():                 # the real tool, currently failing
    calls["n"] += 1
    raise RuntimeError("downstream down")

for _ in range(3):                  # three real failures trip the breaker
    try: breaker.call(order_status)
    except RuntimeError: pass
print("state after 3 failures:", breaker.get_state().value)

# the breaker is open, so the next call is refused without calling the tool
try: breaker.call(order_status)
except CircuitOpenError: print("open: tool skipped, calls still", calls["n"])

state after 3 failures: OPEN
open: tool skipped, calls still 3

Recovering after the timeout

Once the recovery timeout passes, the breaker lets one trial call through. If the dependency is healthy again, that call succeeds and the breaker closes; the agent is back to normal with no manual reset.

import time

time.sleep(0.35)                    # wait past the recovery timeout

def order_status():                 # the dependency is healthy again
    return "A-2291: out for delivery"

print("recovered:", breaker.call(order_status))
print("state:", breaker.get_state().value)

recovered: A-2291: out for delivery
state: CLOSED

Tracking an error budget

An ErrorBudget counts how much failure a target can absorb before you must act. The trap is the method name: there is no .consume a reader might reach for. You record each event as good or bad with record_event, and read consumed, remaining and is_exhausted. A burn_rate method is there too, for how fast the budget is being spent.

from agent_sre import ErrorBudget

# there is no .consume; you record each event as good or bad
print("has consume:", hasattr(ErrorBudget, "consume"))

budget = ErrorBudget(total=20)              # 20 failures allowed in the window
for good in [True, True, False, False, False]:
    budget.record_event(good=good)

print("consumed :", budget.consumed)
print("remaining:", budget.remaining)
print("exhausted:", budget.is_exhausted)

has consume: False
consumed : 3.0
remaining: 0.85
exhausted: False

An SLI is a class you subclass

The last trap is SLI, the service-level indicator that feeds a budget. It is abstract: it declares a collect method with no body, so calling SLI() raises rather than giving you an object. You subclass it with your own collect, or use one of the concrete indicators the toolkit ships.

from agent_sre import SLI

# SLI declares an abstract collect(); it cannot be built directly
SLI()

Traceback (most recent call last):
  File "main.py", line 4, in <module>
    SLI()
TypeError: Can't instantiate abstract class SLI without an implementation for abstract method 'collect'

What the breaker and the budget gave you

• The breaker opened on the third failure and then refused the call without touching the tool, so a dead dependency is not hammered on every message.
• It healed itself after the recovery timeout, letting one trial call through and closing on success.
• The budget counts, it does not block. record_event tallies good and bad calls so you can alert before the budget is spent; enforcing a limit is still the gate's job.

The three names that trip you up

Where you use a circuit breaker

• A tool that calls a flaky external service, so a run of failures opens the breaker instead of retrying forever.
• An agent in a loop, where one broken tool should not stall every turn while it times out.
• A reliability target, where the error budget tells you how much failure is left before the target is missed.

Related

• Previous: Sandbox policy, without running a sandbox
• Next: The governed support agent
• See also: Two ways the engine checks a call