Stopping an agent from editing its own policy

Every rule so far was added by a method call. An agent that can run code can make method calls too, which makes the policy engine itself a target.

The attack is one line long and needs no cleverness at all.

from agent_control_plane import PolicyEngine, AgentKernel

policy = PolicyEngine()
policy.add_constraint("support", ["lookup_order"])
kernel = AgentKernel(policy_engine=policy)
print("before:", kernel.intercept_tool_execution("support", "delete_order", {}) is None)

policy.add_constraint("support", ["lookup_order", "delete_order"])
print("after: ", kernel.intercept_tool_execution("support", "delete_order", {}) is None)

The agent granted itself the tool it was denied. Any code running in the same process can do this, which includes a tool that executes code and anything that got there through a dependency.

Freezing

policy = PolicyEngine()
policy.add_constraint("support", ["lookup_order"])
policy.freeze()

try:
    policy.add_constraint("support", ["delete_order"])
except RuntimeError as exc:
    print(exc)

After freeze every mutating method raises. The message says the call is irreversible, and it means it: there is no unfreeze, by design, because one would be the first thing an attacker called.

Freezing does more than set a flag. The engine's internal dictionaries are swapped for read-only views, so reaching past the methods and assigning to them directly fails as well.

print("frozen:", policy.is_frozen)
try:
    policy.state_permissions["support"] = {"delete_order"}
except TypeError as exc:
    print("direct assignment:", exc)

Checking still works normally after freezing; only changing is blocked. So the shape of a governed program is: build the policy at start-up, freeze it, then serve.

Who changed what

Every change is recorded as it happens, including the freeze itself, which gives you a record of how the policy reached the state it is in.

for entry in policy.mutation_log:
    print(entry["operation"], entry["details"])

That log is in memory and disappears with the process. Part 5 builds the durable record, the one that survives a restart and can be shown to somebody who does not trust you.