Rate limits, retries and giving up

A real endpoint answers 429 when you send too fast. PyRIT retries a rate-limited call on its own, and when the retries run out the attack fails cleanly rather than crashing the campaign.

A target that rate-limits

class Busy(ShopAssistant):
    """Refuses the first `failures` calls with a rate-limit error, like a real endpoint."""

    def __init__(self, *, failures=2, custom_configuration=None):
        super().__init__(custom_configuration=custom_configuration)
        self.failures = failures
        self.calls = 0

    @pyrit_target_retry
    async def _send_prompt_to_target_async(self, *, normalized_conversation):
        self.calls += 1
        if self.calls <= self.failures:
            raise RateLimitException()
        return await super()._send_prompt_to_target_async(
            normalized_conversation=normalized_conversation)

pyrit_target_retry is the decorator PyRIT puts on its own hosted targets. It catches a RateLimitException and retries with a growing wait, ten times by default. Wrapping the stand-in's send method in it makes a Python object behave like a throttled endpoint, so the behaviour can be seen with no network.

Recovering

import os
os.environ.update(RETRY_WAIT_MIN_SECONDS="0", RETRY_WAIT_MAX_SECONDS="1")
db = await arena()
target = Busy(failures=2)
result = await PromptSendingAttack(objective_target=target).execute_async(
    objective="How do I return a jacket?")
print("calls:", target.calls, "| retries:", result.total_retries)
print(result.last_response.converted_value)

Two calls failed and the third succeeded, so the reader saw one answer and a count of two retries. result.total_retries records them, and result.retry_events holds one entry each with the exception and the wait. The env vars shorten the wait so the lesson runs fast; the defaults start at five seconds.

Giving up, without losing the campaign

import os
os.environ.update(RETRY_WAIT_MIN_SECONDS="0", RETRY_WAIT_MAX_SECONDS="1", RETRY_MAX_NUM_ATTEMPTS="3")
db = await arena()
attack = PromptSendingAttack(objective_target=Busy(failures=99))
results = await AttackExecutor().execute_attack_async(
    attack=attack, objectives=["a", "b"], return_partial_on_failure=True)
print("completed:", len(results.completed_results))
print("incomplete:", [objective for objective, _ in results.incomplete_objectives])
print("errors:", [type(exc).__name__ for exc in results.exceptions])

When every retry is exhausted, a single execute_async raises. Under AttackExecutor with return_partial_on_failure=True, the failing objectives, each with its exception, land in incomplete_objectives (and the exceptions alone in exceptions), while the rest of the campaign finishes. raise_if_incomplete turns that back into an error when you would rather stop. This is the lesson 25 result put to work.