BestOfN: try again until a check passes

BestOfN runs a module several times with different rollout ids and keeps the first answer whose reward reaches your threshold, or else the best one.

A reward function gets the inputs and a prediction and returns a number. This one rewards billing, which you know is right for money-back tickets:

def is_billing(args, prediction):
    print("  try:", prediction.category)
    return 1.0 if prediction.category == "billing" else 0.0

best = dspy.BestOfN(module=dspy.Predict(Triage), N=3, reward_fn=is_billing, threshold=1.0)
print(best(ticket="I want my money back").category)

The first try guessed account and scored 0. BestOfN ran the module again with the next rollout id and temperature=1.0, the second try scored 1.0, and it stopped there. For a real model the rollout id and temperature give a fresh sample; the stand-in reads the rollout id and guesses differently, which is the part of shop_lm.py that uses attempt.

print(best(ticket="My parcel never arrived").category)

Here the keyword rule answered shipping every time, so all three tries scored 0, and BestOfN returned the best of them. Retrying only helps when the model's answers vary.

Rewards are not labels

In production you do not know the right category, so a reward checks something you can check: the reply is under 300 characters, the JSON has an order id, the answer cites a document. Every try is another paid call.

Refine

dspy.Refine takes the same arguments. After a try that misses the threshold, it asks the model a second question: given the program's code, its inputs and outputs, and the reward function's code, what should each step do differently? The answer goes into the next try as an extra hint_ input. Writing useful advice needs a real model, so Refine is not run here.