Random search: several demo sets, keep the best

BootstrapFewShotWithRandomSearch builds several candidate demo sets, scores each on a validation set, and returns the best. It trades more model calls for a better pick.

The signature and metric move into triage.py, so a script can import them:

import dspy

from shop_lm import ShopLM
from tickets import devset, trainset
from triage import Triage, exact

dspy.configure(lm=ShopLM(model="shop/keywords"))

optimizer = dspy.BootstrapFewShotWithRandomSearch(
    metric=exact, max_bootstrapped_demos=4, max_labeled_demos=4, num_candidate_programs=4, num_threads=1,
)
best = optimizer.compile(dspy.Predict(Triage), trainset=trainset)

print("candidates, scored on trainset:")
for candidate in best.candidate_programs:
    print(f"  seed {candidate['seed']:2}  score {candidate['score']}")

evaluate = dspy.Evaluate(devset=devset, metric=exact, num_threads=1, display_progress=False)
print("best on devset:", evaluate(best).score)

python optimize.py

The optimizer prints as it goes, and the lines starting with a date are DSPy's log, on stderr. num_candidate_programs=4 adds four shuffled candidates to three that are always there, seven in all. The three fixed ones: seed -3 is the program with no demos, seed -2 is LabeledFewShot, seed -1 is BootstrapFewShot on the training set in order. The rest shuffle the training set and bootstrap again. Each candidate was scored on trainset, because no valset was passed.

The winner scored 83.33 on the training tickets and 75.0 on devset. The gap illustrates lesson 12's note: a candidate chosen because it scores best on some tickets looks better on those tickets than on new ones. Pass valset= with examples the demos were not drawn from to choose more fairly, and keep devset for the final number.

Cost

Every candidate is a full evaluation. With seven candidates and 12 training tickets that is dozens of calls to the stand-in, which cost nothing; with a hosted model and hundreds of examples, the default num_candidate_programs=16 can cost real money. Start small.