BootstrapFewShot: demos the program earned

BootstrapFewShot runs your program on training examples, keeps the runs your metric passes, and uses their inputs and outputs as demos, for every step of the program.

optimizer = dspy.BootstrapFewShot(metric=exact, max_bootstrapped_demos=4, max_labeled_demos=4)
compiled = optimizer.compile(dspy.Predict(Triage), trainset=trainset)
for demo in compiled.demos:
    print(demo.get("augmented", False), demo.category, "|", demo.ticket)
print(evaluate(compiled).score)

It ran a copy of your program, the teacher, on the training tickets in order, and kept each run the metric passed. augmented marks a demo made from a run, not copied from the data. The teacher had four labelled demos of its own, from max_labeled_demos, which is how it got the money-back ticket right. The first four tickets all passed, the limit was four, so it stopped. The score, 62.5, is lower than LabeledFewShot's 75.0 with the same number of demos.

Why it did worse here

The demos decide, and these are the first four training tickets, three of them easy keyword tickets. LabeledFewShot's random four happened to include a money ticket, a profile ticket and a box ticket, much closer to the devset's hard ones. Bootstrapping also only ever keeps runs that passed, so a ticket the teacher gets wrong can never become a demo. Where it shines:

• A multi-step program, like Desk, has no labels for its middle steps. Bootstrapping records what each step produced in a passing run, so every predictor gets demos.
• With ChainOfThought, the demos include the reasoning that led to a right answer, which your data does not contain.
• A stronger teacher program, for example the same module on a bigger model, can produce the passing runs for a cheaper student.

Past the bootstrapped demos, max_labeled_demos fills remaining slots with plain training examples. Here both limits were 4, so no plain examples fit.