Evaluate: a score for the whole program

dspy.Evaluate runs a program on every example of a dataset, applies the metric to each, and returns the overall score with every result.

evaluate = dspy.Evaluate(devset=devset, metric=exact, num_threads=1, display_progress=False)
result = evaluate(dspy.Predict(Triage))
print(result.score)

25.0: two of the eight devset tickets were sorted correctly. For a metric that returns bools, score is the percentage that passed; for numbers, the average times 100. num_threads runs examples in parallel; one keeps the order easy to follow.

for example, prediction, score in result.results:
    mark = "ok  " if score else "miss"
    print(mark, f"{prediction.category:9} {example.category:9} {example.ticket}")

result.results is a list of (example, prediction, score). Every miss is a ticket with no keyword, where the stand-in guessed account. Of the two hits, one matched the keyword email, and "I cannot sign in to my profile" was right only because the guess happened to be account. The score hides that; the list shows it. Read the failures before changing anything.

Comparing modules

for program in [dspy.Predict(Triage), dspy.ChainOfThought(Triage)]:
    print(type(program).__name__, evaluate(program).score)

The same score: reasoning cannot fix missing keywords for this stand-in. The comparison, not the idea that reasoning helps, is what tells you whether to pay for it.