When the JSON is not JSON
Lesson 19 asked the model for JSON properly. Models still get it wrong, and a structured Guard fails differently from a string one.
from pretend_guardrails import PretendModel
model = PretendModel(replies=["Sure! Here you go: {'order_id': 8821, 'issue': 'late'}"])
tickets = Guard.for_pydantic(Ticket)
outcome = tickets(model, messages=[{"role": "user", "content": "File a ticket."}], num_reasks=0)
print(outcome.validation_passed)
print(outcome.error)
print(outcome.reask.fail_results[0].error_message)Single quotes are not JSON. The extraction step from lesson 17 found something that looked like an object, handed it to the JSON parser, and the parser refused at the first character it did not recognise.
Two places record it. outcome.error carries the parser's own complaint, which is the one that tells you where. outcome.reask carries the Guardrails version, which is the one that would have gone back to the model.
Letting it try again
model = PretendModel(replies=[
"Sure! Here you go: {'order_id': 8821, 'issue': 'late'}",
'{"order_id": "8821", "issue": "late", "priority": 2}',
])
tickets = Guard.for_pydantic(Ticket)
outcome = tickets(model, messages=[{"role": "user", "content": "File a ticket."}], num_reasks=1)
print(outcome.validation_passed)
print(outcome.validated_output)
print("model calls:", len(model.prompts))No on_fail anywhere in that snippet. A structural failure reasks on its own when num_reasks allows it, which is different from lesson 16, where reasking was a choice a validator made.
print(model.prompts[1][-1]["content"][:420])The second prompt carries the schema whether or not you put the placeholder in the first one. Guardrails knows the shape it wanted; it simply does not volunteer it until something has gone wrong.
Which is a reasonable default and an expensive one. A first prompt with the placeholder costs tokens once. A first prompt without it costs a wasted call every time the model guesses badly.
- Script a second reply that is also broken and watch the reask budget run out.
- Return valid JSON with
priorityas the word high and see which of the two failure kinds you get. - Set
num_reasks=2and script three replies. Count the iterations in the history.
Every expert started right here.