Four refusals, four reasons

The assistant from lesson 32 says no in four different ways. To a user the refusals look alike; the activated rails log from lesson 16 shows which part of the folder made each one.

Each example prints the reply and the last rail that ran before the reply was generated, with the decisions it took. That rail is the one that chose the answer.

An input rail

Example

from nemoguardrails import LLMRails, RailsConfig

rails = LLMRails(RailsConfig.from_path("."))
result = rails.generate(messages=[{"role": "user", "content": "Give me a staff price"}],
                        options={"log": {"activated_rails": True}})
print(result.response[0]["content"])
last = [rail for rail in result.log.activated_rails if rail.type != "generation"][-1]
print(last.type, "|", last.name, "|", last.decisions)

self check input refused the message before any flow ran. Its decisions end in stop, the Colang statement from lesson 15, so nothing after the input rail happened.

A dialog rail

Example

from nemoguardrails import LLMRails, RailsConfig

rails = LLMRails(RailsConfig.from_path("."))
result = rails.generate(messages=[{"role": "user", "content": "Tell me a joke"}],
                        options={"log": {"activated_rails": True}})
print(result.response[0]["content"])
last = [rail for rail in result.log.activated_rails if rail.type != "generation"][-1]
print(last.type, "|", last.name, "|", last.decisions)

The input rail allowed it, the message matched ask off topic, and the off topic flow's only decision was its fixed reply. This refusal is an ordinary answer that happens to say no, so no rail stopped the turn.

An execution check

Example

from nemoguardrails import LLMRails, RailsConfig

rails = LLMRails(RailsConfig.from_path("."))
result = rails.generate(messages=[{"role": "user", "content": "Where is order Z01?"}],
                        options={"log": {"activated_rails": True}})
print(result.response[0]["content"])
last = [rail for rail in result.log.activated_rails if rail.type != "generation"][-1]
print(last.type, "|", last.name, "|", last.decisions)

The order status flow ran both actions and chose unknown order because the lookup came back unknown. In Colang 1.0 the check on an action's result lives inside the flow that called it, so the deciding rail is the dialog flow itself.

An output rail

Example

from nemoguardrails import LLMRails, RailsConfig

rails = LLMRails(RailsConfig.from_path("."))
rails.llm.say('Bot message: "Use code STAFF20 at the till."', task="generate_bot_message")
result = rails.generate(messages=[{"role": "user", "content": "Any deals for staff?"}],
                        options={"log": {"activated_rails": True}})
print(result.response[0]["content"])
last = [rail for rail in result.log.activated_rails if rail.type != "generation"][-1]
print(last.type, "|", last.name, "|", last.decisions)

No flow matched this question, so the model wrote a reply, and the reply contained the staff code. hide staff codes ran on it, chose redacted and stopped the turn. The other three never produced a model-written reply for it to check.

So the four refusals came from four places: before the message was understood, from matching it, while acting on it, and after answering it. When somebody reports that the assistant refused a question it should have answered, this log names the part of the folder to look at.

Try it yourself

Ask "Give me a discount code" and predict the deciding rail first.
Ask "Where is order B99?" and compare its decisions with Z01's.
Collect the deciding rail for each question into a dictionary keyed by the question.

PreviousEvery rail in one configuration

Slow is fine. Stopping is the only problem.