$bot_message, and a leak that gets through

An output rail you write yourself reads the reply from $bot_message. This lesson writes one, and then finds a reply it never sees, which is the most useful thing to know about output rails.

An output rail of your own

define bot redacted
  "That answer had an internal code in it, so I cannot send it."

define flow hide staff codes
  $leak = execute has_staff_code
  if $leak
    bot redacted
    stop

async def has_staff_code(context: dict):
    return "STAFF20" in (context.get("bot_message") or "")

The flow asks an action whether the reply contains a staff code, and refuses if it does. The action reads bot_message from the context, the value a flow sees as $bot_message. Lesson 18 listed what the context holds; bot_message is set only once there is a reply, which is why it belongs to output rails.

import pretend_nemo
from nemoguardrails import LLMRails, RailsConfig

from config_actions import has_staff_code

rails = LLMRails(RailsConfig.from_path("."))
rails.register_action(has_staff_code, "has_staff_code")
rails.llm.say('Bot message: "Use code STAFF20 for a staff price."', task="generate_bot_message")
print(rails.generate(messages=[{"role": "user", "content": "Any deals for staff?"}])["content"])

No model is asked whether the reply is acceptable. The check is a plain string test, so it costs nothing, and it is exactly as good as the test: staff20 in lower case would pass it.

The reply the rail never sees

Now a question the configuration answers itself, with a sentence written in define bot that contains the same code.

import pretend_nemo
from nemoguardrails import LLMRails, RailsConfig

from config_actions import has_staff_code

rails = LLMRails(RailsConfig.from_path("."))
rails.register_action(has_staff_code, "has_staff_code")
result = rails.generate(messages=[{"role": "user", "content": "What is the staff price?"}],
                        options={"log": {"activated_rails": True}})
print(result.response[0]["content"])
print([rail.type for rail in result.log.activated_rails])

The code went out, and the log has no output entry at all. The rail did not fail: it never ran. In nemoguardrails/actions/llm/generation.py, when the reply comes from a define bot block, the runtime sets skip_output_rails next to a comment that says so: # We skip output rails for predefined messages.

The reasoning is that output rails exist to check text a model wrote, and a sentence in rails.co was written by you. The consequence is that output rails are not a place to catch secrets in your own configuration. Keep those out of define bot blocks, or check them with a test before they ship.