The guarded support desk

Lesson 0 promised a support desk that will not promise a refund, will not name a competitor, and will not send a customer's password to a model. Every piece of it was built alone in an earlier lesson. This is all of them in one file.

The file

Three validators, one Guard with two targets, and a function that turns a question into something safe to send. Nothing here is new: NoSecrets is lesson 15, NoRefunds and its fix_value are lessons 8 and 10, NoCompetitors and its metadata are lesson 8, refrain is lesson 11, and the two use targets are lesson 6.

python

"""The shop's support desk: one Guard, three rules, and a model."""
from typing import Dict

from guardrails import Guard, register_validator
from guardrails.validators import FailResult, PassResult, ValidationResult, Validator
from guardrails_ai.valid_length import ValidLength

RIVALS = {"competitors": ["ShopFast", "QuickCart"]}


@register_validator(name="shop/no-secrets", data_type="string")
class NoSecrets(Validator):
    def _validate(self, value: str, metadata: Dict) -> ValidationResult:
        for word in ("password", "card number", "cvv"):
            if word in value.lower():
                return FailResult(error_message=f"The message contains a {word}.")
        return PassResult()


@register_validator(name="shop/no-refunds-desk", data_type="string")
class NoRefunds(Validator):
    def _validate(self, value: str, metadata: Dict) -> ValidationResult:
        if "refund" in value.lower():
            return FailResult(
                error_message="The reply promises a refund.",
                fix_value=value.replace("refund", "offer store credit for"),
            )
        return PassResult()


@register_validator(name="shop/no-competitors-desk", data_type="string")
class NoCompetitors(Validator):
    def _validate(self, value: str, metadata: Dict) -> ValidationResult:
        named = [c for c in metadata.get("competitors", []) if c.lower() in value.lower()]
        if named:
            return FailResult(error_message=f"The reply names {named[0]}.")
        return PassResult()


def build():
    return (Guard()
            .use(NoSecrets(on_fail="exception"), on="messages")
            .use(NoRefunds(on_fail="fix"),
                 NoCompetitors(on_fail="refrain"),
                 ValidLength(min=1, max=80, on_fail="noop")))


def ask(model, question):
    desk = build()
    try:
        answer = desk(model, messages=[{"role": "user", "content": question}],
                      metadata=RIVALS)
    except Exception as error:
        return desk, "[not sent] %s" % error
    if answer.validated_output is None:
        return desk, "[held back] a human will reply to this one"
    return desk, answer.validated_output

The length rule is on noop on purpose. It is a new rule and nobody has measured how often a real answer runs past eighty characters, so for now it only records, which is the argument lesson 9 made for noop.

The ordinary case

Example

from pretend_guardrails import PretendModel
from support_desk import ask

print(ask(PretendModel(), "Where is order 8821?")[1])
print(ask(PretendModel(), "hello")[1])

A question the desk can answer, and one it cannot. Both replies went out unchanged, because no rule objected to either.

The three failures

A desk that only works is a demo. These are the three ways this one refuses, and each of them is the behaviour a different lesson set up.

Example

from pretend_guardrails import PretendModel
from support_desk import ask

print(ask(PretendModel(), "Can I get a refund on order 8821?")[1])
print(ask(PretendModel(), "my password is hunter2, where is 8821?")[1])
print(ask(PretendModel(replies=["Try ShopFast, they are quicker."]), "who is faster?")[1])

The refund was repaired. NoRefunds returned a fix_value and the customer got an offer the shop can actually honour.

The password never left the building. NoSecrets is attached to messages with on_fail="exception", so the Guard raised before the model was called.

The competitor answer was withheld. refrain gives back None, and ask turns that into a sentence a human can pick up. This is the case lesson 11 said to plan for: None means the Guard will not vouch for anything, and your code has to decide what that looks like to a customer.

Proving the model was never called

Example

from pretend_guardrails import PretendModel
from support_desk import ask

model = PretendModel()
print(ask(model, "my card number is 4111 1111 1111 1111")[1])
print("model calls:", len(model.prompts))

What the desk recorded

Example

from pretend_guardrails import PretendModel
from support_desk import ask

desk, reply = ask(PretendModel(), "Can I get a refund on order 8821?")

for log in sorted(desk.history.last.validator_logs, key=lambda l: l.validator_name):
    print(log.validator_name, log.validation_result.outcome)

Four validators ran, one failed, and the reply still went out because the failure was repaired. That single list is what you would put on a dashboard: which rule, how often, and whether the customer ever saw the difference.

Try it yourself

Move the length rule from noop to fix and see which of the five replies changes.
Add QuickCart to a reply and confirm the second competitor is caught too.
Replace the refrain on NoCompetitors with exception and decide which one you would ship.

PreviousA real model, in one line

This is what real progress feels like.