A model NeMo can call, with no API key

Lesson 3 left a configuration that names a model it cannot reach. A provider in NeMo Guardrails is any object with five members, and you can register your own under any engine name. That is the whole trick that makes this course free to run.

Build the smallest possible one first: a model that says the same sentence every time. It is useless as an assistant and perfect for seeing what the runtime does with it.

The class

from nemoguardrails import LLMResponse, LLMResponseChunk, register_provider


class EchoModel:
    def __init__(self, model="echo-1", **kwargs):
        self._model = model

    model_name = property(lambda self: self._model)
    provider_name = property(lambda self: "echo")
    provider_url = property(lambda self: None)

Three properties, and the runtime reads all three when it builds a model. provider_url may be None; the other two are used in logs and error messages.

    async def generate_async(self, prompt, *, stop=None, **kwargs):
        return LLMResponse(content="Ask the shop, not me.",
                           model=self._model, finish_reason="stop")

This goes inside the class. generate_async is where a real provider would call an API. prompt arrives as a string or as a list of messages, and the reply has to be an LLMResponse.

The bug: five members, not four

from nemoguardrails import LLMRails, RailsConfig

ECHO_YML = """
models:
  - type: main
    engine: echo
    model: echo-1
"""
register_provider("echo", EchoModel)
rails = LLMRails(RailsConfig.from_content(yaml_content=ECHO_YML))
print(rails.generate(messages=[{"role": "user", "content": "hello"}])["content"])

No traceback, just a polite apology. The runtime built the model, tried to call it, and refused. The real message went to the log, and the reason is a protocol check.

from nemoguardrails.types import LLMModel

print(isinstance(EchoModel(), LLMModel))

LLMModel is a protocol with five members and this class has four. The missing one is stream_async, which generate never uses and still insists on.

The fix

    async def stream_async(self, prompt, *, stop=None, **kwargs):
        yield LLMResponseChunk(delta_content="Ask the shop, not me.",
                               model=self._model)
        yield LLMResponseChunk(model=self._model, finish_reason="stop")

Also inside the class. It has to be an async generator, so it yields rather than returns: one chunk carrying the text, then one carrying the reason it stopped.

register_provider("echo", EchoModel)
rails = LLMRails(RailsConfig.from_content(yaml_content=ECHO_YML))
print(rails.generate(messages=[{"role": "user", "content": "hello"}])["content"])
print(rails.generate(messages=[{"role": "user", "content": "anything at all"}])["content"])

A working guardrails application with no key and no network. It is a terrible assistant, and that is the point: from here on, every strange thing you see comes from the runtime, not from a model you cannot inspect.

From echo to something that decides

A model that always says one sentence cannot show you anything about rails, because every rail would see the same answer. Lesson 5 turns the echo into a stand-in that reads the question and chooses, and lesson 6 shows you how to watch it doing so.