Main model and API key

The main model is the entry under models in config.yml with type: main: the LLM that NeMo calls to answer, and to judge messages, unless a rail names another.

Last updated: 30 Sep, 2026 · NeMo Guardrails 0.24.1

The Config folder lesson loaded a model entry without explaining it. This lesson reads it line by line and shows what happens without the key.

Syntax:

models:
  - type: main                      # the model NeMo uses by default
    engine: openai                  # the client: any OpenAI-compatible API
    model: openai/gpt-oss-120b      # the model name the provider expects
    api_key_env_var: GROQ_API_KEY   # which environment variable holds the key
    parameters:
      base_url: https://api.groq.com/openai/v1   # where the API lives
      temperature: 0

engine and base_url

Groq's API speaks the OpenAI format, so the openai engine can call it; base_url points that client at Groq instead of OpenAI. NeMo's documentation gives this route for any OpenAI-compatible provider.

api_key_env_var

Names the variable that holds the key. Without it, the openai engine looks for OPENAI_API_KEY.

models:
  - type: main
    engine: openai
    model: openai/gpt-oss-120b
    api_key_env_var: GROQ_API_KEY
    parameters:
      base_url: https://api.groq.com/openai/v1
      temperature: 0

instructions:
  - type: general
    content: |
      You are an Enterprise IT Assistant specialising in Kubernetes,
      Intel hardware, and enterprise networking.
      Only answer questions about these topics.
      Answer in one or two short sentences.

Calling the model through NeMo

from nemoguardrails import LLMRails, RailsConfig

rails = LLMRails(RailsConfig.from_path("."))


def chat(message):
    reply = rails.generate(messages=[{"role": "user", "content": message}])
    print("User:", message)
    print("Bot :", reply["content"])

chat("What is a Kubernetes ConfigMap?")

User: What is a Kubernetes ConfigMap?
Bot : A ConfigMap stores non‑secret configuration data (key‑value pairs or files) that pods can consume as environment variables, command‑line arguments, or mounted volumes. It lets you decouple configuration from container images.

One question, one answer from Groq. With no rails in the config, NeMo passes the message to the model and hands back its reply.

Starting without the key

The same folder, run where GROQ_API_KEY is not set:

from nemoguardrails import RailsConfig

try:
    RailsConfig.from_path(".")
except ValueError as error:
    print(error.errors()[0]["msg"])

Value error, Model API Key environment variable 'GROQ_API_KEY' not set.

NeMo checks the variable when it loads the config, before any call, and names the variable it looked for. Export the key as in Installation and setup and the load succeeds.

The video's model setup vs this course's

The video's notebook and app create a LangChain ChatGroq and pass it as LLMRails(config, llm=guard_llm), with engine: openai and model: gpt-3.5-turbo left in the YAML as a placeholder. NeMo prints Both an LLM was provided via constructor and a main LLM is specified in the config and uses the constructor's model. This course names the real model in config.yml instead, the route NeMo's model documentation gives for OpenAI-compatible providers since release 0.22, so no LangChain package is needed and the file says which model runs.

# The video's way, shown as it ran there (needs langchain-groq; its llama models are retired)
from langchain_groq import ChatGroq

guard_llm = ChatGroq(api_key=GROQ_API_KEY, model="llama-3.3-70b-versatile", temperature=0)
rails_exp2 = LLMRails(config_exp2, llm=guard_llm)

Where you change this entry

• Moving to another provider: Gemini or OpenRouter, in Groq or Gemini.
• Giving a rail its own smaller model, with another entry of a different type.

Related

• Previous: Config folder
• Next: Instructions that shape the answer
• Reference: Model configuration