embeddings_only, and the threshold

Matching a message to an intent normally costs a model call. embeddings_only makes the embedding model decide alone, which is free and fast, and it needs two more settings before it is safe.

No model call at all

yaml

rails:
  dialog:
    user_messages:
      embeddings_only: true

Example

import pretend_nemo
from nemoguardrails import LLMRails, RailsConfig

rails = LLMRails(RailsConfig.from_path("."))
for asked in ["When do I get my money back?", "What is the weather"]:
    result = rails.generate(messages=[{"role": "user", "content": asked}], options={"log": {"activated_rails": True, "llm_calls": True}})
    print(asked, "->", result.response[0]["content"], [call.task for call in result.log.llm_calls])

The refund question was answered correctly with no model call. So was the weather question, with the wrong answer: it was matched to ask order status. With embeddings_only on its own, the nearest intent always wins, however far away it is.

A threshold

Example

import pretend_nemo
from nemoguardrails import LLMRails, RailsConfig

rails = LLMRails(RailsConfig.from_path("."))
for asked in ["When do I get my money back?", "What is the weather"]:
    result = rails.generate(messages=[{"role": "user", "content": asked}], options={"log": {"activated_rails": True, "llm_calls": True}})
    print(asked, "->", result.response[0]["content"], [call.task for call in result.log.llm_calls])

Below the threshold, the match is not trusted and the question goes back to the model for generate_user_intent, the usual path. The refund question still costs nothing.

A fallback intent

Example

import pretend_nemo
from nemoguardrails import LLMRails, RailsConfig

rails = LLMRails(RailsConfig.from_path("."))
for asked in ["When do I get my money back?", "What is the weather"]:
    result = rails.generate(messages=[{"role": "user", "content": asked}], options={"log": {"activated_rails": True, "llm_calls": True}})
    print(asked, "->", result.response[0]["content"], [call.task for call in result.log.llm_calls])

With embeddings_only_fallback_intent set, anything below the threshold is given that intent without asking the model, and the off-topic flow from lesson 24 answers it. That is the combination that makes the setting useful: zero calls for questions you expected, and a fixed reply for everything else.

The threshold is not cosine similarity

Example

import math

for cosine in [1.0, 0.5, 0.0]:
    print(cosine, round(1 - math.sqrt(2 - 2 * cosine) / 2, 3))

The score compared with the threshold is 1 - sqrt(2 - 2·cos) / 2, which nemoguardrails/embeddings/basic.py computes this way so that thresholds written for an older index keep their meaning. Vectors with no negative numbers, like the stand-in's, never score below 0.293, so a threshold of 0.2 would let every match through.

Try it yourself

Set the threshold to 0.2 and run the weather question again.
Print the score for a cosine of 0.9 and pick a threshold that sits between it and 0.5.
Leave the threshold out but keep the fallback intent, and see whether the fallback is ever used.

PreviousDialog rails: only what you defined

Every expert started right here.