The prompt cache, and a fallback

The SDK caches prompts in memory, so only the first get_prompt waits for the network. With no cache and no server, get_prompt raises, unless you give it a fallback.

Lesson 20 mentioned a cache that delays a label change in other programs. The same cache is why fetching a prompt on every request is fine, and why an application keeps working through a short Langfuse outage.

langfuse.create_prompt(name="desk-system", prompt="Answer in one short sentence.", labels=["production"])

for _ in range(3):
    langfuse.get_prompt("desk-system")
print(local_langfuse.REQUESTS)

python cache.py

Three calls, one request. The first fetched the prompt; the other two came from memory. After the 60 second lifetime, the next call still returns the cached prompt at once and fetches a new copy in the background. cache_ttl_seconds changes the lifetime, and 0 turns caching off, which the documentation suggests only outside production.

When Langfuse cannot be reached

A cache only helps once it has something in it. A freshly started process has an empty cache. Port 9 on your machine, where nothing listens, stands in for a Langfuse that is down.

from langfuse import Langfuse

langfuse = Langfuse(public_key="pk-lf-local", secret_key="sk-lf-local", base_url="http://127.0.0.1:9")

try:
    prompt = langfuse.get_prompt("desk-system")
except Exception as error:
    print("no prompt:", type(error).__name__)

python down.py

With nothing cached and no answer after its retries, get_prompt raised. Uncaught, that error would stop the desk from answering anyone, which is a worse outage than Langfuse's.

from langfuse import Langfuse

langfuse = Langfuse(public_key="pk-lf-local", secret_key="sk-lf-local", base_url="http://127.0.0.1:9")

prompt = langfuse.get_prompt("desk-system", fallback="Answer in one short sentence.")
print(prompt.is_fallback, "|", prompt.compile())

python down.py

With fallback, the SDK logs the failure and returns a prompt built from the text you gave it, marked is_fallback. Keep the fallback close to the production prompt. The other option in the documentation is to fetch every prompt when the application starts and refuse to start without them, so a new instance never serves without its prompt.

Versions and labels live in Langfuse; the cache and the fallback live in your process. Between them, a prompt change reaches the desk within a minute, and a Langfuse outage does not reach customers at all.