Middleware: logging and default settings

wrapLanguageModel puts middleware around a model: code that runs before and after every call, for logging, default settings, caching or guardrails.

Examplemiddleware.ts

import { generateText, wrapLanguageModel, defaultSettingsMiddleware, type LanguageModelMiddleware } from "ai";
import { shopModel } from "./shop-model.ts";

const logging: LanguageModelMiddleware = {
  wrapGenerate: async ({ doGenerate, params }) => {
    console.log("temperature:", params.temperature, "| messages:", params.prompt.length);
    const result = await doGenerate();
    console.log("finish:", result.finishReason.unified);
    return result;
  },
};

const model = wrapLanguageModel({
  model: shopModel,
  middleware: [defaultSettingsMiddleware({ settings: { temperature: 0 } }), logging],
});
console.log((await generateText({ model, prompt: "My parcel never arrived" })).text);

Example

npx tsx middleware.ts

The wrapped model is still a model: generateText cannot tell the difference. defaultSettingsMiddleware set temperature: 0 on every call, and the logging middleware saw it in params before calling doGenerate, then read the result. Middleware listed first wraps the ones after it, so the defaults were applied before logging ran.

wrapStream does the same for streaming calls, and transformParams changes the request without wrapping the call, for example to add retrieved documents to the prompt.

Try it yourself

Log result.usage in wrapGenerate.
Swap the order of the two middlewares.
Write a middleware that returns a cached answer for a repeated prompt.

PreviousLoop control: stopWhen and prepareStep

You understood something today that you didn't yesterday.