Middleware

This page covers writing your own middleware. See Middleware in Core for the full hook contract, composition order, and error handling rules.

VernLLMMiddleware is a plain object, no base class or builder required:

minimal-middleware.ts

import { VernLLM, type VernLLMMiddleware } from 'vern-llm';
import { openai } from './openai-client'; // your configured OpenAI client

const loggingMiddleware: VernLLMMiddleware = {
  name: 'logging',
  wrap: async (request, next, ctx) => {
    console.log(`[${ctx.requestId}] calling ${ctx.primaryProvider}/${ctx.primaryModel}`);
    const result = await next();
    console.log(`[${ctx.requestId}] resolved`, result.meta);
    return result;
  },
};

const llm = new VernLLM({
  client: openai,
  model: 'gpt-4o',
  middleware: [loggingMiddleware],
});

`createMiddleware` for the "I only care about failures" case

Wiring a full wrap just to observe a failure is more ceremony than the use case needs. createMiddleware builds one from an onError callback:

create-middleware-onerror.ts

import { createMiddleware } from 'vern-llm';

const alertOnFailure = createMiddleware({
  name: 'alerting',
  onError: (error, ctx) => {
    alerting.page(`${ctx.primaryProvider} call failed: ${error.message}`);
  },
});

onError is called with the error its own next() chain rejects with, not necessarily the call's final outcome: an outer wrap may catch the rethrown error and still return a successful CallResult. Never called on success, and never called for a failure another middleware's wrap already swallowed by short circuiting with its own CallResult. The original error is always rethrown afterward; onError only observes it, the same fire and forget contract onUsage/onEvent already have.

Everything else on CreateMiddlewareOptions passes through unchanged: transform, enabled, priority, onEvent. wrap itself can't be set alongside onError: onError builds its own wrap internally, and supplying both would silently drop one of the two.

Two middleware that need to share a value import the same MiddlewareStateKey<T> reference, created once with createStateKey:

shared-state-key.ts

import { createStateKey, type VernLLMMiddleware } from 'vern-llm';

// Exported so any middleware that needs the same value imports this, not a string.
export const spanIdKey = createStateKey<string>('tracing.spanId');

tracing-and-cost-middleware.ts

import { type VernLLMMiddleware } from 'vern-llm';
import { spanIdKey } from './span-id-key';

const tracing: VernLLMMiddleware = {
  name: 'tracing',
  priority: 1,
  wrap: async (request, next, ctx) => {
    ctx.state.set(spanIdKey, startSpan(ctx.requestId));
    return next();
  },
};

const costTracking: VernLLMMiddleware = {
  name: 'cost-tracking',
  priority: 0, // outer, so its post-next phase runs last, after tracing's own
  wrap: async (request, next, ctx) => {
    const start = Date.now();
    const result = await next();
    recordCost(ctx.state.get(spanIdKey), Date.now() - start, result.meta);
    return result;
  },
};

There's no string key anywhere in this path, so a typo is a missing import or an undefined variable, a compile error, not a silently created new property.

Composition order still matters for ctx.state: see Composition order for when a shared value's write and read ordering is order independent versus order dependent.

Writing your own ordering

priority is a plain number VernLLM sorts by. Nothing stops a small helper computing that number from before/after name references, entirely in userland, without VernLLM ever knowing:

resolve-order.ts

import type { VernLLMMiddleware } from 'vern-llm';

type OrderedMiddleware = VernLLMMiddleware & {
  name: string; // required here, optional on VernLLMMiddleware itself
  before?: string[];
  after?: string[];
};

/**
 * Topologically sorts before/after references into priority numbers, then
 * strips them, returning a plain VernLLMMiddleware[]. Throws on a cycle or
 * an unknown middleware name, naming the offending middleware either way.
 */
export function resolveOrder(entries: OrderedMiddleware[]): VernLLMMiddleware[] {
  const byName = new Map(entries.map((entry) => [entry.name, entry]));

  const dependsOn = new Map<string, Set<string>>();
  const edgesFor = (name: string): Set<string> => {
    let edges = dependsOn.get(name);
    if (!edges) {
      edges = new Set();
      dependsOn.set(name, edges);
    }
    return edges;
  };

  for (const entry of entries) {
    edgesFor(entry.name);

    for (const afterName of entry.after ?? []) {
      if (!byName.has(afterName)) {
        throw new Error(`resolveOrder: "${entry.name}" is after unknown middleware "${afterName}"`);
      }
      edgesFor(entry.name).add(afterName);
    }

    for (const beforeName of entry.before ?? []) {
      if (!byName.has(beforeName)) {
        throw new Error(
          `resolveOrder: "${entry.name}" is before unknown middleware "${beforeName}"`,
        );
      }
      edgesFor(beforeName).add(entry.name);
    }
  }

  const priority = new Map<string, number>();
  const visiting = new Set<string>();

  function assign(name: string): number {
    if (priority.has(name)) return priority.get(name)!;
    if (visiting.has(name)) throw new Error(`resolveOrder: cycle detected at "${name}"`);

    visiting.add(name);
    const deps = [...(dependsOn.get(name) ?? [])];
    const resolved = deps.length === 0 ? 0 : Math.max(...deps.map(assign)) + 1;
    priority.set(name, resolved);
    visiting.delete(name);
    return resolved;
  }

  for (const entry of entries) assign(entry.name);

  return entries.map(({ before, after, ...rest }) => ({
    ...rest,
    priority: priority.get(rest.name!),
  }));
}

resolve-order-usage.ts

import { VernLLM } from 'vern-llm';
import { resolveOrder } from './resolve-order'; // the helper defined above
import { openai } from './openai-client'; // your configured OpenAI client
import { redact } from './redact'; // your redaction helper: (messages) => messages
import { tracingWrap } from './tracing-wrap'; // your tracing `wrap` middleware fn
import { costWrap } from './cost-wrap'; // your cost-tracking `wrap` middleware fn

const llm = new VernLLM({
  client: openai,
  model: 'gpt-4o',
  middleware: resolveOrder([
    { name: 'redaction', transform: (request) => ({ messages: redact(request.messages) }) },
    { name: 'tracing', after: ['redaction'], wrap: tracingWrap },
    { name: 'cost-tracking', before: ['tracing'], wrap: costWrap },
  ]),
});

middleware only ever needs to be a plain VernLLMMiddleware[] by the time it reaches the constructor, so an addon like this, or one that scopes a set of middleware to a single fallback target via enabled, needs no change inside VernLLM itself.

See the Middleware guide for a full worked example combining transform, wrap, and ctx.state.

Middleware

createMiddleware for the "I only care about failures" case

Sharing state with createStateKey

Writing your own ordering

On this page

`createMiddleware` for the "I only care about failures" case

Sharing state with `createStateKey`