Middleware Tracing and Cost Tracking

Build a request-tracing and cost-tracking pair with transform, wrap, and ctx.state

This guide builds a middleware pair: one opens a tracing span around every call, a second, independent one reads that span back out to record cost, without either needing to know how the other is implemented.

See Middleware for the full hook contract this guide builds on.

1. Create a shared state key

Two middleware that need to share a value do it through ctx.state, keyed by a typed reference both sides import, not a string either could typo:

span-id-key.ts

import { createStateKey } from 'vern-llm';

export const spanIdKey = createStateKey<string>('tracing.spanId');

2. Write the tracing middleware

wrap opens a span before the call and closes it once next() resolves, spanning the whole logical call, retries and fallback included:

tracing-middleware.ts

import type { VernLLMMiddleware } from 'vern-llm';
import { spanIdKey } from './span-id-key';

export const tracing: VernLLMMiddleware = {
  name: 'tracing',
  priority: 1,
  wrap: async (request, next, ctx) => {
    const spanId = startSpan(ctx.requestId, {
      provider: ctx.primaryProvider,
      model: ctx.primaryModel,
    });
    ctx.state.set(spanIdKey, spanId);

    try {
      const result = await next();
      endSpan(spanId, { status: 'ok', provider: result.meta?.provider });
      return result;
    } catch (error) {
      endSpan(spanId, { status: 'error' });
      throw error;
    }
  },
};

wrap's own ctx is a PreDispatchContext: ctx.primaryProvider/primaryModel describe the primary target before next() resolves, and that's the only identity it has. There's no requestedProvider or isFallbackAttempt to read here at all. result.meta?.provider is what actually served the call, which can differ once a fallback target answers instead.

3. Write the cost-tracking middleware

A second, independent middleware reads the span ID tracing set, without importing anything from tracing beyond the shared key:

cost-tracking-middleware.ts

import type { VernLLMMiddleware } from 'vern-llm';
import { spanIdKey } from './span-id-key';

export const costTracking: VernLLMMiddleware = {
  name: 'cost-tracking',
  priority: 0, // outer: see "Why cost-tracking is priority 0" below
  wrap: async (request, next, ctx) => {
    const start = Date.now();
    const result = await next();

    recordCost({
      spanId: ctx.state.get(spanIdKey),
      elapsedMs: Date.now() - start,
      tokens: result.meta ? undefined : 0, // see step 5 for reading real usage
    });

    return result;
  },
};

4. Register both, in either order

middleware is one flat array; priority decides nesting, not array position:

middleware-registration.ts

import { VernLLM } from 'vern-llm';
import { openai } from './openai-client'; // your configured OpenAI client
import { costTracking } from './cost-tracking-middleware';
import { tracing } from './tracing-middleware';

const llm = new VernLLM({
  client: openai,
  model: 'gpt-4o',
  middleware: [tracing, costTracking], // order here doesn't matter, priority does
});

5. Read real token usage instead of a placeholder

result.meta doesn't carry token counts, onUsage does. Pair wrap with onUsage for cost that needs real spend, not just wall clock time:

cost-tracking-with-usage.ts

import { VernLLM } from 'vern-llm';
import { openai } from './openai-client'; // your configured OpenAI client
import { costTracking } from './cost-tracking-middleware';
import { tracing } from './tracing-middleware';

const llm = new VernLLM({
  client: openai,
  model: 'gpt-4o',
  middleware: [tracing, costTracking],
  onUsage: (usage) => {
    recordTokenCost(usage.requestId, usage.totalTokens);
  },
});

onUsage fires per real provider response; wrap's next() resolves once per logical call. Correlate the two by requestId for per-attempt token cost instead of just the logical call's total elapsed time.

Why `cost-tracking` is `priority: 0`

Lower priority is outermost: first to start, last to finish. cost-tracking needs to start its timer before tracing opens the span, and read the span ID back out after tracing's own wrap has already resolved, so it needs to be the outer one:

cost-tracking starts timer  (pre-next, priority 0, runs first)
  tracing opens span, sets spanIdKey  (pre-next, priority 1, runs second)
    the real call happens
  tracing closes span, resolves  (post-next, priority 1, resolves first)
cost-tracking reads spanIdKey, records cost  (post-next, priority 0, resolves last)

This pair would also work with the priorities swapped, since spanIdKey is written in one middleware's pre-next and read in the other's post-next, and every pre-next finishes before any post-next begins regardless of nesting. Order would start to matter if cost-tracking needed to read spanIdKey in its own pre-next instead, say to tag a log line before the call even starts, since then tracing would need to run its own pre-next first. See Composition order for the general rule.

Filtering to one target

enabled runs from two stages: once gating wrap composition, before any target is chosen (ctx.stage: 'pre-dispatch'), and once per attempt gating transform, once a target has been selected for that attempt (ctx.stage: 'attempt'). Its own ctx is always the full union, so a predicate reading isFallbackAttempt narrows on ctx.stage first. Scope a transform to just the primary target, or just fallback attempts, this way:

scope-transform-to-primary.ts

import type { VernLLMMiddleware } from 'vern-llm';

export const primaryOnlyRedaction: VernLLMMiddleware = {
  name: 'primary-only-redaction',
  enabled: (ctx) => ctx.stage === 'pre-dispatch' || !ctx.isFallbackAttempt,
  transform: (request) => ({ messages: redact(request.messages) }),
};

isFallbackAttempt only exists on AttemptContext (ctx.stage === 'attempt'). It's not on PreDispatchContext at all, so the example above returns true unconditionally on 'pre-dispatch' (nothing meaningful to filter there yet) and applies the real check only once ctx.stage === 'attempt'. This means gating a wrap-only middleware with isFallbackAttempt can't work, no matter how it's narrowed: wrap itself only ever receives a PreDispatchContext, never an AttemptContext, so there's no stage at which it could read the real answer. Read result.meta?.provider after next() resolves instead if wrap needs to know which target actually answered. See wrap for details.

Both middleware here use wrap, never transform: neither changes the outgoing request, so there's nothing for transform to patch. Reach for transform instead when a middleware needs to redact content or add a tool before dispatch.