Configuration

Everything below is set once on the VernLLM instance and applies to every call() unless overridden per request. Nothing is required beyond client and model; every other option has a sensible default.

import { InMemoryCacheAdapter } from 'vern-llm';

const llm = new VernLLM({
  client: openai,
  model: 'gpt-4o',
  name: 'primary',
  maxRetries: 1,
  timeoutMs: 25_000,
  chunkIdleTimeoutMs: 30_000,
  baseDelayMs: 500,
  defaultMaxTokens: 1000,
  nonRetryableStatus: [400, 401, 403, 404, 422],
  circuitBreaker: true,
  rateLimit: { requestsPerMinute: 500, tokensPerMinute: 200_000, maxConcurrent: 20 },
  fallback: { client: anthropicClient, model: 'claude-sonnet-5', circuitBreaker: true },
  debug: false,
  cache: new InMemoryCacheAdapter(),
  parseJson: (content) => {
    try {
      return JSON.parse(content);
    } catch {
      return undefined;
    }
  },
  onUsage: (usage) => billing.record(usage),
  onUsageFailure: (usage, error) => billing.record(usage),
  onEvent: (event) => observability.record(event),
  logger: myLogger,
});

Required

Provider identity

name does not affect request routing or the model sent to the provider. It is an application-level label that lets you distinguish multiple VernLLM instances in shared billing and observability pipelines.

Retry & timeout

Request defaults

Caching

The cache option controls the adapter used by cachedCall.

By default, VernLLM creates an InMemoryCacheAdapter. This stores cached responses in the current process memory using a bounded in-memory map. It is useful for local development and single-process deployments, but entries are not shared between application instances.

For distributed deployments, provide your own adapter backed by shared storage such as Redis or Upstash.

InMemoryCacheAdapter accepts an optional maxSize argument to limit the number of entries kept in memory. When the limit is reached, it evicts by 'fifo' (default), removing the oldest inserted entry, or 'lru', removing the least recently used entry. See Eviction.

Two additional built-in adapters compose on top of any CacheAdapter: NormalizedCacheAdapter (normalizes keys before matching, including case, whitespace, and punctuation) and TieredCacheAdapter (checks a fast local adapter first, falling back to a shared one). See Caching for usage.

Observability

See Observability for the full onEvent union and event shapes.

Circuit breaker

The circuit breaker tracks provider and transport-level health. Response parsing and validation errors do not count as provider failures. unknown_tool and duplicate_tool_call_id tool-contract errors also do not count.

With isolateByModel: false (the default), one circuit is shared across models used by the same target. With isolateByModel: true, each resolved model gets its own circuit and getCircuitState({ model }) can be used to inspect that model's state. Omitting model reads or acts on the target's own configured model rather than an unlabeled bucket.

halfOpenProbes (default 1) and halfOpenSuccessRatio (default 1) control how many trial calls run during the half-open window and what fraction must succeed to close the circuit again.

cooldownBackoff accepts ExponentialBackoffOptions | CooldownBackoff:

• ExponentialBackoffOptions: { multiplier: number; maxMs?: number }. Scales cooldownMs by multiplier on each repeat open, capped at maxMs (default Infinity). Always applies full jitter.
• CooldownBackoff: (reopenCount: number, baseCooldownMs: number) => number, an escape hatch for anything beyond exponential growth, or an exact deterministic value. Never jittered automatically.

Omitted (the default), cooldownMs stays fixed.

tripping accepts { kind: 'consecutive', threshold } | { kind: 'rolling', windowMs, minCalls, failureRatio } | TrippingPolicy:

• { kind: 'consecutive', threshold }: the default, matching threshold on its own. Opens after that many failures in a row.
• { kind: 'rolling', windowMs, minCalls, failureRatio }: opens once at least minCalls calls have landed in the trailing windowMs and the failure ratio among them reaches failureRatio.
• TrippingPolicy: { onSuccess(), onFailure(): boolean, reset() }, an escape hatch for anything else. No class required, a plain object satisfying the interface works.

onStateChange, the circuit_state event, and the open-circuit error message always report a true consecutive-failure count, tracked independently of tripping.

Omitted (the default), tripping is { kind: 'consecutive', threshold }.

See Circuit Breaker for failure conditions, half-open behavior, cooldown backoff, tripping policy, failure attribution, per-model isolation, and state inspection.

Rate limiting

A local rate-limit rejection fails immediately with LLMError('rate_limited') and code: 'rate_limit_queue_timeout', 'rate_limit_queue_full', or 'rate_limit_capacity_exceeded'. It is never retried and does not count as a circuit-breaker failure.

A provider 429 carries code: 'provider_rate_limited' and is handled separately from local rate-limit rejections. See Error codes.

Retry budget

Once at least minCalls calls have landed in the trailing windowMs and the retry ratio among them reaches retryRatio, further retries against that target fail immediately with LLMError('rate_limited') and code: 'retry_budget_exhausted'. It is never retried and does not count as a circuit-breaker failure. Like circuitBreaker and rateLimit, a target's own retryBudget is never inherited: each target needs its own if it wants one.

Fallback

Each FallbackTarget needs its own client and model. name, maxRetries, timeoutMs, chunkIdleTimeoutMs, baseDelayMs, defaultMaxTokens, defaultTemperature, nonRetryableStatus, and detectSoftFailure inherit the parent instance's resolved value when omitted.

circuitBreaker, rateLimit, and retryBudget are never inherited. Each target has its own independent breaker, rate limiter, and retry budget when configured.

An open target breaker is treated as a target failure and can cause the chain to move to the next target according to fallbackOn.

When every target fails, call() throws FallbackExhaustedError containing the normalized error from each attempted target in declaration order. See FallbackExhaustedError.

Per call defaults and overrides

A handful of options are set per call() rather than on the instance. The following options have their own defaults worth knowing:

Usage metering

reserveUsage and refundUsage are provided per call, not on the VernLLM instance.

These hooks are application controlled. VernLLM only manages the reservation lifecycle; it does not decide quota rules or billing policy.

userContent, model, maxTokens, and reasoningEffort have no call-level default beyond falling back to the corresponding instance-level field, or being undefined when omitted.

schema, jsonSchema, tools, signal, and requestId have no instance-level equivalent; they are undefined unless set on the call itself.

requestId is an optional identifier you can provide per call:

const result = await llm.call({
  userContent: 'Generate a summary',
  requestId: `document-summary:${documentId}`,
});

Multimodal userContent

ContentBlock is the shape used when userContent is passed as an array instead of a plain string:

interface TextBlock {
  type: 'text';
  text: string;
}

interface ImageBlock {
  type: 'image';
  data: string; // base64-encoded bytes, no `data:` prefix
  mimeType: string; // e.g. 'image/png', 'image/jpeg', 'image/webp', 'image/gif'
}

type ContentBlock = TextBlock | ImageBlock;

Image blocks validate mimeType against VernLLM's supported image set before a provider request is created. Supported values are image/png, image/jpeg, image/gif, and image/webp.

Unsupported values fail with LLMError('validation').