Circuit Breaker

const llm = new VernLLM({
  client: fromOpenAI(openai),
  model: 'gpt-4o',
  circuitBreaker: { threshold: 5, cooldownMs: 30_000 }, // or `true` for defaults
});

llm.getCircuitState(); // 'closed' | 'open' | 'half-open' | undefined

Once threshold consecutive failures occur, further calls fail immediately with LLMError('circuit_open') until cooldownMs elapses. The next call is then allowed through as a trial in the half-open state. A successful trial closes the circuit and resets the failure count. A failed trial reopens the circuit and restarts the cooldown.

What counts as a failure

Counts as a failure:

• Timeouts
• Non 2xx responses
• Network errors
• An empty response body
• Other errors indicating that the provider did not successfully return a usable response

Does not count as a failure:

• LLMError('validation')
• LLMError('invalid_params')
• LLMError('parse')
• LLMError('aborted')
• LLMError('validation') with code: 'unknown_tool'
• LLMError('validation') with code: 'duplicate_tool_call_id'
• LLMError('validation') with code: 'tool_choice_none_violated'
• LLMError('validation') with code: 'unexpected_tool_calls'
• LLMError('rate_limited') with code: 'rate_limit_queue_full', 'rate_limit_queue_timeout', or 'rate_limit_capacity_exceeded', a local rateLimit rejection that never reached the provider, whether from a queue giving up or from a call rejected outright before it ever queued. See Interaction with the circuit breaker.
• LLMError('quota_exceeded'). A usage reservation rejection is a caller or account level limit, not a signal that the provider itself is unhealthy, so it's excluded even though it's still retried.

The tool contract errors are model response defects rather than provider health failures. An unknown_tool error means the model requested a tool that was not offered. A duplicate_tool_call_id error means the model reused a tool call ID. Neither error indicates that the provider is unavailable, so neither contributes to the circuit failure count.

These tool contract errors are also not retried. Retrying the same request does not change the available tool contract or repair the duplicate ID.

A response that arrives successfully can still be turned into a circuit breaker failure via detectSoftFailure, the same as a real transport or provider failure would be. See Soft Failure Detection.

Half open behavior

When the cooldown elapses, the next call to check the breaker through assertClosed transitions the state from open to half-open and is allowed through as a trial.

halfOpenProbes sets how many trial calls are allowed through during each half open window, default 1.

Any call beyond that count that checks the breaker while every trial slot is claimed is rejected immediately with LLMError('circuit_open'), without waiting for the outstanding trials to finish. This also applies when several calls check the breaker at approximately the same time after the cooldown expires.

halfOpenSuccessRatio sets the fraction of halfOpenProbes that must succeed to close the circuit, default 1 (every trial must succeed). The circuit stays half-open until every admitted trial has reported a result, then closes if the success ratio was met, or reopens otherwise.

const llm = new VernLLM({
  client: fromOpenAI(openai),
  model: 'gpt-4o',
  circuitBreaker: {
    threshold: 5,
    cooldownMs: 30_000,
    halfOpenProbes: 3,
    halfOpenSuccessRatio: 2 / 3, // 2 of 3 trials succeeding closes the circuit
  },
});

A reopened circuit restarts the cooldown from that moment. The threshold does not need to be reached again, since a failed trial phase is enough to reopen the circuit on its own.

Cooldown backoff

cooldownBackoff grows cooldownMs on each repeat open instead of the same fixed wait every time.

const llm = new VernLLM({
  client: fromOpenAI(openai),
  model: 'gpt-4o',
  circuitBreaker: {
    threshold: 5,
    cooldownMs: 30_000,
    cooldownBackoff: { multiplier: 2, maxMs: 5 * 60_000 },
  },
});

{ multiplier, maxMs } scales the cooldown by multiplier on each repeat open, capped at maxMs (default Infinity). A repeat open counts a trial that failed back to open, not the first open from a closed circuit.

The shorthand always applies full jitter (the computed cooldown is randomized anywhere between zero and its full value), so several client instances don't reopen in lockstep. See AWS's Exponential Backoff and Jitter for why full jitter is preferred over equal jitter. For anything beyond exponential growth, or an exact deterministic value, pass a CooldownBackoff function instead, never jittered automatically. See Circuit Breaker → Customization for a worked example.

The growth resets once the circuit recovers. A successful trial or a manual closeCircuit() sets the repeat count back to 0.

Tripping policy

tripping decides when a bucket's failures should open the circuit, replacing plain threshold counting with a choice of two built in policies, or a custom one.

{ kind: 'consecutive', threshold } is the default. It matches threshold on its own and opens after that many failures in a row.

{ kind: 'rolling', windowMs, minCalls, failureRatio } opens once at least minCalls calls have landed in the trailing windowMs and the failure ratio among them reaches failureRatio. This suits a provider whose failures are frequent but not literally back-to-back. minCalls must be a non-negative integer and failureRatio a finite number in [0, 1]; invalid values throw a RangeError during construction.

const llm = new VernLLM({
  client: fromOpenAI(openai),
  model: 'gpt-4o',
  circuitBreaker: {
    cooldownMs: 30_000,
    tripping: { kind: 'rolling', windowMs: 60_000, minCalls: 20, failureRatio: 0.5 },
  },
});

For anything else, pass a TrippingPolicy: an object with onSuccess(key), onFailure(key) (returning whether this failure should open the circuit), and reset(key). No class is required, a plain object satisfying the interface works. key is the resolved model under isolateByModel, or one fixed shared key otherwise. See Circuit Breaker → Customization for a worked example.

isolateByModel gives each model its own independent tripping state, for the two built in shorthands and for a custom TrippingPolicy alike, since key is what carries the isolation: a single policy instance is always shared, but every call passes the resolved model as key, so a policy that tracks its own state per key gets real per-model isolation automatically, no factory or special handling needed. A custom policy that ignores key and tracks one flat counter stays intentionally shared across every model, that's a choice the policy makes, not a limitation of isolateByModel itself. See Circuit Breaker → Customization for both patterns.

Scope

Each VernLLM instance owns its own circuit breaker. State is never shared across instances. If you are running multiple providers side by side as separate instances, one instance opening its circuit has no effect on another.

The same independence holds within a single instance configured with fallback: each declared target, primary and every fallback target, gets its own breaker if circuitBreaker is set on it. Tripping one target's breaker never affects another target's. An open primary breaker's error is passed to fallbackOn like any other failure, and the chain advances to the next target only when fallbackOn returns 'next'; it does not fall over automatically. See Provider Fallback for a setup with multiple targets on one instance.

The circuit state APIs expose the state of each target independently. getCircuitState() continues to return the primary target's state, while the fallback-aware state API returns the state for every target in declaration order.

const llm = new VernLLM({
  client: fromOpenAI(openai),
  model: 'gpt-4o',
  circuitBreaker: {
    threshold: 5,
    cooldownMs: 30_000,
  },
  fallback: {
    client: anthropic,
    model: 'claude-sonnet-4',
    circuitBreaker: {
      threshold: 5,
      cooldownMs: 30_000,
    },
  },
});

llm.getCircuitState(); // primary target's state

llm.getCircuitStates();
// [
//   { provider: 'primary', index: 0, isFallback: false, isolateByModel: false, state: 'closed' },
//   { provider: 'fallback[0]', index: 1, isFallback: true, isolateByModel: false, state: 'closed' },
// ]

Each target's state is independent. Opening the primary breaker does not open the fallback breaker, and opening a fallback breaker does not affect the primary.

Within a single instance, the default is one shared circuit for every call regardless of which model was used. This includes calls that provide a per call model override. See Per Call Overrides.

Sharing the circuit is useful when several models use the same underlying provider because a burst of failures on any model can indicate that the provider itself is struggling.

Per model isolation

If a single instance is deliberately used for several unrelated models, set isolateByModel: true to give each resolved model its own independent circuit. The sequence below makes the precondition explicit: it drives gpt-4o to the configured threshold with repeated failures while the later gpt-4o-mini call remains successful.

const llm = new VernLLM({
  client: fromOpenAI(openai),
  model: 'gpt-4o',
  circuitBreaker: {
    threshold: 5,
    cooldownMs: 30_000,
    isolateByModel: true,
  },
});

for (let i = 0; i < 5; i++) {
  await llm.call({ userContent: '...', model: 'gpt-4o' }).catch(() => {});
}

await llm.call({
  userContent: '...',
  model: 'gpt-4o-mini',
});

llm.getCircuitState({ model: 'gpt-4o' }); // 'open'
llm.getCircuitState({ model: 'gpt-4o-mini' }); // 'closed'

With isolateByModel enabled, a failure on one model cannot open another model's circuit.

The tradeoff is that a provider wide outage affecting several models takes longer to detect. Each model has its own failure counter, so every affected model must independently reach threshold. With the default shared circuit, failures across models contribute to the same counter and the provider can be marked unhealthy sooner.

Use isolateByModel when models sharing an instance represent genuinely different failure dependencies. If the models are simply different model IDs on the same provider and a provider wide outage should stop traffic quickly, the default shared circuit is usually preferable.

threshold and cooldownMs apply equally to every model circuit. They cannot be configured independently for individual models.

Observing state changes

circuitBreaker.onStateChange fires after every real circuit state transition. It does not fire for a state that remains unchanged.

The callback receives the previous state, the new state, the current consecutive failure count, and the resolved model that triggered the transition:

const llm = new VernLLM({
  client: fromOpenAI(openai),
  model: 'gpt-4o',
  circuitBreaker: {
    threshold: 5,
    cooldownMs: 30_000,
    onStateChange: (from, to, consecutiveFailures, model) => {
      if (to === 'open') {
        alerting.page(`circuit opened for ${model}`);
      }
    },
  },
});

When isolateByModel is disabled, which is the default, model identifies the call that triggered the transition. The failure counter is still shared across all models, so a circuit can open after failures from several different models.

When isolateByModel is enabled, the failure counter and state belong exclusively to the resolved model reported by the callback.

The onStateChange callback is observational. It does not control whether the circuit opens, closes, or transitions to half open.

Every transition also reports a 'circuit_state' event, both on the top-level onEvent option and on each applicable middleware's own onEvent. See Middleware.

Checking and controlling state manually

llm.getCircuitState(); // primary target: 'closed' | 'open' | 'half-open' | undefined

llm.getCircuitStates('gpt-4o');
// [
//   { provider: 'primary', index: 0, isFallback: false, isolateByModel: false, state: 'closed' },
//   { provider: 'fallback[0]', index: 1, isFallback: true, isolateByModel: false, state: 'open' },
//   { provider: 'fallback[1]', index: 2, isFallback: true, isolateByModel: false, state: 'half-open' },
// ]

getCircuitState(target?) and getCircuitStates(model?) are read-only. openCircuit(target?) and closeCircuit(target?) mutate a target's breaker directly:

llm.openCircuit(); // pull the primary out of rotation ahead of known maintenance
llm.closeCircuit(); // put it back, skipping the cooldown

llm.openCircuit({ index: 1 }); // same, for the first fallback target

openCircuit sets the breaker to open and starts the cooldown from now, exactly as if threshold consecutive real failures had just happened. closeCircuit sets it to closed and resets the failure count to zero, without requiring an actual successful call first. Neither call requires a configured circuitBreaker; both are a no-op on a target that doesn't have one.

A manual openCircuit/closeCircuit still reports its 'circuit_state' event to every applicable middleware's own onEvent, the same as a transition triggered by a real call. There's no logical call behind a manual invocation, so it's given a fresh AttemptContext of its own rather than being tied to any in-flight request.

getCircuitState, openCircuit, and closeCircuit all take an optional target: { index?, model? } (getCircuitStates takes a bare model? and reports every target at once):

• index selects which target in the fallback chain to read or act on, defaulting to 0, the primary. An out-of-range index throws RangeError, so it stays distinguishable from a real target that simply has no breaker configured, which returns undefined (for reads) or is a no-op (for openCircuit/closeCircuit) instead of throwing.
• model selects which model's bucket to read or act on, for a target with circuitBreaker.isolateByModel enabled. Passing model to a target that doesn't isolate by model still works, since the shared bucket is used regardless, but logs a warning because the argument had no effect. Omitting model on a target that does isolate defaults to that target's own configured model, the same bucket its real call failures and successes are recorded under, not an unlabeled bucket shared across every caller that also omits it.

getCircuitState() returns the state of the primary target. getCircuitStates() returns the circuit state of every target in the fallback chain, in declaration order, each entry also carrying that target's own isolateByModel setting so a caller sweeping model across a mixed chain can tell which entries actually honored it.

const state = llm.getCircuitState({ model: 'gpt-4o' });

if (state === 'open') {
  console.log('gpt-4o is currently blocked by the circuit breaker');
}

For fallback chains, use getCircuitStates() when you need to inspect all configured targets rather than only the primary:

const states = llm.getCircuitStates();

for (const target of states) {
  if (target.state === 'open') {
    console.log(`${target.provider} is currently blocked by the circuit breaker`);
  }
}

Failure attribution

getFailureBreakdown(target?) returns the current bucket's failure counts by LLMErrorCode, 'unknown' for a failure that carried no code:

llm.getFailureBreakdown();
// { server_error: 3, request_timeout: 1 }

llm.getFailureBreakdown({ index: 1 }); // first fallback target
llm.getFailureBreakdown({ model: 'gpt-4o' }); // for a target with isolateByModel

It takes the same target: { index?, model? } shape as getCircuitState, and returns undefined for a target with no breaker configured, or {} for a bucket that hasn't failed yet.

The breakdown is attribution only. It never decides whether the circuit trips, and clears whenever the bucket does, on a successful call, a successful half-open trial, or a manual closeCircuit().