Rate Limiting

const llm = new VernLLM({
  client: fromOpenAI(openai),
  model: 'gpt-4o',
  rateLimit: {
    requestsPerMinute: 500,
    tokensPerMinute: 200_000,
    maxConcurrent: 20,
  },
});

rateLimit queues calls locally to stay under configured limits, instead of dispatching every call and letting the provider reject the ones that go over. This is proactive: the existing Retry-After handling in Retries is reactive, it only recovers after a self-inflicted 429 has already cost a round trip. rateLimit avoids tripping the limit in the first place.

Every bucket is independent and optional. Omit rateLimit entirely, or omit any individual field within it, and that dimension is unlimited, exactly matching behavior before this option existed.

The three buckets

A call only proceeds once every configured bucket has room. If any one of them is short, the call queues until all three clear.

Queueing

Calls that can't get capacity immediately queue in strict FIFO order. A large call is never starved by a stream of smaller ones queued behind it, VernLLM never reorders the queue to let a smaller request skip ahead.

const llm = new VernLLM({
  client: fromOpenAI(openai),
  model: 'gpt-4o',
  rateLimit: {
    requestsPerMinute: 60,
    maxQueueMs: 10_000, // default 30000, pass 0 to wait indefinitely
    maxQueueSize: 50, // default 0, unbounded
  },
});

A call that exceeds maxQueueMs, or arrives when the queue is already at maxQueueSize, fails with LLMError('rate_limited') carrying code: 'rate_limit_queue_timeout' or code: 'rate_limit_queue_full' respectively:

import { isLLMError } from 'vern-llm';

try {
  await llm.call({ userContent: '...' });
} catch (err) {
  if (
    isLLMError(err) &&
    (err.code === 'rate_limit_queue_timeout' || err.code === 'rate_limit_queue_full')
  ) {
    // Never reached the provider. Retrying immediately won't help,
    // the wait already happened. See "Interaction with retries" below.
  }
}

An aborted signal on a queued call removes it from the queue immediately and rejects with LLMError('aborted'), freeing that queue slot for the next waiter. See Cancellation & Timeouts.

Estimating tokens

tokensPerMinute needs a token count before the request is sent, when the real count isn't known yet. The default estimate is a chars / 4 heuristic over every message's content, plus the requested maxTokens:

Math.ceil(messagesChars / 4) + (request.max_tokens ?? 0);

This is intentionally rough. Once the call completes, the estimate is reconciled against the provider's real reported usage, so a systematically over- or under-estimating heuristic self-corrects over time rather than compounding.

Provide estimateTokens to use a real tokenizer, or any other heuristic your provider or model mix calls for:

import { encode } from 'gpt-tokenizer';

const llm = new VernLLM({
  client: fromOpenAI(openai),
  model: 'gpt-4o',
  rateLimit: {
    tokensPerMinute: 200_000,
    estimateTokens: (request) => {
      const promptTokens = request.messages.reduce(
        (sum, m) => sum + encode(typeof m.content === 'string' ? m.content : '').length,
        0,
      );
      return promptTokens + (request.max_tokens ?? 0);
    },
  },
});

Per-attempt, not per-call

Capacity is acquired for each attempt, including retries, not once for the whole logical call. Every retry is a real request against the same provider limits, so it needs to clear the same buckets again:

const llm = new VernLLM({
  client: fromOpenAI(openai),
  model: 'gpt-4o',
  maxRetries: 3,
  rateLimit: { requestsPerMinute: 500 },
});

If the first attempt fails and a retry is due, that retry queues for capacity exactly like the original attempt did.

Bulkhead isolation with fallback

Each target in a fallback chain, primary and every fallback, gets its own rateLimit, entirely independent of every other target's. Setting maxConcurrent per target keeps one target's load from starving another:

const llm = new VernLLM({
  client: fromOpenAI(openai),
  model: 'gpt-4o',
  rateLimit: { maxConcurrent: 50 },

  fallback: {
    client: fromAnthropic(anthropic),
    model: 'claude-sonnet-5',
    rateLimit: { maxConcurrent: 10 },
  },
});

Without this, a burst against a struggling primary could consume every concurrency slot in a single shared pool, starving calls that would have gone to a healthy fallback instead. Because each target's limiter is its own, the fallback keeps its reserved capacity no matter how saturated the primary gets. rateLimit is one of the two options never inherited from the parent instance, see Fields that don't inherit.

stream: true

For a streaming call, capacity is held for the connection's entire lifetime and released only once the stream completes, either by finishing normally or by a mid-stream failure, not the moment it merely opens. A stream holds a real connection to the provider the whole time it's open, so it continues to count against maxConcurrent for as long as that connection is live.

See Streaming for the rest of the stream: true contract.

Interaction with retries

The three local rate-limit codes (rate_limit_queue_full, rate_limit_queue_timeout, rate_limit_capacity_exceeded) are never retried. For the two queue codes, the wait already happened while the call was queued, so retrying immediately would only requeue it behind the same limit with nothing changed; for rate_limit_capacity_exceeded, the call could never fit the configured capacity regardless of how long it waits:

if (
  error instanceof LLMError &&
  (error.code === 'rate_limit_queue_full' ||
    error.code === 'rate_limit_queue_timeout' ||
    error.code === 'rate_limit_capacity_exceeded')
) {
  return false;
}

See What gets retried for the full list of non-retried error conditions.

Interaction with the circuit breaker

A local rate-limit error never reached the provider, so it says nothing about the provider's health. It does not count toward the circuit breaker's failure threshold, the same treatment tool contract errors already get. See What counts as a failure.

Provider 429s

rateLimit is a local, proactive control. It does not change how an actual provider 429 is handled, that still flows through the existing retry and Retry-After machinery. A provider 429 does now also carry code: 'provider_rate_limited', so you can tell a rate limit the provider itself imposed apart from one VernLLM enforced locally, without inspecting status directly:

if (isLLMError(err) && err.code === 'provider_rate_limited') {
  // A real provider 429. Already retried per the normal retry/Retry-After flow.
}

if (
  isLLMError(err) &&
  (err.code === 'rate_limit_queue_full' || err.code === 'rate_limit_queue_timeout')
) {
  // Never left this process. VernLLM's own queue gave up.
}

AIMD

requestsPerMinute can be a fixed ceiling, or it can adjust itself: growing on success, shrinking on a real rate limit, via aimd (additive increase / multiplicative decrease):

const llm = new VernLLM({
  client: fromOpenAI(openai),
  model: 'gpt-4o',
  rateLimit: {
    requestsPerMinute: 500,
    aimd: { increaseBy: 10, decreaseFactor: 0.5, minCapacity: 50, maxCapacity: 1000 },
  },
});

increaseBy is added to the ceiling on every clean release. decreaseFactor is multiplied in on a rate-limit signal, bounded to [minCapacity, maxCapacity]. Requires requestsPerMinute to be set; ignored otherwise.

Reactive, on a real 429, works for every adapter with no configuration beyond aimd itself: the ceiling shrinks by decreaseFactor unconditionally, before the retry/Retry-After handling above runs.

Proactive, before a 429 happens, needs proactiveFloor plus, for two of the four built-in adapters, an explicit opt-in:

const llm = new VernLLM({
  client: fromOpenAI(openai, { supportsWithResponse: true }),
  model: 'gpt-4o',
  rateLimit: {
    requestsPerMinute: 500,
    aimd: {
      increaseBy: 10,
      decreaseFactor: 0.5,
      minCapacity: 50,
      maxCapacity: 1000,
      proactiveFloor: 20,
    },
  },
});

Once a response reports remainingRequests at or below proactiveFloor, the ceiling shrinks right there, rather than waiting for an actual rejection. Default 0, meaning off. Not every client can produce this hint:

supportsWithResponse only ever affects the proactive path; every adapter above still reacts to a real 429 regardless, since that only needs the response status, not header access.

Observing waits

onEvent reports a rate_limited event whenever an attempt actually had to wait for capacity:

{
  kind: 'rate_limited',
  requestId: string,
  provider: string,
  model: string,
  waitedMs: number,
  reason: 'concurrency' | 'rpm' | 'tpm',
}

reason identifies which bucket was blocking the call just before it cleared. An attempt that never had to wait (capacity was immediately available) does not emit this event.

const llm = new VernLLM({
  client: fromOpenAI(openai),
  model: 'gpt-4o',
  rateLimit: { requestsPerMinute: 500, maxConcurrent: 20 },
  onEvent: (event) => {
    if (event.kind === 'rate_limited') {
      metrics.observe('llm.rate_limit.wait_ms', event.waitedMs, { reason: event.reason });
    }
  },
});

See Event kinds for the rest of the onEvent union.

Sharing a limiter across processes

rateLimit normally builds a fresh, in-process limiter from the config object above. That limiter has no idea about any other process. Two VernLLM instances, or a horizontally scaled deployment, each get their own independent buckets, so the real ceiling a provider enforces is never actually shared across any of them.

Pass a RateLimiterAdapter instead of a config object to use your own limiter, coordinated however you like, Redis backed or otherwise:

import { VernLLM, type RateLimiterAdapter } from 'vern-llm';

const sharedLimiter: RateLimiterAdapter = new MyRedisBackedRateLimiter({
  redis,
  key: 'openai:gpt-4o',
  requestsPerMinute: 500,
});

const llm = new VernLLM({
  client: fromOpenAI(openai),
  model: 'gpt-4o',
  rateLimit: sharedLimiter,
});

RateLimiterAdapter is the exact surface the built in limiter exposes:

interface RateLimiterAdapter {
  estimate(request: WireRequest): number;
  acquire(estimatedTokens: number, signal?: AbortSignal): Promise<RateLimitAcquireResult>;
  signalRateLimit(): void;
  reactToRateLimitHint(hint: ProviderRateLimitHint | undefined): void;
}

An object satisfying this shape is used exactly as given, never wrapped or reconstructed. The same instance can be shared across the primary and a fallback target on purpose, when two targets genuinely draw on one provider account's real ceiling, see Bulkhead isolation with fallback above for the opposite, more common case of keeping each target's capacity separate.

Options reference

See Configuration for how rateLimit sits alongside every other constructor option.