Caching

VernLLM ships three cache adapters out of the box: InMemoryCacheAdapter (exact match), NormalizedCacheAdapter (normalized/fuzzy match), and TieredCacheAdapter (two-tier local + shared cache), plus a CacheAdapter interface for bringing your own.

The built-in InMemoryCacheAdapter provides a simple in-memory cache with no external dependencies. It is useful for local development or single-process deployments; use a shared adapter such as Redis or Upstash when multiple processes need the same cache.

import { InMemoryCacheAdapter } from 'vern-llm';

const llm = new VernLLM({
  client: fromOpenAI(openai),
  model: 'gpt-4o',
  cache: new InMemoryCacheAdapter(1000), // maxSize limit; defaults to 1000
});

const result = await llm.cachedCall({
  cacheKey: `cv:${cvId}`,
  ttl: 3600,
  call: { systemPrompt, userContent },
});

Eviction

Once maxSize is exceeded, InMemoryCacheAdapter evicts an entry to make room. Two policies:

• 'fifo' (default): drops the oldest inserted entry, regardless of reads.
• 'lru': drops the entry read or written longest ago.

const cache = new InMemoryCacheAdapter(1000, 'lru');

The shorthand form on VernLLMOptions.cache covers the same thing without importing anything:

const llm = new VernLLM({
  client: fromOpenAI(openai),
  model: 'gpt-4o',
  cache: { maxSize: 1000, eviction: 'lru' },
});

There's no third in-process policy to opt into. Anything beyond fifo/lru is a different cache backend through CacheAdapter (Redis with an LRU maxmemory-policy, for example), not another built-in algorithm here.

Concurrent misses are coalesced

If several calls with the same cacheKey miss the cache at roughly the same time, only the first one (the "trigger") actually calls the underlying call(). Every other concurrent caller waits on that same in-flight call instead of starting its own, and all of them resolve (or reject) together once it settles.

This prevents a cache stampede where many simultaneous callers trigger their own expensive request before the first one has populated the cache.

Each coalesced caller has its own signal and its own usage-hook lifecycle. Aborting one coalesced caller does not cancel the shared in-flight operation for other callers; that caller's top-level signal can reject its own call before the shared operation settles. Only callers that remain active receive the shared operation's result.

Non-exact matching with resolveKey

By default, cacheKey is matched exactly, get/set receive whatever string you pass in, and concurrent calls only coalesce if their cacheKey values are identical.

If your adapter matches on something other than exact string equality (a normalized/fuzzy key, an embedding-based semantic match, etc.), implement the optional resolveKey method on your adapter:

interface CacheAdapter<T = unknown> {
  get(key: string): Promise<{ hit: boolean; value: T | null }>;
  set(key: string, value: T, ttl: number): Promise<void>;
  delete?(key: string): Promise<void>;
  resolveKey?(key: string): Promise<string>;
}

When present, resolveKey runs once at the start of cachedCall, before the cache lookup and before the in-flight coalescing check. Its return value, not the original cacheKey, is what gets used for both. This means concurrent calls that resolveKey maps to the same string share one in-flight call, the same way exact-match callers already do; without it, two calls that your adapter would consider "the same" for lookup purposes could still each trigger their own call if they happened to race.

Adapters that don't implement resolveKey are unaffected, the caller-supplied cacheKey is used as-is, exactly like before.

Built-in adapters beyond InMemoryCacheAdapter

NormalizedCacheAdapter

Wraps another adapter (defaults to InMemoryCacheAdapter) and normalizes keys with lowercase, trim, punctuation-to-space replacement, and whitespace collapsing. Normalization is applied by normalize() across get, set, delete, and resolveKey. No external dependencies or network calls.

import { NormalizedCacheAdapter } from 'vern-llm';

const llm = new VernLLM({
  client: fromOpenAI(openai),
  model: 'gpt-4o',
  cache: new NormalizedCacheAdapter(), // wraps InMemoryCacheAdapter by default
});

// These two calls share one cache entry:
await llm.cachedCall({
  cacheKey: '  What is the capital of France?  ',
  ttl: 3600,
  call: { userContent: q1 },
});
await llm.cachedCall({
  cacheKey: 'what is the capital of france',
  ttl: 3600,
  call: { userContent: q2 },
});

Pass any other adapter to normalize keys in front of it instead of the in-memory default:

const cache = new NormalizedCacheAdapter(new UpstashCacheAdapter());

See Normalized Caching for more on when this is (and isn't) enough, versus reaching for semantic matching.

TieredCacheAdapter

Composes two adapters into an L1/L2 cache: checks a fast local adapter first, falls back to a shared/slower adapter on a miss, and writes the result back into L1 so the next lookup on this process skips L2 entirely.

import { InMemoryCacheAdapter, TieredCacheAdapter } from 'vern-llm';

const cache = new TieredCacheAdapter(
  new InMemoryCacheAdapter(), // L1: fast, per-process
  new UpstashCacheAdapter(), // L2: shared across processes, slower
);

const llm = new VernLLM({ client: fromOpenAI(openai), model: 'gpt-4o', cache });

TieredCacheAdapter forwards resolveKey to whichever tier implements it, preferring L1 since get() checks L1 first. This means it composes with NormalizedCacheAdapter (or any other adapter that implements resolveKey) in either position:

// L1 normalizes; TieredCacheAdapter forwards L1's resolveKey automatically
const cache = new TieredCacheAdapter(new NormalizedCacheAdapter(), new UpstashCacheAdapter());

Deleting cached entries

Cache invalidation is application-specific, so VernLLM does not automatically decide when a cached response becomes stale. When your application knows that a cached response should no longer be used, you can remove the entry manually:

await llm.deleteCache(`cv:${cvId}`);

cachedCall: cached + retried in one call

cachedCall combines cache lookup, cache writes, and concurrent miss coalescing with the normal retry/timeout/circuit-breaker behavior from call(): the call option you pass is exactly the CallParams you'd give llm.call() directly, and on a cache miss cachedCall runs it through the same resilience pipeline before writing the result to cache.

There is no separate way to cache an arbitrary non-LLM function through VernLLM; cachedCall always wraps a call() invocation. If you need general-purpose caching or coalescing unrelated to an LLM call, reach for a dedicated caching library (e.g. async-cache-dedupe) at the application level instead.

Usage hooks

Usage hooks are applied at the cache level. reserveUsage/refundUsage no longer type-check inside the nested call options at all. CachedCallParams's call field omits them, so putting them there is a compile error, not just a mistake caught later. If a caller bypasses that (plain JS, or an explicit type cast) and sets them inside call anyway, cachedCall throws LLMError('validation') before doing anything else, rather than silently ignoring them and continuing: silently dropping a cost-control hook fails open, not safe, and a caller may never see a warning-level log. Set them at the top level instead. When reserveUsage and refundUsage are provided, they run once per caller on cache misses, including coalesced callers. The coalesced value passed to the hooks indicates whether the caller joined an existing in-flight operation.

signal can be provided at the top level to cancel that caller's cached operation lifecycle. Each coalesced caller has its own signal and usage-hook lifecycle. Aborting one coalesced caller does not cancel the shared in-flight operation for other callers. The top-level signal only controls the cached wrapper's lifecycle; pass the same signal into the nested call options as well if the underlying request itself should also be cancellable.

const result = await llm.cachedCall({
  cacheKey: `cv:${cvId}`,
  ttl: 3600,
  signal,
  call: {
    systemPrompt,
    userContent,
    signal,
  },
});

Choosing a cache key

There's no automatic key derivation from the call's contents. cacheKey is entirely up to you.

If you reuse a key across calls with different systemPrompt, schema, or model, you may receive a cached result that does not match what the new request would have produced.

A safer pattern is to include values that affect the output:

const cacheKey = `cv:${cvId}:${model}:v2`; // bump version when prompt/schema changes

Custom cache adapter

Bringing your own CacheAdapter (Redis, Upstash, or anything else), including the specific things to get right when implementing one (hit semantics, resolveKey, ttl convention, and how a broken adapter is handled) is covered as a customization topic, with a full worked example, over on Caching → Customization.