VernLLMVernLLM
Adapters

Custom Providers - fromFetch

Raw HTTP escape hatch for providers with no SDK

For a provider with no SDK, or where pulling one in isn't worth it, fromFetch is a raw HTTP escape hatch; supply the URL, headers, and two small mapping functions, and retries/timeouts/circuit-breaker/JSON handling all still apply.

fromfetch-basic-setup.ts
import { VernLLM, fromFetch } from 'vern-llm';

const llm = new VernLLM({
  client: fromFetch({
    url: 'https://api.example.com/v1/generate',
    headers: () => ({ Authorization: `Bearer ${process.env.EXAMPLE_API_KEY}` }),
    mapRequest: (params) => ({
      model: params.model,
      prompt: params.messages.map((m) => m.content).join('\n\n'),
      max_tokens: params.max_tokens,
    }),
    mapResponse: (json) => ({
      content: json.output,
      usage: { promptTokens: json.usage?.input, completionTokens: json.usage?.output },
    }),
  }),
  model: 'example-model-v1',
});

Non-2xx responses throw with .status set, so nonRetryableStatus still fails fast on 401/403 as usual. The thrown error also carries .headers (the response's Headers-like object), so retry logic can read things like Retry-After.

For GET/HEAD requests, the adapter skips attaching a body and a Content-Type header, since those methods don't support a request body.

fromFetch supports tool calling. mapRequest receives params.tools/params.tool_choice like any other adapter, and mapResponse can return a toolCalls array alongside (or instead of) content to surface the model's tool requests through CallWithToolsResult. See Tool calling below.

Tool calling

Forward params.tools/params.tool_choice into your provider's own request shape inside mapRequest, and return a toolCalls array from mapResponse when the provider's response includes one. Each entry needs id, name, and arguments already JSON-encoded as a string, the same wire format every other adapter produces, VernLLM parses (and validates, if argumentsSchema was set) that string internally:

fromfetch-tool-calling.ts
import { VernLLM, fromFetch } from 'vern-llm';

const llm = new VernLLM({
  client: fromFetch({
    url: 'https://api.example.com/v1/generate',
    headers: () => ({ Authorization: `Bearer ${process.env.EXAMPLE_API_KEY}` }),
    mapRequest: (params) => ({
      model: params.model,
      prompt: params.messages.map((m) => m.content).join('\n\n'),
      max_tokens: params.max_tokens,
      tools: params.tools?.map((t) => ({
        name: t.function.name,
        description: t.function.description,
        parameters: t.function.parameters,
      })),
      tool_choice: params.tool_choice,
    }),
    mapResponse: (json: unknown) => {
      const body = json as {
        output: string;
        usage?: { input?: number; output?: number };
        tool_calls?: Array<{ id: string; name: string; arguments: unknown }>;
      };

      return {
        content: body.output,
        usage: { promptTokens: body.usage?.input, completionTokens: body.usage?.output },
        toolCalls: body.tool_calls?.map((tc) => ({
          id: tc.id,
          name: tc.name,
          arguments: JSON.stringify(tc.arguments),
        })),
      };
    },
  }),
  model: 'example-model-v1',
});

content may be empty or omitted on a pure tool-call turn, don't assume mapResponse always has text to return. toolCalls can be omitted (or left undefined) entirely on calls that didn't request a tool; an empty array is treated identically to undefined, so there's no need to special case it either way.

For stream: true, tool-call deltas go through the existing mapStreamEvent seam, see Streaming below, there's no separate config for streaming vs non-streaming tool calls.

Streaming

stream: true requires two additional config fields: mapStreamEvent, which turns one parsed stream event into zero, one, or more WireStreamChunks, and optionally requestStream, which opens the streaming HTTP request and defaults to native fetch. Without mapStreamEvent, stream: true throws a clear LLMError('validation') rather than a confusing runtime failure or a silently empty stream.

requestStream is only optional when request is left at its default too. If you've set a custom request transport, stream: true also requires requestStream (it needs an async-iterable byte stream, which request's buffered response has no way to provide) or fromFetch throws LLMError('validation') instead of silently falling back to native fetch.

By default, fromFetch assumes the provider frames its stream as Server-Sent Events, data: ... blocks separated by a blank line, with the [DONE] sentinel honored, which covers the large majority of LLM providers' streaming endpoints:

fromfetch-streaming-sse.ts
import { VernLLM, fromFetch } from 'vern-llm';

const llm = new VernLLM({
  client: fromFetch({
    url: 'https://api.example.com/v1/generate',
    headers: () => ({ Authorization: `Bearer ${process.env.EXAMPLE_API_KEY}` }),
    mapRequest: (params) => ({
      model: params.model,
      prompt: params.messages.map((m) => m.content).join('\n\n'),
      max_tokens: params.max_tokens,
      stream: true,
    }),
    mapResponse: (json) => ({
      content: json.output,
      usage: { promptTokens: json.usage?.input, completionTokens: json.usage?.output },
    }),
    mapStreamEvent: (event) => {
      const parsed = event as { delta?: string; done?: boolean };

      if (parsed.delta) {
        return { type: 'text-delta', delta: parsed.delta };
      }

      return undefined; // skip events that carry nothing VernLLM needs
    },
  }),
  model: 'example-model-v1',
});

A :-prefixed comment line, the mechanism providers use for SSE keep-alive pings, is recognized automatically under the default SSE framing and resets VernLLM's chunkIdleTimeoutMs clock. It never reaches mapStreamEvent, there's nothing to configure for this to work. See Per-chunk idle timeout.

parseSseStream and the SSE_PING sentinel yields for comment-only frames, the same parser and ping detection fromFetch uses internally under the default SSE framing, are exported from vern-llm too. Useful if a custom parseStreamFrames still wants VernLLM's own SSE parsing (and ping handling) rather than reimplementing it.

mapRequest receives the same params object for both create and createStream, mapRequest itself does not change based on whether the call is streaming. If your provider's streaming endpoint needs a different request shape (e.g. a stream: true flag in the body), check params.stream inside mapRequest yourself.

Non-SSE stream framing

If your provider frames its stream differently, newline-delimited JSON (NDJSON) with no SSE envelope, for example, override parseStreamFrames to split the raw bytes into events yourself:

fromfetch-streaming-ndjson.ts
import { VernLLM, fromFetch } from 'vern-llm';

const llm = new VernLLM({
  client: fromFetch({
    url: 'https://api.example.com/v1/generate',
    headers: () => ({ Authorization: `Bearer ${process.env.EXAMPLE_API_KEY}` }),
    mapRequest: (params) => ({ ...params, stream: true }),
    mapResponse: (json) => ({ content: json.output }),
    parseStreamFrames: async function* (chunks) {
      const decoder = new TextDecoder();
      let buffer = '';

      for await (const chunk of chunks) {
        buffer += typeof chunk === 'string' ? chunk : decoder.decode(chunk, { stream: true });

        const lines = buffer.split('\n');

        buffer = lines.pop() ?? '';

        for (const line of lines) {
          if (line.trim()) yield JSON.parse(line);
        }
      }

      buffer += decoder.decode(); // flush any held-back trailing bytes

      if (buffer.trim()) yield JSON.parse(buffer);
    },
    mapStreamEvent: (event) => {
      const parsed = event as { text?: string };

      return parsed.text ? { type: 'text-delta', delta: parsed.text } : undefined;
    },
  }),
  model: 'example-model-v1',
});

A different transport for streaming

requestStream swaps out the streaming transport the same way request does for non-streaming requests, defaulting to native fetch. Unlike request, which resolves to a fully-buffered ResponseLike, requestStream resolves to an AsyncIterable of progressively-arriving Uint8Array or string chunks. axios's Node Readable in responseType: 'stream' mode already satisfies this with no extra glue code:

fromfetch-streaming-custom-transport.ts
import axios from 'axios';
import { VernLLM, fromFetch } from 'vern-llm';

const llm = new VernLLM({
  client: fromFetch({
    url: 'https://api.example.com/v1/generate',
    headers: () => ({ Authorization: `Bearer ${process.env.EXAMPLE_API_KEY}` }),
    mapRequest: (params) => ({ ...params, stream: true }),
    mapResponse: (json) => ({ content: json.output }),
    requestStream: async (url, init) => {
      const res = await axios.request({
        url,
        method: init.method,
        headers: init.headers,
        data: init.body,
        signal: init.signal,
        responseType: 'stream',
      });

      return res.data; // a Node Readable, already async-iterable
    },
    mapStreamEvent: (event) => {
      const parsed = event as { text?: string };

      return parsed.text ? { type: 'text-delta', delta: parsed.text } : undefined;
    },
  }),
  model: 'example-model-v1',
});

See Streaming for the full stream: true contract that applies once chunks reach VernLLM, independent of how fromFetch itself is configured.

mapStreamEvent can return tool_call_delta chunks too ({ type: 'tool_call_delta', index, id?, name?, argumentsDelta?, complete? }), the same WireStreamChunk shape every other adapter emits for tool calls. complete: true marks argumentsDelta as the whole set of arguments in one shot rather than a fragment, useful if your provider (like Gemini) can't stream tool-call arguments incrementally. There's nothing fromFetch-specific to configure beyond mapping your provider's own tool-call events into that shape.

Bringing your own HTTP client

By default fromFetch uses the global fetch. Pass request to swap in axios, node-fetch, undici, or anything else, as long as it resolves to a Response-compatible object (ok, status, headers.get(name), text(), json()):

fromfetch-custom-request.ts
import axios from 'axios';
import { VernLLM, fromFetch } from 'vern-llm';

const llm = new VernLLM({
  client: fromFetch({
    url: 'https://api.example.com/v1/generate',
    headers: () => ({ Authorization: `Bearer ${process.env.EXAMPLE_API_KEY}` }),
    request: async (url, init) => {
      const res = await axios.request({
        url,
        method: init.method,
        headers: init.headers,
        data: init.body,
        signal: init.signal,
        validateStatus: () => true, // let fromFetch handle non-2xx itself
      });

      return {
        ok: res.status >= 200 && res.status < 300,
        status: res.status,
        headers: { get: (name) => res.headers[name.toLowerCase()] ?? null },
        text: async () => JSON.stringify(res.data),
        json: async () => res.data,
      };
    },
    mapRequest: (params) => ({
      model: params.model,
      prompt: params.messages.map((m) => m.content).join('\n\n'),
      max_tokens: params.max_tokens,
    }),
    mapResponse: (json: unknown) => {
      const body = json as { output: string; usage?: { input?: number; output?: number } };

      return {
        content: body.output,
        usage: { promptTokens: body.usage?.input, completionTokens: body.usage?.output },
      };
    },
  }),
  model: 'example-model-v1',
});

AWS Bedrock

Use the fromBedrock adapter with AWS Bedrock Converse API

Development

Scripts, test layout, and contribution guidelines for VernLLM

On this page

Tool callingStreamingNon-SSE stream framingA different transport for streamingBringing your own HTTP client