Providers and prompt cache

Flary owns the provider-neutral session contract. The host owns which account the current user or tenant is allowed to use.

Supported adapter shapes include OpenAI-compatible endpoints, Anthropic Messages, Cloudflare AI Gateway, Google, Moonshot, and provider-specific subscription connections. A provider change keeps visible Flary history but starts a new provider cache lineage.

Configure an adapter

import {
  AnthropicMessagesAdapter,
  CloudflareAIGatewayAdapter,
  OpenAICompatibleAdapter,
  ProviderAdapterRegistry,
} from "flary/providers";

const providers = new ProviderAdapterRegistry({
  adapters: [
    new OpenAICompatibleAdapter({
      id: "openai",
      baseUrl: "https://api.openai.com/v1",
      apiKey: env.OPENAI_API_KEY,
    }),
    new AnthropicMessagesAdapter({
      id: "anthropic",
      apiKey: env.ANTHROPIC_API_KEY,
    }),
    new CloudflareAIGatewayAdapter({
      accountId: env.CLOUDFLARE_ACCOUNT_ID,
      gatewayId: env.CLOUDFLARE_GATEWAY_ID,
      apiToken: env.CLOUDFLARE_API_TOKEN,
    }),
  ],
});

const adapter = providers.resolve({
  provider: "anthropic",
  model: "claude-sonnet-4-5",
});

The normalized request supports text and image content, tools, JSON output, reasoning effort, verbosity, token limits, and provider parameters. Adapters translate it into the selected provider API and return normalized completion or stream events.

Credential precedence

For a user-owned subscription or a tenant BYOK connection, resolve the credential in trusted code:

1. The current user’s authorized subscription connection.
2. An authorized tenant BYOK connection.
3. A managed deployment credential.
4. A typed provider_credentials_missing error.

The admitted credential connection ID, source, billing mode, provider, and credential version are stored with the submission. Recovery uses that exact credential. It does not silently switch to an environment key after a restart.

Tokens and authorization results must not appear in model input, events, transcripts, logs, or API responses. Use Flary’s vault contracts or the host’s encrypted secret store.

Subscription connections

The host presents the login UI. Flary exposes provider-neutral OAuth operations for hosted device authorization, local browser callbacks, code completion, refresh, disconnect, and encrypted credential handoff.

const login = await client.startProviderOAuth({
  provider: "anthropic",
  loginMethod: "authorization_code",
});

showLoginInstructions(login);

const connection = await client.completeProviderOAuth(login.id, {
  authorizationCode: codeFromProvider,
});

Treat a subscription connection as user-owned unless the host has an explicit organization grant. Track billing mode as subscription, byok, or managed. Subscription cost is unknown when the provider does not report a price; do not record it as zero.

Cache retention

Set cache policy on the admitted model selection:

const model = {
  provider: "anthropic",
  model: "claude-sonnet-4-5",
  cacheRetention: "long",
};

Flary stores one opaque cache affinity per provider lineage. It survives a Worker restart and Durable Object eviction. Anthropic uses cache-control breakpoints for stable instructions, tools, and conversation prefixes. OpenAI-compatible routes use the stable cache key or session transport that the route supports. The effective retention can be shorter than requested.

Normalized usage reports requested policy, effective policy, provider, model, cache hit, read tokens, and write tokens. A model switch keeps visible history but does not reuse a cache lineage from another provider.

Provider recovery rules

• Persist the admitted provider and credential version before starting work.
• Refresh one credential under the D1 refresh lock.
• Reuse native provider continuation state only for the same provider.
• Treat revoked or deleted credentials as a typed recovery failure.
• Never fall back to a different credential for an admitted turn.
• Test cache reads after a Durable Object restart and after provider switching.