AI providers

Purpose. One interface every AI feature speaks, and adapters for the providers a customer can point it at. The customer supplies the inference — their own API key or a model running on their own hardware — so no message content passes through us and no inference cost lands in ours. AI is off until an admin configures a provider.

Which providers can I use?

Provider	Transport	Endpoint	Key	Local-only	Verified
Anthropic (Claude)	Anthropic Messages API	`https://api.anthropic.com`	required	no	yes
OpenAI	chat-completions	`https://api.openai.com/v1`	required	no	yes
Google Gemini	chat-completions	`https://generativelanguage.googleapis.com/v1beta/openai/`	required	no	not yet
xAI (Grok)	chat-completions	`https://api.x.ai/v1`	required	no	not yet
Meta Llama API	chat-completions	`https://api.llama.com/compat/v1/`	required	no	not yet
Groq	chat-completions	`https://api.groq.com/openai/v1`	required	no	not yet
vLLM (self-hosted)	chat-completions	yours	optional	yes	not yet
Ollama (local)	Ollama	`http://localhost:11434`	none	yes	not yet
Other OpenAI-compatible	chat-completions	yours	optional	yes	never

There are only three transports; most vendors above speak the middle one. The list is not implied by that fact — it is written down in AI_PROVIDER_CATALOGUE (packages/core/src/ai.ts), which is the single source the provider picker, this table and the config validator all read. A customer should never have to discover that Gemini works by guessing a base URL.

“Verified” means we ran it. A row says not yet until every AI feature has been run against that provider. Everything else is compatible: it speaks a protocol we implement and is expected to work, and the UI labels it that way. Compatibility is not a guarantee, and the catalogue never claims otherwise on its own. Earning the flag is mechanical — point packages/ai/src/live.test.ts at the provider (see below), watch the contract pass, flip verification on its entry.

Anthropic and OpenAI are verified as of 2026-08-31 — claude-haiku-4-5 and gpt-4o-mini. Each has passed the provider contract (5/5), the prompt evaluation — 11/11 for Anthropic and 12/12 for OpenAI, as the catalogue entries in packages/core/src/ai.ts record — and every AI feature: thread summary streamed, explain-this-email, smart replies, all four compose actions, and the admin copilot including multi-step tool use, a stored proposal and its approval.

Ollama has had exactly the same treatment and its row still says not yet, on purpose. What a local runtime can do belongs to the weights somebody pulled, not to the server: gemma3:4b has no tool support at all, deepseek-r1:8b claims it and never calls one, and llama3.2:1b is intermittent on tools that take arguments. One good run does not generalise to “Ollama works” the way it does for a hosted API, so the flag stays off and the run-book carries a measured per-model table instead.

The third transport is where cancellation broke. stream() in all three adapters checked the abort signal only after the response loop finished. Anthropic’s SDK iterator ends promptly on abort and Ollama’s chunks are small enough to hide the gap, so both passed; OpenAI’s SSE reader cheerfully yielded another token after the caller had cancelled, which means a customer billed for output they stopped. The check now runs at the top of every iteration in all three. Two transports agreeing is not a contract — the third is what tested it.

Model names are read from the endpoint, not from us. Model ids move faster than our release cycle, so the catalogue carries at most a suggestion; the authoritative list is whatever the endpoint returns when the admin tests the connection (ping() → /v1/models, /models or /api/tags). A configured model the endpoint does not have is reported at that point rather than at the first real call.

Behaviour.

The interface (AiProvider): ping (credentials and reachability, spends no output tokens), complete, stream, and embed where the provider has embeddings. How a provider without them refuses depends on which class it is, and both shapes are live: embed is genuinely absent on the Anthropic provider, so a caller gets a type error rather than a runtime one — but the OpenAI-compatible class implements it for every entry in its catalogue and throws AiError('unsupported') when that entry declares no embeddings endpoint, which is the case for grok, llama and groq. Ollama’s embed always sends the request (POST api/embed): its catalogue entry declares embeddings: true because Ollama-the-server has the endpoint, and a pulled model without embeddings is refused by Ollama itself, which the error taxonomy reports as unsupported. Only ping().modelCapabilities.embeddings reflects the model. So a caller that needs embeddings must check capabilities rather than test whether the method exists — capabilities.embeddings for the hosted entries, and the model’s own value from ping() for Ollama; only Anthropic is answerable by the method test. stream yields text deltas, then each tool call whole once its arguments have finished arriving, then exactly one done carrying the stop reason and token usage.
Cancellation is first-class. Every call takes an AbortSignal, and a cancelled stream throws AiError('cancelled') rather than emitting done — a caller cannot mistake an aborted answer for a finished one. Cancelling and timing out are separate outcomes even though both arrive as an abort.
Errors are normalised to one taxonomy — auth, billing, rate_limit, context_length, unsupported, network, timeout, cancelled, local_only_refused, provider — with a retryable flag. Three providers fail in three shapes, and every surface has to tell “your key is wrong” (an admin fixes it) from “the model is busy” (retry) from “this thread is too long” (send less). Nothing above the provider layer parses a vendor’s error body.

billing is the one kind that cannot be read from the status code, and it was added after the first live run rather than designed in. An account with a valid key and no credit is refused by Anthropic with 400, by OpenAI with 429 insufficient_quota, and by others with 402 — so the body is checked before the status. Getting this wrong is not cosmetic: as a rate_limit it is retryable, and the caller would retry an account that can never succeed; as auth it would send an admin to replace a key that works.
Prompt caching is Anthropic-only today. A caller marks the end of a cacheable prefix with cache: true on a text block; providers without caching ignore it, so callers can always set it. The API allows four breakpoints — beyond that they are dropped rather than sent, because failing a request over a caching hint would be worse than caching less.
Local-only enforcement refuses any endpoint that is not on a private network. An IP literal is judged as it stands. localhost, a name under a suffix reserved for local networks (.local, .localhost, .internal, .intranet, .lan, .home.arpa) and a single-label name such as ollama are trusted by name, with no lookup. Every other name is resolved, and every address it answers with must be private (loopback, RFC 1918, link-local, unique-local, CGNAT); one public address is enough to refuse. So a trusted name that is in fact a CNAME to a hosted API, ollama.internal pointed at one, is not caught. Hosted providers are refused at construction, before a request exists. The check wraps the adapter rather than living inside it, so a fourth transport cannot forget it.
Compatibility shim. “OpenAI-compatible” is a spectrum. When an endpoint rejects a request naming an optional parameter we sent, that parameter is dropped and the request retried, once per quirk, and the quirk is remembered for the life of the provider. Two are known to differ in the wild: max_tokens, which OpenAI’s newer models renamed max_completion_tokens, and stream_options, which some self-hosted servers reject. Without the shim each looks like a hard failure on an endpoint that otherwise works.
Ollama differences the adapter absorbs: newline-delimited JSON instead of SSE, num_predict instead of max_tokens, and tool calls with no ids — ids are synthesised positionally. llama3.2 and llama3.2:latest are treated as the same pull.
For Ollama, the catalogue’s capabilities describe the server and the model decides. Tools, vision and embeddings belong to the weights you pulled: gemma3:4b reports ['completion','vision'] and refuses a tool call with a 400, while the catalogue entry says tools: true because Ollama-the-server supports them. So ping() asks /api/show and returns modelCapabilities for the configured model. Without it an admin configures Ollama, sees a green connection test, and only learns at call time that the copilot cannot work — a failure discovered in the wrong place, by the wrong person. Best-effort: an older Ollama has no capabilities field and a proxy may not expose the route, and knowing no more than the catalogue is never a reason to fail an otherwise good connection test. The hosted APIs do not expose anything equivalent, so they return nothing.
The audit log is metadata only (auditRecord). A record is built from a call rather than assembled at each call site, and parsed through the AiCallRecord schema, which drops anything not on the list: provider, model, feature, prompt version, outcome, error kind, token counts, latency, and a truncated SHA-256 digest of the prompt. The digest lets identical calls be correlated and a repeated failure be traced to a repeated prompt; nothing about the mail is recoverable from it. packages/ai/src/audit.test.ts asserts this by building a record from a request full of confidential text and checking that no fragment of it survives serialisation.

Configuration. Per tenant, in ai_provider_config (migration 0016): provider, model, endpoint, sealed key, key hint and localOnly. No row means AI is off, which is the state every install ships in; deleting the row is the off switch, and there is no separate enabled flag. endpoint is resolved from the catalogue at write time rather than read time, so a later change to a vendor’s default URL cannot silently redirect an existing install’s traffic. SECRET_ENCRYPTION_KEY is required — see Keys at rest. Setup steps are in the run-book ai-provider-setup.

The admin screen is /ai in the console (apps/admin/src/routes/ai.tsx), owner/admin only, with three tabs: the provider, Limits (the server-wide daily AI limit, with links to each domain’s own AI section, where the per-domain policy now lives), and the call log.

It renders the catalogue served by GET /ai/providers rather than importing AI_PROVIDER_CATALOGUE from @mailserver/core. Both would work — core is browser-safe — but fetching it means the picker and the config validator read one list that arrives together, so a deployment can never pair a new UI with an older API and offer a provider the server will reject.

Three decisions the screen makes, all of them about not lying to the admin:

Local-only filters the picker rather than validating after the fact. A hosted vendor can never satisfy localOnly, so switching it on removes those options instead of leaving them selectable and refusing the save.
Test connection tests the saved configuration, not the form, and is disabled until something is saved. The key does not come back out of the server once stored, so there is nothing for the form to send — save first, then test. The result names the failure in the provider’s own words, and says when the endpoint does not list the configured model.
The key field is write-only and blank means “keep”. A stored key is described by its last four characters only, so saving a model change cannot silently wipe a working key. When keyUnreadable is set the screen says the encryption key changed and to re-enter it — a remedy, rather than an error an admin cannot act on.

Per-user opt-in has deliberately no control here. It is the user’s own consent and belongs to them (see ai-privacy); an admin switch for it would make the word meaningless.

The Setup Wizard offers the same form, as an optional step, so a customer who already has a key does not have to find this screen afterwards (setup-wizard). It is the same component (apps/admin/src/components/ai-provider-form.tsx) against setup-phase endpoints (GET|PUT|DELETE /setup/ai, POST /setup/ai/test, GET /setup/ai/providers), which are unauthenticated while setup is pending — there is no session yet — and answer 409 once it completes. One component rather than two is the point: a second picker would be a second place for the served catalogue, the local-only filter and the write-only key field to drift. Skipping the step writes nothing, which is what keeps “AI is off until an admin configures a provider” true of a wizard install. The answers file has the matching optional ai key (headless-install).

API (owner/admin only — choosing where mail content is processed is server-wide, not per-domain):


`GET /ai/providers`	The catalogue, so the picker and its “verified/untested” labels come from one place rather than being hard-coded in the SPA.
`GET /ai/config`	`{configured:false}`, or the configuration without the key — `hasKey`, a `keyHint` of `…` plus the key’s last four characters (`null` for a key under 8 characters), and `keyUnreadable`.
`PUT /ai/config`	Install or replace. Omitting `apiKey` keeps the stored one; `""` clears it; changing `provider` drops it.
`DELETE /ai/config`	Turn AI off.
`POST /ai/config/test`	Ping the endpoint: models advertised, whether the configured model is among them. POST because it makes an outbound call to a third party, which is not a safe method.

Keys at rest. Sealed with SecretBox from packages/db under purpose ai.provider-key, using the server’s SECRET_ENCRYPTION_KEY from .env. The plaintext exists only inside AiService; the API never returns it and the audit log never sees it — both audit snapshots come from view(), which has never held the key. Losing SECRET_ENCRYPTION_KEY makes the stored key unreadable, and that is reported as keyUnreadable with a “re-enter your key” message rather than as a failure, because re-entering it is the actual remedy.

Changing the provider drops the stored key. An Anthropic key is meaningless to Groq, and silently carrying it across would produce an auth failure that reads like a bug rather than a missing key. Editing only the model keeps it.

Testing. Unit tests drive a stubbed fetch with recorded wire bodies, chunked at arbitrary offsets so the readers are exercised across chunk boundaries — mid-line, mid-JSON, mid-multibyte-character. Live checks in packages/ai/src/live.test.ts skip when their credentials are absent, so pnpm test stays green on a bare machine and no key is ever committed:

AI_TEST_ANTHROPIC_API_KEY=sk-ant-…  AI_TEST_ANTHROPIC_MODEL=claude-haiku-4-5 \
AI_TEST_OLLAMA_ENDPOINT=http://localhost:11434  AI_TEST_OLLAMA_MODEL=llama3.2 \
AI_TEST_OPENAI_PROVIDER=gemini  AI_TEST_OPENAI_API_KEY=…  AI_TEST_OPENAI_MODEL=… \
  pnpm exec vitest run packages/ai/src/live.test.ts

Set AI_TEST_OPENAI_PROVIDER to any chat-completions entry (gemini, grok, llama, groq, vllm, openai-compatible) to run the same contract against it. Run vitest from the root as above rather than with pnpm --filter: packages/ai defines no test script, so a filtered run against it silently does nothing. (packages/license is the one workspace package that does define one — vitest run — so the blanket claim that used to stand here was wrong, and would have sent anyone checking it to the wrong conclusion about why their filter did nothing.)

Limits & edge cases.

From the licence’s degraded stage every AI call is refused 402 license_degraded, the admin copilot’s included (ai-privacy, Licensing). POST /ai/config/test is not one of them: it reaches the provider without an ai_call_log row, since it sends no mail, and like any other administrator’s write it is refused from restricted.
Every AI feature now calls a provider. The inbox features, the prompt library and the admin copilot all ship; the privacy controls that govern them are ai-privacy. This bullet said the opposite until 2026-09-10, months after they landed.
SECRET_ENCRYPTION_KEY is required, and an existing install gains a fresh one on mailctl upgrade. That is harmless only because nothing was encrypted before it existed; the release introducing this must be the release that introduces the variable.
The worker does not get the encryption key yet. Nothing in the worker calls a provider, so it has no way to decrypt one — that changes if delivery-time AI ever moves there (post-1.0).
TOTP seeds moved onto SecretBox too, which is what writing it generically was for — sealed by 0024_totp_secrets_sealed.sql under purposes auth.user-totp and auth.mailbox-totp. Nothing in the database holds an unencrypted secret any more.
@mailserver/ai is node-only (DNS, streaming HTTP, the Anthropic SDK) and apps/admin, apps/webmail and packages/web-shared must never import it — same rule as packages/deliverability and packages/license. The catalogue and the shapes that cross the API boundary live in browser-safe packages/core (ai.ts) for exactly this reason.
GitHub Copilot is not a provider. Its API is scoped to editor and agent integrations under a GitHub identity, and Microsoft 365 Copilot exposes no third-party inference API, so there is nothing a customer could paste a key into. Recorded here so the question is not reopened from scratch.
Gemini goes through its compatibility route, which does not expose Gemini’s native context caching or safety settings. If a feature turns out to need those, a native adapter is a separate change; the abstraction supports one.
Retries are the caller’s decision — the SDK’s own retries are off. An inbox feature the user just cancelled must not keep retrying underneath, and the taxonomy already says what is worth retrying.
A model that emits invalid JSON tool arguments yields {} rather than crashing the call.