AI admin copilot

Purpose. A tool-using assistant for the person running the server, grounded in the deliverability data the server already collects. It answers from tools rather than from memory, and it can propose changes that an administrator approves — it cannot make one.

Control	Route	Shape
Readiness	GET /ai/copilot/readiness	{available, reason, provider, model, toolsSupported}
Ask	POST /ai/copilot	{conversationId, history[], message} → text/event-stream
Pending changes	GET /ai/proposals	AiProposal[]
Approve / reject	POST /ai/proposals/{id}/…	no body → AiProposal
Tool log	GET /ai/copilot/tool-calls	AiToolCallEntry[]

owner and admin only. Admin calls carry a null mailbox: the copilot reads server telemetry, not anyone’s mail, so the per-user and per-domain consent gates have nothing to say about them and the route’s role check is the control. Everything still goes through AiService.runStream(), so the copilot inherits the same call log and the same refusal path as the inbox features without knowing they exist (ai-privacy).

The write tools store a proposal; they do not preview one

Every propose* tool computes a change, writes it to ai_proposal (migration 0019), and returns an id. The administrator approves by id, with an empty request body, and the server applies exactly what it generated.

The alternative — the model returns a change, the UI shows it, the client posts it back on approval — is a confirm gate over a client-supplied payload, which is decorative. Anyone who can call apply can apply anything, and the model’s involvement stops being verifiable. It also cannot satisfy “every tool call logged”: a preview leaves no record of what was proposed or whether it was taken.

Preconditions make a late approval fail closed. A proposal records what was true when it was computed — the timestamp of the sending block, the active DKIM selector, an alert’s lastSeen — and apply re-checks it against live rows. If containment fired again in the twenty minutes an admin spent reading the card, approving would silently release the new block; instead the proposal is marked failed and says why. Proposals expire after 30 minutes, so the list never offers a button that is going to fail.

The card the administrator approves shows the proposal’s title, its target, when it expires and a summary the server wrote from the same live rows the change was computed from — the model does not write it. The stored change and precondition themselves are not rendered: the summary describes the change and usually names what the precondition pins (the time a mailbox was blocked, for instance), and the precondition is re-checked at approval rather than shown.

The conversation is held by the client

The browser keeps the turns and posts them back; the server stores tool calls and proposals, not the conversation. This looks like a trust problem and is not. Every consequential act is a proposal, and a proposal is computed server-side by running the tool for real at the moment it is made — so a client that fabricated a tool result could at most mislead the model into proposing something, and that proposal is still built from live data, still carries a precondition, and still has to be approved by a human who can read it. The caller is already an owner or admin who could call each of these endpoints directly.

Only the text of prior turns is carried across. Tool results are not replayed: they are large, they go stale, and the model can call the tool again if it still needs the answer.

Tools

Read tools are tenant-scoped and return metadata only — never message bodies.

Tool	Reads
listDomains	domains with AI on — ids, active, mailbox counts — and how many have AI off
getDeliveryLog	delivery events: recipient, sender, relay, status, DSN, server response
explainBounce	every event for one queue id, with the remote server’s words in full
checkDns	required records vs. what the last verification observed
getReputation	DNSBL listings, reverse DNS, TLS expiry
getAlerts	what the scheduled checks have raised — every AlertKind, by name
getLicenseState	state, tier, expiry, grace, usage — never the licence key
getMailbox	one mailbox: active, suspended, and whether sending is blocked, with why
getQueue	what is waiting in the Postfix queue now, with Postfix’s own delay reason

Write tools: proposeRestoreSending (lift an outbound-abuse containment block), proposeRotateDkimKey, proposeResolveAlert.

getDeliveryLog, explainBounce and getQueue are scoped to domains where AI is enabled, which settles the question the decision log left open. Admin calls skip the per-domain gate, and that is right for consent — but these tools read across domains, and their rows carry recipients. Without scoping, a domain whose admin switched AI off would still have its delivery metadata sent to a provider. Scoping is cheap; explaining why it was not would not be. The other tools honour the same switch: getMailbox and checkDns refuse a mailbox or domain with AI off, listDomains leaves such domains out and says how many, and getAlerts leaves out any alert about one (by its domain, or the address it names). Server-wide alerts, such as blocklists, rDNS, certificates and disk, name no domain and are kept. getAlerts reads the server-wide alerts the worker writes as well as the tenant’s own, as the dashboard does: until it read only the latter, and missed nearly every alert there is. The write tools too: proposeRestoreSending and proposeRotateDkimKey refuse a mailbox or domain with AI off, and proposeResolveAlert can reach the server-wide alerts getAlerts lists.

Recipient addresses do leave the server, and that is stated rather than mitigated. getDeliveryLog and explainBounce carry recipients, senders and bounce text to whichever provider the customer configured. Redacting them would leave the copilot unable to do the debugging it exists for, so the page says so under the input box and this doc says so here.

getQueue was built on 2026-09-07; getRspamdStats still has no data source. Both were originally shipped as neither built nor faked, because nothing in the control plane could reach postqueue or Rspamd’s controller. The queue half was closed by giving the postfix container a second file-handshake agent beside the config watcher (Mail queue (viewer, hold / release / delete)) — the copilot reads the same scoped snapshot the queue screen does. Rspamd is unchanged: nothing in the control plane queries its controller, so the prompt still tells the model plainly that this build cannot see live spam statistics.

The delivery log is not a substitute for either, and the prompt now says so as a specific prohibition rather than as advice: never answer “is it stuck?” from the delivery log, and never conclude a message left the server because it is absent from the queue.

The prompt

Versioned and digest-pinned like the inbox prompts (admin-copilot@7), and the tool schemas are inside the digest — a tool’s description is as much of the prompt as the system text, and renaming an argument changes what the model does just as surely as rewriting a rule.

admin-copilot@7 added disk_low, and nothing else. admin-copilot@6 added dkim_unreadable to the kinds getAlerts names, and nothing else. admin-copilot@5 changed only getAlerts’ description. It named five kinds of alert and the tool returns seven: rate_limit_burst and tls_rpt were missing. The description is now built from a Record<AlertKind, string>, so a new kind fails typecheck until the model is told about it.

There is no fenced untrusted region: the copilot reads telemetry, not mail. The failure it is written against is different — not “the model was talked into something” but “the model said something true-sounding it never looked up”. So the rules name the exact sentence the model must not write: a delivery outcome not in a row it was given, a domain that did not appear in a tool result, a DNS record checkDns did not return, an estimated count. This is the lesson this phase learned twice on gemma3:4b and recorded in ai-prompt-library: on a small model a specific prohibition holds where a principle does not. “Do not speculate” is a principle, and it did not survive either the injection fixture or the [date] placeholder.

The same rule is repeated in each propose* tool’s own result — “Stored for approval. Nothing has changed.” — because that is the last thing the model reads before it answers, and the sentence that must not appear is “I have unblocked them”.

Tool schemas are flat objects of scalars and enums: no $ref, no nesting, no oneOf. That is the intersection all three transports accept — Ollama’s tool support is a thin translation layer, and a schema Anthropic takes happily produces no call at all there. A test asserts the shape rather than leaving it as a comment.

The stable prefix is cached. The system prompt is two blocks: everything that never changes — role, style, rules, the proposal contract — with a cache breakpoint on it, then the one line that does (Server: hostname, version, N domains). Anthropic renders tools before system, so the breakpoint takes the tool schemas with it, which is most of the cached prefix. A turn re-sends that prefix on every step, up to six times; on Anthropic that is one cache write and then reads at a tenth of the price. The server line stays out of the cached region because a prefix is cached only when it matches byte-for-byte, and it changes as domains are added and versions move. Providers without caching ignore the marker (ai-providers), and they flatten the blocks with a blank line between them — which reproduces admin-copilot@2’s single string byte for byte. So @3 changed the structure of the request and not one word the model reads: the version bump is there because the pinned digest covers the whole request, which is the point of pinning it. On OpenAI and Ollama the two versions are indistinguishable. One qualification: Anthropic’s minimum cacheable prefix is model-dependent — 4,096 tokens on claude-haiku-4-5, 1,024 on claude-sonnet-5, 512 on claude-opus-5 — so on Haiku the marker is accepted and silently does nothing, and cache_read_tokens stays zero. Re-measured 2026-09-12: the prefix is ~2,130 tokens by a characters-over-four estimate — 1,415 of them the twelve tool schemas and 711 the stable system block, which is why the breakpoint on the last system block matters much more than its own share. (It read “roughly 1,900 … about 1,200 of them the tool schemas … its own 660” until then, and the code comment beside the prompt said ~1,600; both drifted as tools and rules were added, by 10–18 %.) The conclusion is unchanged in every direction that matters — still far under Haiku’s 4,096 minimum, so the marker is still silently ignored there, and still comfortably over Sonnet’s 1,024 and Opus’s 512. Two things follow for whoever touches this next: the figure moves whenever a tool is added, so re-measure rather than trusting the number, and it moves towards Haiku’s threshold, which is the one boundary a growing prompt could usefully cross. That is the model, not a bug; on Sonnet and Opus the reads show up in the usage log. On the final step the tools are withdrawn, which changes the prefix and misses the cache once per capped turn — the rare case, and the right trade.

Six steps per turn. On the last step the tools are withdrawn rather than discouraged (toolChoice is only a hint on two of the three transports) and the model is told to answer with what it has. A tool call arriving on that step is not executed: its result would be discarded a line later, so running it would spend a query and write a log row for nothing.

What is logged

Two tables, deliberately separate.

• ai_call_log — one row per provider call, so a six-step turn is six rows, because each one is separately billed. Metadata only, by the shape of its columns.
• ai_tool_call — one row per tool the server ran: the tool, its arguments, whether it succeeded, how many rows came back, and the proposal it created. This one is content-bearing — a domain name, a recipient searched for — which is exactly why it is not folded into the other. Only calls the server executed appear; anything the client fabricated was never run.

Applying a proposal is also an ordinary entry in the audit log (ai.proposal_applied), because that is what it is: an administrator changed something.

What the admin sees

• The tool transcript is inline. getDeliveryLog (days: 7) — 3 of 14 events sits above the answer it produced. An answer with no tool line above it is one to distrust, and that has to be visible rather than inferred.
• Readiness explains itself. toolsSupported: false is the case that would otherwise present as a page that works and answers that are useless: under Ollama, tool support belongs to the weights, not the server, and gemma3:4b has none. The page says so and points at the settings page.
• Proposals survive a reload. The stream announces one; the card renders from GET /ai/proposals. A stream-only copy would vanish on refresh, which for something an administrator is about to approve is the wrong half to lose.
• A turn that ran no tool says so. Some questions genuinely need none, so it is stated as a fact rather than an error — but it is never omitted. See below for the model that made this necessary.

A write tool needs a read tool that can see its precondition

getMailbox was added after the first Anthropic run, and the reason is worth keeping.

Asked to fix a mailbox that genuinely could not send, Claude checked getAlerts (empty — the block had been applied without the worker raising one) and getDeliveryLog (old successes), correctly refused to propose a fix it had not confirmed, and told the administrator there was no block. There was.

The prompt’s rule is “propose only after you have read the current state with a read tool”. For proposeRestoreSending, no read tool could see a sending block — so the rule was impossible to follow rather than merely hard, and the model’s careful reasoning produced a confidently wrong answer. getMailbox closes it, and the rules now name the specific mistake: never conclude a mailbox can send because getAlerts was empty or the delivery log shows old successes. Re-running the same question afterwards produced the right sequence — confirm the block, explain it with its real reason and timestamp, propose, say it is waiting for approval.

The general lesson is not about this tool: every propose* tool needs a read tool that can observe its own precondition, or the confirm-before-proposing rule is decoration. Worth checking when the next write tool is added.

A claimed capability is not a measured one

deepseek-r1:8b reports capabilities: ['tools', 'thinking', 'completion'] through Ollama’s /api/show. ping() therefore returns tools: true, readiness says the copilot is available, and the model then answers without calling anything. Asked “what domains are on this server?” in a browser it produced:

{ "domains": ["example.test", "another-domain.test", "yet-another.test"] }

One of those domains is real. The other two do not exist. Formatted as JSON, it reads exactly like a tool result.

This is worse than gemma3:4b, which has no tool support and is refused outright — a refusal is obvious, and this is not. Probing Ollama directly with a single trivial tool reproduced it outside our code entirely: no tool_calls in the response, and a reasoning trace inventing two tools (listdomain, listaccounts) that were never offered. So the adapter is not at fault; the model, or its Ollama chat template, does not use the tools it advertises.

The mitigation is structural rather than another prompt rule, because the prompt already forbids exactly this in as many words and it did not help. Every turn that completes without running a single tool is labelled. The tool transcript on its own is no defence here: an absent tool line is precisely what nobody notices, and the note has to be something that appears rather than something that fails to.

There is no way to know from /api/show which models actually call tools, so the honest procedure is in the run-book: ask the copilot one question with a known answer and check that a tool line appears.

Limits & edge cases

• Streamed text is accumulated, not appended — and read outside the state updater. React batches the updaters from one network chunk and runs them at render time, so an updater closing over the accumulator rewrote the paragraph before a tool call with the text that came after it. Every event looked right in isolation; the transcript was wrong. Caught by a render test, fixed, and covered.
• A cancelled turn logs the steps that completed. Stopping aborts the current provider call, which runStream records as cancelled; the earlier steps in that turn are already logged as ok.
• A tool failure is data, not an exception. It goes back to the model as an error result and the turn continues — a dead stream would leave the admin with no answer and no reason for it.
• checkDns reports the last verification, with its timestamp, and says so in words. It does not run a lookup. A model handed a timestamp it did not ask for will use it, so the tool states that the results are from that run rather than from now.
• explainBounce says “no events were recorded” in a sentence, not by returning an empty array. “No rows” and “delivered fine” are the two readings a model picks between and only one of them is true.
• The whole loop is verified against all three transports — claude-haiku-4-5, gpt-4o-mini and llama3.2:1b. On each: a tool call runs and its result reaches the answer; a propose* tool stores a proposal and changes nothing; approval by id applies it and lands in the audit log. Concretely with llama3.2:1b: a tool call runs and its result reaches the answer; a propose* tool stores a proposal and changes nothing; approval by id lifts the block and lands in the audit log; and an approval given after the world moved is refused with ai_proposal_stale, leaving the mailbox blocked and the proposal failed.
• Tool arguments arrive as strings, whatever the schema says. A live run produced {"days": "7", "limit": "50", "includeResolved": "false"}. Read strictly — typeof v === 'number', v === true — limit silently fell back to the default 20 and includeResolved could never be true: the model asks for one thing and quietly gets another, which is the worst shape a bug can take here because the answer still looks right. Arguments are now coerced rather than rejected, and two integration tests assert the behaviour (a ten-day-old bounce found with days: "30" and missed by the default) rather than what got written down.
• A small model is intermittent on tools that take arguments. llama3.2:1b called the no-argument listDomains reliably, and on one attempt emitted proposeRestoreSending as malformed text — {"name":"proposeRestoreSending","" "parameters":{…}} — which produced no tool call and no proposal, correctly. It succeeded on the retry. This is the model, not the transport, and it is why the run-book says to qualify one.
• Prompt caching was measured before it was wired up. Against Anthropic (claude-haiku-4-5), the first verification pass sent 26,386 input tokens across nine copilot calls with zero cache reads, because admin-copilot@2 built system as one plain string. admin-copilot@3 marks the stable prefix (see The prompt). That figure has not been re-measured: no Anthropic credentials were available to the session that made the change, and on Haiku it would not move anyway — the prefix is under that model’s 4,096-token caching minimum. The place to look is the cache_read_tokens column of ai_call_log on a Sonnet- or Opus-configured server.
• toolsSupported: true means the model claims tool support, not that it will use it. Readiness stays a cheap capability read rather than a billed inference call; the ungrounded-turn label is what covers the difference at the point it matters.