Security incident

When to use: an outbound_abuse alert; a mailbox owner reports mail they did not send, forwarding rules they did not make, or sign-ins they do not recognise; entries in the Audit log nobody can account for; a blocklisting traced to this server (The server IP is on a blocklist); any reason to think someone else has had a shell on the host or a copy of a backup. Preconditions: owner or admin access to the console; root on the host.

Work out first which of three things is compromised, because the response differs: a mailbox (its password, an app password or a webmail session), an admin-console account, or the server (shell access, .env, or a backup — every backup contains .env). When in doubt, treat it as the larger one.

1. Keep the evidence — before changing anything

Several records do not survive the containment steps below: recreating a container discards its docker logs, the mail logs are pruned after 7 days, the delivery log after 30, and the AI call log with the copilot’s tool calls and proposals after 30 (AI_LOG_RETENTION_DAYS). Take copies first; it takes minutes.

mkdir -p /root/incident && cd /root/incident
mailctl backup --out /root/incident                       # database (audit log, sessions, delivery log), all mail, .env
docker cp mailserver-postfix-1:/var/log/mail ./maillog    # Postfix and Dovecot logs, with up to 7 days of rotations
for s in api worker postfix dovecot; do docker logs mailserver-$s-1 > $s.log 2>&1; done

The backup holds .env, which is every secret on the server: keep it somewhere at least as safe as the server. What to look at, and where:

• Audit log (GET /api/v1/audit): every change made through the console or API, with the actor and their IP, newest first. Look for user.create, token.create, forwarder.create, mailbox.filters_update and mailbox.autoresponder_update (made by an administrator, or by the owner in webmail, which the action names as … by <address>: a redirect to an outside address is the classic sign of a stolen session), mailbox.password_reset, mailbox.app_password.create, mailbox.webmail_opened (an administrator opened someone’s mailbox), mailbox.export_started (a mailbox was exported), server.config.apply, auth.ban.clear and auth.login (admin-console sign-ins) you cannot account for.

• Mailbox sign-ins are in Dovecot’s log. rip= is the client’s address, and app= is the start of the hash of the app password used (empty for the account password and for webmail):

  grep -h 'Login: user=<user@example.com>' ./maillog/dovecot.log*

• What a mailbox sent: zgrep -h 'sasl_username=user@example.com'./maillog/postfix.log* gives the queue id of every message it submitted; the Delivery log, searched for the address, gives what happened to each.

• Webmail and console sessions with the address and browser that opened them:

  docker exec mailserver-postgres-1 psql -U mail -d mail -c "SELECT s.session_id, s.created_at, s.ip, s.user_agent, s.revoked_at, s.expires_at, a.email AS opened_by_admin FROM refresh_session s JOIN mailbox m ON m.id = s.mailbox_id LEFT JOIN \"user\" a ON a.id = s.on_behalf_of_user_id WHERE m.address = 'user@example.com' ORDER BY s.created_at DESC LIMIT 50"

  Every refresh writes a new row, and rows with the same session_id are one session’s chain, so a live session is a row with no revoked_at and an expires_at in the future. opened_by_admin names the administrator holding a session they opened from the console (an hour at most); it is empty for the owner’s own. For an admin account, select from refresh_session s JOIN "user" u ON u.id = s.user_id OR u.id = s.on_behalf_of_user_id and filter on u.email: that lists their console sessions and the mailboxes they opened (their own console sessions are also on Account → Where you are signed in). (-U mail -d mail are the defaults of DB_ADMIN_USER and DB_NAME.)

• Blocked sign-ins lists the addresses currently banned for repeated failed passwords (Login rate limiting & brute-force lockout) — a guessing campaign shows up there first.

2. Contain a compromised mailbox

1. Stop it. On the mailbox’s page (Domains → the domain → the mailbox → Account), turn on Suspended. A suspended mailbox cannot authenticate over IMAP, POP3 or SMTP, and every webmail session it has is revoked, so lifting the suspension later does not bring them back; an access token already issued is refused on its next request, and an open webmail tab’s new-mail stream closes within 25 seconds and its IMAP connection is dropped. Its stored mail is kept. Mail sent to it is refused while it is suspended (User unknown in virtual mailbox table, so senders get a bounce), which is the price of stopping it at once.

   Containment’s own block (Restore sending undoes it) refuses only the mailbox’s outgoing mail and leaves it reading and receiving, but only the worker sets it, on an outbound_abuse verdict with Block automatically armed; there is no button to set it by hand (Compromised-mailbox containment).

2. Drop the connections it already has. IMAP and POP3 sessions that are already open survive suspension and a password change until the client reconnects:

   docker exec mailserver-dovecot-1 doveadm who
   docker exec mailserver-dovecot-1 doveadm kick user@example.com

3. Reset its password (Reset password on the same page) and revoke every app password under Security — an app password is a separate credential, and a password reset leaves every one of them working. Revoking takes effect at the client’s next sign-in. The reset also revokes every webmail session of the mailbox, an administrator’s included.

4. Undo what the attacker set up. On the mailbox’s page, check Forwarding (a copy of all incoming mail going elsewhere), Filters (a Sieve rule can redirect or discard), and the Autoresponder. Check its Scheduled messages too: the password reset and a suspension hold every message waiting to be sent later, so none leaves on its own, but each stays in the mailbox’s Scheduled list marked Not sent until somebody sends, moves or cancels it. Before handing the mailbox back, open it in the webmail (Mailboxes) and cancel what the owner did not write, or the owner may send it from the list; or list them with docker exec mailserver-postgres-1 psql -U mail -d mail -c "SELECT s.id, s.send_at, s.envelope->>'subject' FROM scheduled_message s JOIN mailbox m ON m.id = s.mailbox_id WHERE m.address = 'user@example.com' AND s.status IN ('scheduled', 'failed')". On API tokens, revoke any token whose Send from list covers the address and that you do not recognise.

5. Give it back. Untick Suspended once the password is new, the app passwords are revoked, and the rules are clean. If containment blocked its sending, Restore sending now — after the reset, or the next sweep blocks it again. The owner can turn on two-factor sign-in from the webmail (App passwords and mailbox two-factor authentication); an administrator cannot do it for them.

3. Contain a compromised admin-console account

1. Deactivate it: Users & roles, switch the account off. New sign-ins are refused at once, and every session of the account is revoked, the mailboxes it opened on someone’s behalf included, and an access token already issued is refused on its next request. Switching it back on does not bring the sessions back. It also holds every message the account scheduled to send later from a mailbox it opened, so none leaves on its own; check the Scheduled list of each mailbox it opened (the audit log’s mailbox.webmail_opened entries) and cancel what it should not have sent. Before you do, have its owner set a new password with Forgot your password? on the sign-in page (Password reset by email). No administrator can set another administrator’s password.
2. Revoke API tokens on API tokens that the account created or that you do not recognise (token.create in the Audit log names who created each). A token keeps working until it is revoked or expires, whoever made it.
3. Review everything it could change — an admin can change anything on the server. In the Audit log, every entry with that actor. Then check directly: Users & roles for accounts nobody created; Settings → Trusted networks for networks you did not add (a network there may relay through the server without signing in, so an attacker can turn it into an open relay — Relaying & trusted networks) and for a changed Outbound relay (smarthost); forwarders and aliases on each domain; Blocked sign-ins for bans that were cleared. GET /api/v1/server/config/history has every configuration version, and Server can roll back to one.
4. If the account opened mailboxes (mailbox.webmail_opened) or exported them (mailbox.export_started), treat the mail in them as read by someone else.

4. Contain a compromised server

Assume everything in .env and the database has been read: the secrets below, every DKIM private key, the relay credentials and the AI provider key (sealed under SECRET_ENCRYPTION_KEY, which was in the same .env), and the mail itself.

1. Move to a clean host. Install mailctl on a new server, restore your most recent backup from before the compromise with mailctl restore (Backup & restore), move the licence with mailctl license transfer (Move a licence to a different machine), and retire the old host. The restore brings back the backup’s .env, so its secrets are the exposed ones: rotate them next.

2. Rotate the secrets in .env. Edit the value (a new one from openssl rand -hex 32 serves), then recreate the services that read it — recreate, not restart, because Compose reads .env only when it creates a container:

   docker compose -f /opt/mailserver/compose/compose.yaml --env-file /opt/mailserver/.env up -d <services>

   (with -f /opt/mailserver/compose/compose.ipv6.yaml after the first -f when IPV6_ENABLED=true).

   Secret	What it protects	Recreate	What rotating it costs
   JWT_SECRET	Signs the 15-minute access tokens of console and webmail sessions	api	Little, and it does not sign anybody out: sessions are refresh tokens stored in the database, and a browser holding one is issued a new access token signed with the new key. A password reset, a suspension or a deactivation revokes them (sections 2 and 3).
   SECRET_ENCRYPTION_KEY	Seals the AI provider key, every TOTP seed and recovery code, every DKIM private key, the relay password and the installed licence key	api worker	All of those become unreadable at once: every account with two-factor sign-in is refused with totp_unreadable until it is cleared, the relay password and AI key must be entered again, each domain raises dkim_unreadable until its key is rotated, and the licence reads unlicensed until the owner activates the key again. The full table and each recovery is in Backup & restore → SECRET_ENCRYPTION_KEY is part of the backup. Since the DKIM keys, relay password and AI key were exposed anyway, they are being replaced in step 3 regardless.
   DB_ADMIN_PASSWORD	The database owner role the API and worker use	api worker	Needs an ALTER ROLE as well as the .env edit; the exact commands are in Upgrade → One-time: an .env without its secrets.
   DB_DAEMON_PASSWORD	The read-only role Postfix and Dovecot use to read mailboxes and domains	api postfix dovecot	Nothing: the API sets the role’s password from .env on every boot.
   DOVECOT_MASTER_PASSWORD	The master user that opens any mailbox (webmail, filters, DMARC ingest, migrations); accepted only from inside the stack	dovecot api worker	Nothing: Dovecot writes its master-user file at start.
   SRS_SECRET	Signs the rewritten return paths of forwarded mail (SRS — sender rewriting for forwarded mail)	postfix	Bounces of mail forwarded in the last few days no longer find their way back.
   MIGRATION_SECRET_KEY	Seals the source-server credentials of migrations	api worker	A migration in progress needs its source credentials entered again. Change the source servers’ passwords too: those credentials were readable.

   Upgrade → One-time: secrets generated before the CSPRNG fix covers the same rotation in more depth.

3. Replace what the secrets protected, at its source:

   • DKIM keys — rotate every domain’s key and publish the new record (Rotate a domain’s DKIM key). A stolen key lets anyone sign mail as your domains.
   • Relay credentials — revoke them at the relay provider, create new ones, enter them on Settings → Outbound relay.
   • AI provider key — revoke it at the provider and enter a new one under AI.
   • Passwords — every administrator sets a new one through Forgot your password?; reset every mailbox’s password and revoke every app password and API token (sections 2 and 3). Password hashes were in the database.
   • A certificate you supplied yourself (tls.mode byo) — its private key was on the host; have it reissued and install the new one (Bring your own TLS certificate).

5. Check for outbound abuse

A compromised mailbox or server is usually used to send spam, and the consequences outlast the incident.

• Dashboard: outbound_abuse, rate_limit_burst and dnsbl alerts, and Server reputation.
• Delivery log, direction outbound, for the period: senders with many reputation or hard results.
• Mail queue: messages from the senders involved. Hold them rather than delete, while they are evidence; delete them before anything flushes the queue (Outgoing mail is deferring).
• If the server is listed, follow The server IP is on a blocklist once the cause is closed. Consider arming Block automatically under Settings → Reputation protection → Containment so the next compromised mailbox is stopped without waiting for someone to read the dashboard.

6. Notify

The product notifies nobody about an incident: alerts appear on the dashboard only, and the only mail it sends the owners on its own is the licence warnings (Licensing). Tell the owners of affected mailboxes what happened and what you changed; tell your users to expect bounces for mail sent to a mailbox while it was suspended; and check whether the incident is a personal-data breach you are obliged to report (under the GDPR, to the supervisory authority within 72 hours of becoming aware of it). If the server sent spam, the blocklists’ removal forms ask for the cause and the fix (The server IP is on a blocklist).

Verification

The sign-in query in section 1 shows no live session you did not expect; doveadm who shows no connection from an unknown address; no new entries in the Audit log from the actor involved; the outbound_abuse alert is resolved and the queue holds nothing from the senders involved; mailctl doctor is clean after any rotation; each domain’s DNS & routing section shows the new DKIM record verified.

Rollback

Containment is reversible: untick Suspended, switch an account back on, Restore sending. Revoked sessions, app passwords and API tokens are not — the owner signs in again or a new one is issued. A rotated secret can be put back by restoring the old value in .env and recreating the same services, which also makes readable again whatever it sealed; do not do that for a secret that was exposed.