Upgrade

When to use: moving an install to a newer release. Command: mailctl upgrade --tag <version>. Preconditions: mailctl doctor passes; enough disk for a backup (database + maildir).

Steps

1. mailctl upgrade --tag 1.2.0 (default tag latest). It prints the mail queue’s depth first, before anything is pulled, written or restarted — Mail queue: 12 messages (11 deferred, 1 on hold), or could not be read, which is not the same as empty.

2. A pre-upgrade backup is written to /opt/mailserver/backups/<timestamp>/ (Backup & restore). --skip-backup skips it — not recommended.

3. IMAGE_TAG is updated in .env and any key the new release introduced is added with a freshly generated value (e.g. DOVECOT_MASTER_PASSWORD from 0.2.0 — existing keys are never changed, with four exceptions: an empty DB_DAEMON_PASSWORD, DOVECOT_MASTER_PASSWORD or SRS_SECRET gets a freshly generated value too (; see One-time below); an absent or empty DB_ADMIN_PASSWORD is written as mail, the password the database was created with (see One-time: an .env without its secrets); MACHINE_FINGERPRINT is recomputed on the host and overwritten on every upgrade, so it never appears in the Adding new .env keys line; and an empty PUBLIC_IP6 is filled with the detected address while IPV6_ENABLED=true); the compose bundle shipped with the running mailctl is re-materialised (upgrade mailctl first: npm i -g @maildock/mailctl@<version>. Not the bootstrap script — it always goes on to run mailctl install).

   Upgrade mailctl itself first, or the upgrade ships the old compose file and says it succeeded. The bundle re-materialised here comes from the installed mailctl package — materialise copies the CLI’s own assets/ — and never from the tag being upgraded to. So mailctl upgrade --tag X run with an older CLI writes the older compose.yaml, starts the stack from it, reports success, and any compose change in release X never reaches the host. It is the same shape as the re-render trap below: live in the image, absent from what the server runs. npm i -g @maildock/mailctl@<version> first, every time.

   It matters most when the compose change is the fix. From 0.3.0-rc.6 every published port carries an explicit IPv4 bind address; upgrading with an older CLI leaves those ports answering on IPv6, where the daemons see every client as the bridge gateway and trust it.

4. Registry login: LICENSE_ID from .env is exchanged for short-lived pull credentials and docker login runs. With no LICENSE_ID this is skipped (self-built images need no credentials). A refusal stops the upgrade with a non-zero exit and one of:

   • “This subscription has lapsed” — renew it; the running server is unaffected and keeps delivering mail. Nothing has been pulled or restarted, and .env is put back from the snapshot, so only the backup remains on disk.
   • “registered to a different machine” — the licence is bound to another fingerprint than the one this upgrade computed. When this host’s machine ID has changed since .env recorded it (new hardware, a renamed host, a replaced NIC), the message says so, names both IDs, and gives the fix: mailctl license transfer, then the same mailctl upgrade again. The refused upgrade puts .env back as it was, old ID included. transfer records the new ID, restarts the api on it and moves the licence (Move a licence to a different machine). With mailctl 0.4.1 that transfer answered “already here” instead, fixed in 0.4.2. There, ask support to move the licence (licensectl transfer). Without the machine-ID line, the licence really is on another server: see Move a licence to a different machine.
   • Upgrading from 0.4.0 or earlier with a licence that is already bound — it is bound to one container’s fingerprint, not the host’s, so the first upgrade to 0.4.1 or later (which sends the host’s) is refused as above. Have support unbind it (licensectl transfer --unbind) before upgrading; the first heartbeat afterwards binds it to the host.
   • “Could not reach the licence server” — a network problem, not a licence problem; retry.

5. The release is verified before anything is pulled. mailctl fetches https://get.maildock.app/releases/<tag>/manifest.json and its signature, checks the signature against a public key it ships with, then asks the registry — with a HEAD, so no layer moves — what each of the seven tags currently resolves to. Every digest must match the signed manifest. It prints what it checked either way: Release 0.3.0 verified: signature good, 7 image digests match (built … from a1b2c3d). A failure stops the upgrade before the pull, exit 6, with .env and the bundle put back from the snapshot:

   • “…in the registry is sha256:…, but the signed manifest says sha256:…” — do not work around this. The image under that tag is not the one that was released.
   • “No signed manifest is published for <tag>” — 0.2.1 and earlier are unsigned, and so is 0.3.0-rc.2 (its sign job failed); --no-verify is the deliberate way to install one.
   • “not signed by any key this mailctl trusts” — upgrade mailctl first (npm i -g @maildock/mailctl); the signing key may have been rotated past this copy.
   • It is skipped, and says so, for a registry that is not ours, for an install with no LICENSE_ID, and for --no-verify.

6. docker compose pull then up -d --wait — with --force-recreate when the network’s IPv6 setting differs from .env, because Compose would otherwise reattach unchanged containers without their service names. With IPV6_ENABLED=true the upgrade first refuses (exit 5) an engine older than Docker 27, and every compose command adds -f compose/compose.ipv6.yaml — so must a hand-run one. The API runs pending migrations on start; daemons re-apply the last live render from the mailconfig volume before starting. The mail queue survives. /var/spool/postfix is the mailqueue volume as of 2026-09-07, so recreating the postfix container keeps everything waiting, hold included. Step 1 prints the depth before it touches anything; compare it afterwards if delivery looks wrong (Mail queue (viewer, hold / release / delete)).

7. mailctl waits for the API health endpoint (--health-timeout, default 180 s) and prints the running version. If it never reports healthy, the upgrade rolls itself back — see below.

Automatic rollback

A failed health check does not leave the server down. mailctl snapshots .env, compose.yaml, compose.ipv6.yaml and the Caddyfile into /opt/mailserver/.upgrade-rollback/ before it changes anything, and reads the database’s migration count before and after. What happens next depends on whether the database moved:

• The database was not migrated — the pull failed, an image would not start, or the API died before migrating. The snapshot is put back, the stack restarts on the previous tag, and health is re-checked. Rolled back to <tag>: the API is healthy again…, exit 4. Nothing is lost and no action is needed beyond finding out why the new release failed. If the previous release does not come up healthy either, it says so and exits 5 with the restore command.
• The database was migrated — the schema is now the new release’s, and starting the previous release against it would run old code against columns it has never seen. Nothing is reverted. Exit 5, with the mailctl restore --from … command for the backup taken minutes earlier.
• A release whose boot rewrites stored data an earlier release cannot read (from 0.11.0: DKIM keys and the relay password sealed at boot) also carries a migration, so it reads as “was migrated” and is never rolled back past.
• The database could not be read — treated exactly like “was migrated”. A database that cannot be read is not a database known to be unchanged. Exit 5.

Exit codes: 0 upgraded · 3 the licence was refused at the registry · 4 upgrade failed, previous release running again · 5 a person is needed · 6 the release could not be verified against its signed manifest, and nothing was pulled. --no-rollback leaves a failure in place to debug (also exit 5). .upgrade-rollback/ is deleted as soon as the host is back in a known state — an upgrade that succeeded, a rollback that succeeded, or a licence or verification refusal that changed nothing — so a directory that is still there means the last run needed a person.

Configuration is re-rendered after an upgrade (and what to check if it is not)

An upgrade changes images. It does not re-render daemon configuration — mailctl upgrade has no render or apply step, on start the daemons re-apply the last live render from the mailconfig volume, and Caddy keeps the JSON pushed at the last apply: the edge starts with --resume, from the autosave on its caddy_config volume, so a recreated or restarted edge serves that JSON rather than the bootstrap Caddyfile (before it, a host reboot left the edge on plain HTTP with no :443 until the next apply). The other side of that: a change to the bootstrap docker/caddy/Caddyfile reaches only a fresh install (or one whose caddy_config volume is empty). Once the edge has autosaved a config it never reads the file again, so a routing change for existing servers ships in renderCaddy. Left alone, that means a fix which only changes what a template or renderCaddy emits ships in the image and never reaches the running configuration — live in the build and absent from what the server serves.

It shipped that way twice: the DKIM chown fix (a recreated rspamd container could not read its keys, so outbound mail went unsigned), and the Caddy service-host routes — MTA-STS, autoconfig, autodiscover and per-domain webmail stayed dead on a host upgraded to a build containing the fix, answering an empty 200 behind a valid certificate, because on-demand TLS is driven by SNI rather than by routes.

The API now closes this itself. Every applied configuration records the build that rendered it (server_config.rendered_by); at boot, if that differs from the running build, the API re-applies the current configuration once before the warm-up and containment ticks. A row from before the column existed reads as stale, so an install upgraded from an older build repairs itself on first boot. “The running build” is <tag>+<hash of the API bundle and templates>, so an upgrade on latest counts even though the tag does not change. You should see:

re-rendered after upgrade   version=<n> status=applied

If the log line is absent and something rendered still looks wrong, check what the running server thinks rendered its configuration:

curl -s localhost:3000/api/v1/server/config -H "authorization: Bearer <owner token>" \
  | python3 -c 'import json,sys; d=json.load(sys.stdin); print(d["version"], d["renderedBy"], d["status"])'

If renderedBy is already the running build — the image tag, +, and what docker exec mailserver-api-1 cat /app/build-id prints — the configuration is current and the fault is elsewhere. /health and mailctl print only the tag, so they cannot answer this. To force a re-apply regardless, save any change in Settings, or PUT /api/v1/server/config with the body GET returns.

Checking Caddy specifically, since its failure is silent:

docker exec mailserver-caddy-1 wget -qO- http://localhost:2019/config/ \
  | python3 -c 'import json,sys; print(json.dumps([r.get("match") for r in json.load(sys.stdin)["apps"]["http"]["servers"]["srv0"]["routes"]], indent=2))'

A bare mta-sts.* or autoconfig.* (rather than mta-sts.*, mta-sts.*.*, …) means the loaded config predates serviceHostPatterns() and the re-apply did not happen. No postmaster.<your domain> route means it predates the split of the admin panel from the webmail and the same is true. Any webmail.* wildcard means it predates the one-host change — the render now matches the exact host webmail.<your domain> — and the same is true again.

One-time: upgrading across the admin/webmail split

Releases from 2026-09-12 serve the admin panel and the mail client as two applications on two hosts. Four things change for a server that existed before it, and none of them needs downtime:

1. A webmail. host stops serving the admin console. It served the same bundle as the mail hostname, so webmail.<domain>/domains/{id} was the console; now it is the mail client and only the mail client. Nothing to do — this is the change.
2. The admin panel has a name of its own: postmaster.<your domain>. Publish an A record pointing at the host (or a CNAME to the mail hostname). It is optional: https://<hostname>/ goes on serving the console, deliberately, so that an upgrade cannot lock a server out of its own admin panel before anybody has published anything. With tls.mode=acme the certificate is issued on first request; with tls.mode=byo your certificate has to cover the name, and until it does that host presents your certificate anyway, which browsers refuse as a hostname mismatch, while the mail hostname stays trusted.
3. https://<hostname>/mail/… now redirects to https://webmail.<your domain>/… — the same path, the new host. Bookmarks and home-screen shortcuts follow it.
4. The mail client is one host, not one per domain (from 2026-09-12). If you published webmail.<some other domain this server hosts>, that name no longer routes — everybody signs in at webmail.<your own domain> instead, whichever of your domains their mailbox is in. Publish that record if you have not: it is an A record at the host, or a CNAME to the mail hostname. The old per-domain names can be deleted from DNS at your leisure; they resolve to a host that will not answer for them. Publish webmail.<your own domain> before you delete anything, or your users have nowhere to go in between.

Nothing has to be done to browsers. A device that had the webmail installed holds a service worker scoped to the whole origin; both applications drop it on first load and clear the caches it filled.

The one thing to check afterwards is that the re-apply happened at all, with the Caddy check above — this is exactly the class of change that ships in the image and never reaches a running server, because mailctl upgrade does not re-render.

Verification

mailctl doctor all ✔; GET /api/v1/health returns the new version; send/receive a test message. After an exit 4, the same checks should pass against the previous version, and /opt/mailserver/.upgrade-rollback/ should be gone.

One-time: the API no longer runs as root

From the release after 0.12.1, the api container runs as uid and gid 2525. It writes two shared volumes, mailconfig and certs. A fresh install creates both owned by 2525, and the mail daemons correct mailconfig’s handshake directories themselves at every start. The certs volume of an install created before that release stays root-owned, so the API cannot write a bring-your-own certificate there until you hand it over, once, right after the upgrade:

docker run --rm -v mailserver_certs:/certs alpine:3.24 sh -c '
  chown 2525:2525 /certs && chmod 2775 /certs &&
  if [ -d /certs/byo ]; then chown -R 2525:2525 /certs/byo && chmod 640 /certs/byo/privkey.pem; fi'
docker restart mailserver-api-1

Without it mail is unaffected, and so are Let’s Encrypt certificates (cert-sync writes those, and stays root): only writing your own certificate (at setup, or by hand into byo/) fails, with EACCES in docker logs mailserver-api-1. mailctl doctor is unaffected either way. The step changes ownership only; running it twice is harmless.

The worker runs as 2525 from the same release. It reads the Postfix and Dovecot logs through group 2525, which the postfix and dovecot containers hand it at every start and after every rotation, pruning the rotations themselves, so the maillog volume needs nothing. Its migration volume does, if a migration has ever run on this install: imapsync’s cache files in it stay root-owned. Hand it over once, right after the upgrade:

docker run --rm -v mailserver_migration:/m alpine:3.24 chown -R 2525:2525 /m
docker restart mailserver-worker-1

Without it mail is unaffected: only a migration fails, with Permission denied on its cache in the job’s error. An install that never ran one has an empty volume, which Docker hands to 2525 by itself.

One-time: secrets generated before the CSPRNG fix

mailctl up to and including 0.1.0-rc.3 generated every install secret with Math.random(), which is predictable. An install created by one of those builds should have its .env secrets replaced with fresh ones; upgrading does not rotate them, because mailctl upgrade only back-fills variables that are missing (the one key it overwrites is MACHINE_FINGERPRINT, which is not a secret).

Rotation cost differs per secret, so take them in this order:

• Free — JWT_SECRET (invalidates the 15-minute access tokens only; sessions themselves are refresh tokens kept in the database, so nobody is signed out — see Security incident to end sessions), DB_DAEMON_PASSWORD (the API sets the role from .env on boot, so: edit, then recreate api, postfix, dovecot), DOVECOT_MASTER_PASSWORD (recreate dovecot, api, worker). Recreate, not restart: Compose reads .env when it creates a container, so docker compose restart keeps the old value. Use docker compose -f /opt/mailserver/compose/compose.yaml --env-file /opt/mailserver/.env up -d <service…>, which recreates any service whose configuration changed. The same applies to every secret below.
• Has a consequence — SECRET_ENCRYPTION_KEY makes everything sealed under it unreadable at once: the stored AI provider key, every TOTP seed, the relay password (re-enter it on Server), every DKIM private key (each domain then raises a dkim_unreadable alert until its key is rotated and its record republished; it keeps signing with the key file rspamd already has), the installed licence key (the server reads as unlicensed and the heartbeat fails until the key is activated again), and every recovery code, mailbox or admin-console, stops matching, so every account with two-factor sign-in is refused with totp_unreadable until it is cleared and re-enrolled. None of it reads as corruption: the API asks for the AI key and the relay password again, the DKIM alert names its domain, and Backup & restore → SECRET_ENCRYPTION_KEY is part of the backup gives the 2FA recovery — but budget for all of it before rotating, and recreate both api and worker after the edit: the worker opens and reseals the licence key for its heartbeat. MIGRATION_SECRET_KEY unseals nothing afterwards, so rotate it only when no migration is in flight. SRS_SECRET invalidates in-flight forwarded-bounce return paths for a few days.
• Fiddly — DB_ADMIN_PASSWORD needs an ALTER ROLE against Postgres as well as the .env edit.

Take a mailctl backup first, and note that the backup contains .env: an old backup still carries the weak secrets.

One-time: an .env without its secrets

Up to 0.3.0-rc.6 the Compose bundle supplied a publicly known default for JWT_SECRET, SECRET_ENCRYPTION_KEY and MIGRATION_SECRET_KEY whenever .env lacked one. From 0.3.0 it does not: the API refuses to boot without the first two, and without the third it answers 503 migration_unavailable for migrations. An .env written by mailctl install has always carried all three, and mailctl upgrade back-fills any that are absent. So this concerns an .env that was built by hand, or that has one of these lines present but empty. mailctl upgrade leaves such a line alone.

Such a server has been running on the public values, and what it sealed is sealed under them. Setting a new value has the same costs as the rotation section above:

• JWT_SECRET — set a new random value (openssl rand -hex 32). It costs nothing: it signs only the 15-minute access tokens, and a signed-in browser is issued a new one from its stored session.
• SECRET_ENCRYPTION_KEY — a new value makes the stored AI provider key, every TOTP seed and, on a server that has started a release that seals them, every DKIM private key, the relay password and the installed licence key unreadable (the licence then reads unlicensed until the owner activates the key again). An administrator with two-factor sign-in is then refused totp_unreadable until it is cleared — by another owner or admin on the Users screen, or by a sole owner on the host with clear-broken-totp (Backup & restore, from 0.4.0) — and must enrol again; each mailbox owner with 2FA needs an administrator to clear theirs. To avoid that, set it to the old default, dev-only-encryption-key-change-me-32chars!: that is exactly what the server was already running on, and no less safe. Rotate later, when re-enrolling is acceptable.
• MIGRATION_SECRET_KEY — a new value leaves saved source credentials unreadable, so re-enter them on any migration still in progress. Leave it empty to turn migrations off.

Then recreate api and worker, as above: docker compose … up -d api worker.

DB_DAEMON_PASSWORD, DOVECOT_MASTER_PASSWORD and SRS_SECRET defaulted to the public change-me-at-install up to 0.10.0, and an .env line present but empty ran on it. From 0.11.0 the bundle supplies nothing and Postfix and Dovecot refuse to start, naming the key. Here mailctl upgrade and mailctl restore generate a fresh value for an absent or empty one and list it (Generating .env secrets that were empty or missing): nothing stores them where a new value would be refused — the api re-applies the daemon role’s password on every boot, Dovecot writes its master-user file at start — so the only cost is a new SRS_SECRET invalidating forwarded bounces in flight for a few days.

DB_ADMIN_PASSWORD is the same story with one difference. Up to 0.9.0 the bundle supplied mail whenever .env lacked it, and that is the password Postgres stored when the database was created. From 0.10.0 the bundle supplies nothing: a new database refuses to initialise and the API refuses to boot with one line naming DB_ADMIN_PASSWORD. mailctl upgrade does not generate a fresh value for this key, because the database would not accept it. When the key is absent or empty it writes DB_ADMIN_PASSWORD=mail (what the server was already running on) and says so in its output. Then rotate it:

NEW=$(openssl rand -hex 24)
docker compose -f /opt/mailserver/compose/compose.yaml --env-file /opt/mailserver/.env \
  exec -T postgres psql -U mail -d mail -c "ALTER ROLE mail PASSWORD '$NEW'"   # use your DB_ADMIN_USER and DB_NAME if you changed them
sed -i "s/^DB_ADMIN_PASSWORD=.*/DB_ADMIN_PASSWORD=$NEW/" /opt/mailserver/.env
docker compose -f /opt/mailserver/compose/compose.yaml --env-file /opt/mailserver/.env up -d api worker

Rollback

If the upgrade stopped at step 4 or 5 nothing needs rolling back: the images were never pulled, the stack was never restarted, and the snapshot has already put .env back.

Where the automatic rollback refuses (exit 5), mailctl restore --from /opt/mailserver/backups/<timestamp> restores .env (including the previous IMAGE_TAG), the database and mail, and restarts the stack at the previous version. Mail received between the backup and the restore is lost — which is why this is the one path the tool will not take on its own. Restore only when the upgrade is unusable; a mail server that is up but running a bad release is still delivering.

--skip-backup is riskier than it reads. It is the exit-5 path that needs the backup, and that is exactly the path where nothing else can help.