Webmail search and list performance

When to use: searches in the webmail feel slow, return nothing they should return, or a user reports that older mail is not searchable; after a mailctl restore, which deliberately arrives without a search index. Preconditions: shell access to the host running the Compose stack. Every command below runs against the dovecot container.

Full-text search is fts_flatcurve (Xapian), configured in the rendered dovecot.conf and enabled on every install. Each mailbox has its own index in a fts-flatcurve directory beside its other index files under ~/Maildir. The index is derived data — it is excluded from mailctl backup and can be deleted and rebuilt at any time.

Steps

1. Confirm the plugin is actually loaded

docker exec mailserver-dovecot-1 doveconf -n | grep -E 'mail_plugins|fts'

mail_plugins must contain fts fts_flatcurve and the plugin block must show fts = flatcurve. If it does not, the running configuration predates the feature: re-apply the current configuration (Settings → the section → Validate & apply, or PUT /api/v1/server/config with the current config) so the daemon is re-rendered, then check again.

2. Look at the index for the mailbox that is slow

docker exec mailserver-dovecot-1 doveadm fts-flatcurve stats -u <user@domain> 'INBOX'
docker exec mailserver-dovecot-1 doveadm fts-flatcurve check -u <user@domain> '*'

stats prints messages= and last_uid= per mailbox — if messages is far below the real message count, the mailbox is only partly indexed and searches are falling back to a scan for the remainder (which is why they are slow, not wrong: fts_enforced = no). check reports errors= per mailbox; anything non-zero means a damaged shard, so go to step 4.

3. Build or complete the index

# one user, every mailbox
docker exec mailserver-dovecot-1 doveadm index -u <user@domain> '*'
# every user on the server (do this after a restore; it is I/O heavy)
docker exec mailserver-dovecot-1 doveadm index -A '*'

New mail is indexed at delivery (fts_autoindex = yes), so this is only needed for mail that arrived before the feature existed, mail restored from a backup, or after deleting an index. Run it out of hours on a busy server: it reads every message.

4. Rebuild a damaged or bloated index

docker exec mailserver-dovecot-1 doveadm fts-flatcurve remove -u <user@domain> '*'
docker exec mailserver-dovecot-1 doveadm index -u <user@domain> '*'

remove deletes the Xapian databases; the following index rebuilds them. Nothing in the mailbox is touched — only derived data. Searches keep working throughout, unindexed and slower.

Indexes accumulate shards as mail arrives and are compacted automatically (fts_flatcurve_rotate_count = 5000, fts_flatcurve_optimize_limit = 10). To force it now:

docker exec mailserver-dovecot-1 doveadm fts-flatcurve rotate -u <user@domain> 'INBOX'

5. If the list, not the search, is slow

Search and listing are different paths. The list is served by the API’s pooled IMAP bridge, one connection per signed-in mailbox, closed after 10 idle minutes — the first request after that pays a reconnect of a few hundred milliseconds. A list that is consistently slow for one user usually means an enormous folder: the webmail asks for one page at a time — the mailbox’s own page size (settings.messagesPerPage, default 50, 10–200) — and the API fetches the envelope, flags, size, body structure and References header per row, then each row’s preview line in one PREVIEW fetch where Dovecot offers it, so cost scales with the page, not the folder. Three things add to it: an inbox category tab is a second SEARCH over the folder; with pinned at top on (the default), the unfiltered list also asks for up to 50 pinned messages beside the page; and the header search (folder=*) runs the search in every folder but Trash and Junk, one after another. A list that is slow for everyone is the API container, not IMAP — check docker stats and the API logs.

Verification

Search for a word that appears only in the body of a known message, from the webmail or the API:

curl -s -H "authorization: Bearer <token>" \
  'http://localhost:3000/api/v1/mail/messages?folder=INBOX&q=<body-word>' | head -c 300

The message comes back, and doveadm fts-flatcurve stats for that mailbox shows messages= matching the mailbox’s message count. The Compose smoke test asserts exactly this (a body-only token that appears in no subject or address).

Rollback

To disable full-text search without rebuilding images, the backend has to come out of the rendered configuration — it is not a runtime toggle. Searches still work in every case: with fts_enforced = no, a mailbox with no index degrades to Dovecot’s brute-force scan, which is how search behaved before the full-text index shipped. To reclaim the disk instead, doveadm fts-flatcurve remove -A '*' deletes every index; they rebuild on the next delivery or doveadm index.