MTA-STS, explained by what it actually prevents

Mail between servers is usually encrypted. The word doing the work in that sentence is “usually”.

The problem, precisely

When one mail server delivers to another it offers STARTTLS, and if the other side supports it the session is encrypted. If the other side says it does not support it, the sending server delivers anyway, in the clear, because refusing would mean losing mail.

That is opportunistic TLS, and it was the right design in 1999. It has one flaw, and it is not subtle: an attacker in the path can simply strip the STARTTLS offer. Both ends believe the other does not support encryption. Both behave correctly. The mail goes over the wire in plain text, and nothing anywhere logs a problem.

What MTA-STS does about it

You publish a policy that says: mail to my domain must use TLS, and the certificate must be valid for the hostname my MX record names. A sending server that supports MTA-STS fetches your policy, caches it, and from then on refuses to deliver in the clear — because now it knows that “this server does not support TLS” is a lie.

It is three things:

1. A DNS TXT record at _mta-sts.<domain> with a policy id.
2. A policy file served over HTTPS at mta-sts.<domain>/.well-known/mta-sts.txt.
3. An HTTPS certificate for that hostname, which is what makes the policy trustworthy — the whole security of the scheme rests on the policy being fetched over a connection an attacker cannot forge.

The id in the TXT record is a cache key. Change the policy without changing the id and senders keep using the old one until their cache expires.

Start at testing

The policy has a mode. testing means “report failures but deliver anyway”. enforce means “refuse to deliver insecurely”.

Go to testing first and stay there until the reports are quiet. enforce with a certificate problem you do not know about is mail that does not arrive, and it fails in a way the sender sees and you do not.

TLS-RPT is the half people skip

MTA-STS without TLS-RPT is a policy you cannot observe. TLS-RPT is a second TXT record asking senders to send you a daily summary of what happened: how many sessions succeeded, how many failed, and why.

That is the only mechanism by which you find out that a large sender has been failing to connect securely to you for a fortnight. Without it, the failure looks exactly like nothing happening — which is the same reason testing mode exists.

Like DMARC reports, they arrive as JSON in a mailbox, and like DMARC reports almost nobody reads them. MailDock parses both and alerts on them, which is the difference between publishing a policy and operating one.

What it does not do

MTA-STS protects the hop between mail servers. It says nothing about whether the message was encrypted at rest, whether the recipient’s provider reads it, or whether the sender was who they claimed — that last one is SPF, DKIM and DMARC, which are a different checklist.