How to detect automatically generated emails
Written on 22 Dec 2018
When you send out an autoreply from an email system you want to take care to not send replies to automatically generated emails. At best, you will get a useless delivery failure. At worst, you will get an infinite email loop and a world of chaos.
Turns out that reliably detecting automatically generated emails is not always easy. Here are my observations based on writing a detector for this and scanning about 100,000 emails with it (extensive personal archive and a company archive); it’s been working well for several years (I originally wrote this in 2015).
Auto-submitted
header
Defined in RFC 3834.
This is the ‘official’ standard to indicate your message is an autoreply. You
should not send a reply if Auto-Submitted
is present and has a value other
than no
.
X-Auto-Response-Suppress
header
Defined by Microsoft
This header is used by Microsoft Exchange, Outlook, and perhaps some other
products. Many newsletters and such also set this. You should not send a
reply if X-Auto-Response-Suppress
contains DR
(“Suppress delivery
reports”), AutoReply
(“Suppress autoreply messages other than OOF
notifications”), or All
.
List-Id
and List-Unsubscribe
headers
Defined in RFC 2919
Most of the time you don’t want to send autoreplies to mailing lists or newsletters. Pretty much all mailing lists and most newsletters set at least one of these headers. You should not send a reply if either of these headers is present; the value is unimportant.
Feedback-ID
header
Defined by Google.
Gmail uses this header to identify newsletters and uses it to generate statistics/reports for owners of those newsletters. You should not send a reply if this headers is present; the value is unimportant.
Non-standard ways
The above methods are well-defined and clear (even though some are non-standard). Unfortunately some email systems do not use any of them :-( Here are some additional measures.
Precedence
header
Not really defined anywhere, mentioned in RFC 2076 where its use is discouraged (but this header is commonly encountered).
Note that checking for the existence of this field is not recommended, as some
mails use normal
and some other obscure values (this is not very common
though).
My recommendation is to not send a reply if the value case-insensitively
matches bulk
, auto_reply
, or list
.
Other obscure headers
A collection of other somewhat obscure headers I’ve encountered. I would recommend not sending an autoreply if one of these is set. Most mails also set one of the above headers, but some don’t (this is not very common).
-
X-MSFBL
; can’t really find a definition (Microsoft header?), but I only have autogenerated emails with this header. -
X-Loop
; not really defined anywhere, and somewhat rare, but sometimes it is set – most often to the address that should not get emails, butX-Loop: yes
is also encountered. -
X-Autoreply
; fairly rare, and always seems to have a value ofyes
.
Email address
Check if the From
or Reply-To
headers contains noreply
, no-reply
, or
no_reply
(regex: ^no.?reply@
).
HTML only
If an email only has a HTML part and no no text part, it’s a good indication this is an autogenerated email or newsletter. Pretty much all regular mail clients also set a text part.
Delivery failures
Many delivery failure messages don’t really indicate that they’re failures. Some ways to check this:
From
containsmailer-daemon
orMail Delivery Subsystem
Specific mail library footprints
Many mail libraries leave some sort of footprint, and most regular mail clients override this with their own data. Checking for this seems to work fairly well.
-
X-Mailer: Microsoft CDO for Windows 2000
– Set by some MS software; I can only find it on autogenerated emails. Yes, it’s still used in 2015. -
Message-ID
header contains.JavaMail.
– I’ve found a few (5 on 50k) regular messages with this, but not many; the vast majority (thousands) of messages are newsletters, order confirmations, etc. -
^X-Mailer
starts withPHP
. This should catch bothX-Mailer: PHP/5.5.0
andX-Mailer: PHPmailer blah blah
. The same asJavaMail
applies. -
X-Library
presence; only Indy seems to set this. -
X-Mailer
starts withwdcollect
. Set by some Plesk mails. -
X-Mailer
starts withMIME-tools
.
Final precaution: limit the number of replies
Even when following all of the above advice you may still encounter an email
program that will slip through. This can very dangerous as email systems that
simply IF email THEN send_email
have the potential to cause infinite email
loops.
For this reason I recommend keeping track of which emails you’ve sent an autoreply to and rate limiting this to at most n emails in n minutes per email address. This will break the back-and-forth chain.
We use one email per five minutes, but something less strict will probably also work well.
What you need to set on your auto-response
The specifics for this will vary depending on what sort of mails you’re sending. This is what we use for autoreply emails:
Auto-Submitted: auto-replied
X-Auto-Response-Suppress: All
Precedence: auto_reply
Also please don’t forget to set In-Reply-To: <Message-ID of original message>
;
this ensures the emails are threaded properly (a surprising number of
autoresponders seem to forget this).