Exim, Mailman3 and missing Message-ID:
Just a note that may help some people.
After a bit of a learning curve, managed to get Mailman3 working in Debian 10 using the Debian package.
I have mailman3-full + exim4 + apache2 setup:
I did quite a lot of testing and getting it working, then when all seemed well, declared success and made it live.
As (bad) luck would have it, the very first real message bounced: :-(
2019-07-25 20:53:09 1hqjnY-0005ms-RD <= xxxxxxx@ntlworld.com H=cloudi.spamtitan.com [52.26.55.167] P=esmtps X=TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256 CV=no S=7854 DKIM=ntlworld.com 2019-07-25 20:53:09 1hqjnY-0005ms-RD ** mylist@lentil.org R=mailman3_router T=mailman3_t H=127.0.0.1 [127.0.0.1]: LMTP error after DATA: 550 No Message-ID header provided 2019-07-25 20:53:09 1hqjnZ-0005mx-3F <= <> R=1hqjnY-0005ms-RD U=Debian-exim P=local S=9222
The user forwarded me the bounce, which contained the original message and headers, and indeed the original message did not contain a Message-ID: header. (Oddly, his message to me reporting the error using the same mail client DID contain a Message-ID: header.)
Mail client: X-Mailer: Microsoft Outlook 16.0
So it seems in certain circumstances, Outlook doesn't put a Message-ID: header in its messages.
The change from Mailman2.x to Mailman3 means it's now using LMTP to deliver to mailman instead of a local submission seems to have exposed this issue.
Exim tries to be helpful for "locally submitted" messages:
<https://www.exim.org/exim-html-current/doc/html/spec_html/ch-message_processing.html>
"12. The Message-ID: header line
If a locally-generated or submission-mode incoming message does not contain a Message-ID: or Resent-Message-ID: header line, and the suppress_local_fixups control is not set, Exim adds a suitable header line to the message. If there are any Resent-: headers in the message, it creates Resent-Message-ID:. The id is constructed from Exim's internal message id, preceded by the letter E to ensure it starts with a letter, and followed by @ and the primary host name."
However: "Note: Messages received over TCP/IP on the loopback interface (127.0.0.1 or ::1) are not considered to be locally-originated. Exim does not treat the loopback interface specially in any way."
Therefore, Mailman3 using LMTP means exim will NOT "fix up" this problem before mailman3 receives the message.
My solution for now (apart from telling people not to use old broken Outlook!) is to modify the exim mailman3_transport so that any message with a missing Message-ID: will have one generated by exim before it's passed to LMTP. This reproduces the same behaviour exim uses for locally submitted messages.
(headers_remove ensures that we end up only with one Message-ID: header and not two, then we put back either the original Message-ID, or if that's not there, use Exim's ID to generate a new Message-ID header.)
Here's an updated version of the example at: https://mailman.readthedocs.io/en/latest/src/mailman/docs/mta.html#exim
# /etc/exim4/conf.d/main/25_mm3_macros # The colon-separated list of domains served by Mailman. domainlist mm_domains=list.example.net
MM3_LMTP_PORT=8024
# MM3_HOME must be set to mailman's var directory, wherever it is # according to your installation.
# debian default is /var/lib/mailman3 MM3_HOME=/opt/mailman/var MM3_UID=list MM3_GID=list
################################################################ # The configuration below is boilerplate: # you should not need to change it.
# The path to the list receipt (used as the required file when # matching list addresses) MM3_LISTCHK=MM3_HOME/lists/${local_part}.${domain}
# /etc/exim4/conf.d/router/455_mm3_router
mailman3_router:
driver = accept
domains = +mm_domains
require_files = MM3_LISTCHK
local_part_suffix_optional
local_part_suffix =
-bounces : -bounces+* :
-confirm : -confirm+* :
-join : -leave :
-owner : -request :
-subscribe : -unsubscribe
transport = mailman3_transport
# /etc/exim4/conf.d/transport/55_mm3_transport mailman3_transport: driver = smtp protocol = lmtp allow_localhost hosts = localhost port = MM3_LMTP_PORT rcpt_include_affixes = true
mailman3_transport: driver = smtp protocol = lmtp allow_localhost hosts = localhost port = MM3_LMTP_PORT rcpt_include_affixes = true headers_remove = message-id headers_add = "Message-ID: ${if def:header_message-id:{$h_message-id:}{<E${message_exim_id}@${qualify_domain}>}}"
On 7/26/19 6:20 AM, Robert Lister wrote:
The change from Mailman2.x to Mailman3 means it's now using LMTP to deliver to mailman instead of a local submission seems to have exposed this issue.
It is not the use of LMTP for delivery that is the issue. It is the fact that Mailman 3 uses a hash of the Message-ID: in various contexts to identify/retrieve a message. Thus, every message must have a Message-ID.
See <https://gitlab.com/mailman/mailman/issues/490> for more on this.
...
My solution for now (apart from telling people not to use old broken Outlook!) is to modify the exim mailman3_transport so that any message with a missing Message-ID: will have one generated by exim before it's passed to LMTP. This reproduces the same behaviour exim uses for locally submitted messages.
(headers_remove ensures that we end up only with one Message-ID: header and not two, then we put back either the original Message-ID, or if that's not there, use Exim's ID to generate a new Message-ID header.)
Here's an updated version of the example at: https://mailman.readthedocs.io/en/latest/src/mailman/docs/mta.html#exim ... mailman3_transport: driver = smtp protocol = lmtp allow_localhost hosts = localhost port = MM3_LMTP_PORT rcpt_include_affixes = true headers_remove = message-id headers_add = "Message-ID: ${if def:header_message-id:{$h_message-id:}{<E${message_exim_id}@${qualify_domain}>}}"
Thank you for this. I have created <https://gitlab.com/mailman/mailman/merge_requests/540> to add this to the doc.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Robert:
Thank you very much for the analysis and the configuration.
Robert Lister writes:
As (bad) luck would have it, the very first real message bounced: :-(
This is *very* bad luck. I don't think I've seen messages without Message-ID in decades!
Mail client: X-Mailer: Microsoft Outlook 16.0
So it seems in certain circumstances, Outlook doesn't put a Message-ID: header in its messages.
Outlook is not my favorite MUA (and I don't mean that I don't like using it). Nevertheless, RFC 5321 suggests that the responsibility for dealing with this or rejecting the message is with the submission host:
The following changes to a message being processed MAY be applied when necessary by an originating SMTP server, or one used as the target of SMTP as an initial posting (message submission) protocol:
o Addition of a message-id field when none appears
o Addition of a date, time, or time zone when none appears
o Correction of addresses to proper FQDN format
The less information the server has about the client, the less likely these changes are to be correct and the more caution and conservatism should be applied when considering whether or not to perform fixes and how. These changes MUST NOT be applied by an SMTP server that provides an intermediate relay function.
The context strongly suggests that a similar caution applies to final delivery servers, and I would assume Mediators such as Mailman. I conclude that the post is broken and very good reason would be needed to patch it up.
Therefore, Mailman3 using LMTP means exim will NOT "fix up" this problem before mailman3 receives the message.
As I read the passage you quoted, Exim won't fix up the problem regardless of whether it delivers to Mailman via LMTP or by pipe, etc. The problem is that Mailman's LMTP server rejects malformed mail. I seem to recall that in Mailman 2, Mailman itself would add a Message-ID field if one was not present. I don't know how easy it would be to provide this feature, since we would have to patch the LMTP server code so the message could make it to Mailman proper (Mailman 3 imports the LMTP server from the stdlib).
My solution for now (apart from telling people not to use old broken Outlook!) is to modify the exim mailman3_transport so that any message with a missing Message-ID: will have one generated by exim before it's passed to LMTP. This reproduces the same behaviour exim uses for locally submitted messages.
I'm not sure it's a good idea to do this (and certainly not by default). The only times I've seen messages without a Message-ID are spam messages. I suppose we can add this to the documentation.
Steve
On 7/26/19 11:05 AM, Stephen J. Turnbull wrote:
The problem is that Mailman's LMTP server rejects malformed mail. I seem to recall that in Mailman 2, Mailman itself would add a Message-ID field if one was not present. I don't know how easy it would be to provide this feature, since we would have to patch the LMTP server code so the message could make it to Mailman proper (Mailman 3 imports the LMTP server from the stdlib).
It's a simple patch to the lmtp runner (Mailman's LMTP server). There's a closed issue on this at <https://gitlab.com/mailman/mailman/issues/490> - closed because I convinced the submitter it wasn't the right thing to do.
My solution for now (apart from telling people not to use old broken Outlook!) is to modify the exim mailman3_transport so that any message with a missing Message-ID: will have one generated by exim before it's passed to LMTP. This reproduces the same behaviour exim uses for locally submitted messages.
I'm not sure it's a good idea to do this (and certainly not by default). The only times I've seen messages without a Message-ID are spam messages. I suppose we can add this to the documentation.
I added Robert's solution at <https://mailman.readthedocs.io/en/latest/src/mailman/docs/mta.html#troubleshooting>
The requirement for a Message-ID: is documented, albeit not very prominently, at <https://mailman.readthedocs.io/en/latest/build/lib/mailman/runners/docs/lmtp.html?highlight=message-id#lmtp-server>.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
PGRpdiBkaXI9J2F1dG8nPjxkaXYgZGlyPSJhdXRvIj48YnI+PC9kaXY+T25lIHRoaW5nIHR oYXQg ZG9lcyBvY2N1ciB0byBtZSBpcyB0aGF0IG15IGV4aW0gaGVhZGVyIGZ1ZGdlIHdpbGwgcHJ vYmFi bHkgY2F1c2UgdGhlIE1lc3NhZ2UtSUQgaGVhZGVyIHRvIG1vdmUgYXJvdW5kIGZyb20gaXR zIG9y aWdpbmFsIHBvc2l0aW9uLCBhbmQgaWYgc29tZW9uZSBpcyBkb2luZyBoYXNoaW5nIG9yIGF jdHVh bGx5IHZhbGlkYXRpbmcgKGdwZyBzaWduaW5nPykgdGhlIGhlYWRlcnMgaGF2ZW4ndCBiZWV uIHRh bXBlcmVkIHdpdGgsIHRoZW4gaXQgbWlnaHQgY29tcGxhaW4sIHNpbmNlIGFsdGhvdWdoIHR oZSBt ZXNzYWdlLWlkIGhhc24ndCBjaGFuZ2VkLCBpdCBtaWdodCBiZSBpbiBhIGRpZmZlcmVudCB wbGFj ZSBpbiB0aGUgaGVhZGVycy48ZGl2IGRpcj0iYXV0byI+PGJyPjwvZGl2PjxkaXYgZGlyPSJ hdXRv Ij5UaGVuIGFnYWluLCBtYWlsbWFuIGNhbiBtYWtlIHBsZW50eSBvZiBvdGhlciBtb2RpZml jYXRp b25zIGFuZCBhZGRpdGlvbnMgdG8gdGhlIGhlYWRlcnMgYW55d2F5LCBzbyBJJ20gbm90IHN 1cmUg aXQgbWF0dGVycy4mbmJzcDs8L2Rpdj48ZGl2IGRpcj0iYXV0byI+PGJyPjwvZGl2PjxkaXY gZGly PSJhdXRvIj5Qcm9iYWJseSB0aGUgbW9yZSBjb3JyZWN0IHdheSB0byBkbyBpdCBpcyBvbmx 5IGFk ZCBhIG5ldyBoZWFkZXIgaWYgaXQncyBtaXNzaW5nLCBhbmQgaWYgaXQncyB0aGVyZSwgbGV hdmUg aXQgYWxvbmUuPC9kaXY+PGRpdiBkaXI9ImF1dG8iPjxicj48L2Rpdj48ZGl2IGRpcj0iYXV 0byI+ VGhpcyBpcyBwcm9iYWJseSBwb3NzaWJsZSBpbiBleGltIHdpdGggc29tZSBtb3JlIHZhcml hYmxl cyBhbmQgYSBsb3Qgb2YgbmVzdGVkIGN1cmx5IGJyYWNlcy4uLiZuYnNwOzwvZGl2PjxkaXY gZGly PSJhdXRvIj48YnI+PC9kaXY+PGRpdiBkaXI9ImF1dG8iPlJvYiZuYnNwOzwvZGl2PjwvZGl 2Pjxk aXYgY2xhc3M9ImdtYWlsX2V4dHJhIj48YnI+PGRpdiBjbGFzcz0iZ21haWxfcXVvdGUiPk9 uIDI2 IEp1bCAyMDE5IDIwOjAzLCBNYXJrIFNhcGlybyAmbHQ7bWFya0Btc2FwaXJvLm5ldCZndDs gd3Jv dGU6PGJyIHR5cGU9ImF0dHJpYnV0aW9uIiAvPjxibG9ja3F1b3RlIGNsYXNzPSJxdW90ZSI gc3R5 bGU9Im1hcmdpbjowIDAgMCAuOGV4O2JvcmRlci1sZWZ0OjFweCAjY2NjIHNvbGlkO3BhZGR pbmct bGVmdDoxZXgiPjxwIGRpcj0ibHRyIj5PbiA3LzI2LzE5IDExOjA1IEFNLCBTdGVwaGVuIEo uIFR1 cm5idWxsIHdyb3RlOiYjMTM7PGJyPgomIzEzOzxicj4KJmd0OyBUaGUgcHJvYmxlbSBpcyB 0aGF0 IE1haWxtYW4ncyBMTVRQIHNlcnZlciByZWplY3RzIG1hbGZvcm1lZCBtYWlsLiZuYnNwOyB JJiMx Mzs8YnI+CiZndDsgc2VlbSB0byByZWNhbGwgdGhhdCBpbiBNYWlsbWFuIDIsIE1haWxtYW4 gaXRz ZWxmIHdvdWxkIGFkZCBhJiMxMzs8YnI+CiZndDsgTWVzc2FnZS1JRCBmaWVsZCBpZiBvbmU gd2Fz IG5vdCBwcmVzZW50LiZuYnNwOyBJIGRvbid0IGtub3cgaG93IGVhc3kgaXQmIzEzOzxicj4 KJmd0 OyB3b3VsZCBiZSB0byBwcm92aWRlIHRoaXMgZmVhdHVyZSwgc2luY2Ugd2Ugd291bGQgaGF 2ZSB0 byBwYXRjaCB0aGUmIzEzOzxicj4KJmd0OyBMTVRQIHNlcnZlciBjb2RlIHNvIHRoZSBtZXN zYWdl IGNvdWxkIG1ha2UgaXQgdG8gTWFpbG1hbiBwcm9wZXImIzEzOzxicj4KJmd0OyAoTWFpbG1 hbiAz IGltcG9ydHMgdGhlIExNVFAgc2VydmVyIGZyb20gdGhlIHN0ZGxpYikuJiMxMzs8YnI+CiY jMTM7 PGJyPgomIzEzOzxicj4KSXQncyBhIHNpbXBsZSBwYXRjaCB0byB0aGUgbG10cCBydW5uZXI gKE1h aWxtYW4ncyBMTVRQIHNlcnZlcikuIFRoZXJlJ3MmIzEzOzxicj4KYSBjbG9zZWQgaXNzdWU gb24g dGhpcyBhdCYjMTM7PGJyPgombHQ7aHR0cHM6Ly9naXRsYWIuY29tL21haWxtYW4vbWFpbG1 hbi9p c3N1ZXMvNDkwJmd0OyAtIGNsb3NlZCBiZWNhdXNlIEkmIzEzOzxicj4KY29udmluY2VkIHR oZSBz dWJtaXR0ZXIgaXQgd2Fzbid0IHRoZSByaWdodCB0aGluZyB0byBkby4mIzEzOzxicj4KJiM xMzs8 YnI+CiYjMTM7PGJyPgomZ3Q7Jm5ic3A7ICZndDsgTXkgc29sdXRpb24gZm9yIG5vdyAoYXB hcnQg ZnJvbSB0ZWxsaW5nIHBlb3BsZSBub3QgdG8gdXNlIG9sZCYjMTM7PGJyPgomZ3Q7Jm5ic3A 7ICZn dDsgYnJva2VuIE91dGxvb2shKSBpcyB0byBtb2RpZnkgdGhlIGV4aW0gbWFpbG1hbjNfdHJ hbnNw b3J0IHNvIHRoYXQmIzEzOzxicj4KJmd0OyZuYnNwOyAmZ3Q7IGFueSBtZXNzYWdlIHdpdGg gYSBt aXNzaW5nIE1lc3NhZ2UtSUQ6IHdpbGwgaGF2ZSBvbmUgZ2VuZXJhdGVkIGJ5JiMxMzs8YnI +CiZn dDsmbmJzcDsgJmd0OyBleGltIGJlZm9yZSBpdCdzIHBhc3NlZCB0byBMTVRQLiBUaGlzIHJ lcHJv ZHVjZXMgdGhlIHNhbWUgYmVoYXZpb3VyJiMxMzs8YnI+CiZndDsmbmJzcDsgJmd0OyBleGl tIHVz ZXMgZm9yIGxvY2FsbHkgc3VibWl0dGVkIG1lc3NhZ2VzLiYjMTM7PGJyPgomZ3Q7ICYjMTM 7PGJy PgomZ3Q7IEknbSBub3Qgc3VyZSBpdCdzIGEgZ29vZCBpZGVhIHRvIGRvIHRoaXMgKGFuZCB jZXJ0 YWlubHkgbm90IGJ5JiMxMzs8YnI+CiZndDsgZGVmYXVsdCkuJm5ic3A7IFRoZSBvbmx5IHR pbWVz IEkndmUgc2VlbiBtZXNzYWdlcyB3aXRob3V0IGEgTWVzc2FnZS1JRCBhcmUmIzEzOzxicj4 KJmd0 OyBzcGFtIG1lc3NhZ2VzLiZuYnNwOyBJIHN1cHBvc2Ugd2UgY2FuIGFkZCB0aGlzIHRvIHR oZSBk b2N1bWVudGF0aW9uLiYjMTM7PGJyPgomIzEzOzxicj4KJiMxMzs8YnI+CkkgYWRkZWQgUm9 iZXJ0 J3Mgc29sdXRpb24gYXQmIzEzOzxicj4KJmx0O2h0dHBzOi8vbWFpbG1hbi5yZWFkdGhlZG9 jcy5p by9lbi9sYXRlc3Qvc3JjL21haWxtYW4vZG9jcy9tdGEuaHRtbCN0cm91Ymxlc2hvb3Rpbmc mZ3Q7 JiMxMzs8YnI+CiYjMTM7PGJyPgpUaGUgcmVxdWlyZW1lbnQgZm9yIGEgTWVzc2FnZS1JRDo gaXMg ZG9jdW1lbnRlZCwgYWxiZWl0IG5vdCB2ZXJ5JiMxMzs8YnI+CnByb21pbmVudGx5LCBhdCY jMTM7 PGJyPgombHQ7aHR0cHM6Ly9tYWlsbWFuLnJlYWR0aGVkb2NzLmlvL2VuL2xhdGVzdC9idWl sZC9s aWIvbWFpbG1hbi9ydW5uZXJzL2RvY3MvbG10cC5odG1sP2hpZ2hsaWdodD1tZXNzYWdlLWl kI2xt dHAtc2VydmVyJmd0Oy4mIzEzOzxicj4KJiMxMzs8YnI+Ci0tICYjMTM7PGJyPgpNYXJrIFN hcGly byAmbHQ7bWFya0Btc2FwaXJvLm5ldCZndDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJ zcDsm bmJzcDsmbmJzcDsgVGhlIGhpZ2h3YXkgaXMgZm9yIGdhbWJsZXJzLCYjMTM7PGJyPgpTYW4 gRnJh bmNpc2NvIEJheSBBcmVhLCBDYWxpZm9ybmlhJm5ic3A7Jm5ic3A7Jm5ic3A7IGJldHRlciB 1c2Ug eW91ciBzZW5zZSAtIEIuIER5bGFuJiMxMzs8YnI+Cl9fX19fX19fX19fX19fX19fX19fX19 fX19f X19fX19fX19fX19fX19fX19fX19fJiMxMzs8YnI+Ck1haWxtYW4tdXNlcnMgbWFpbGluZyB saXN0 IC0tIG1haWxtYW4tdXNlcnNAbWFpbG1hbjMub3JnJiMxMzs8YnI+ClRvIHVuc3Vic2NyaWJ lIHNl bmQgYW4gZW1haWwgdG8gbWFpbG1hbi11c2Vycy1sZWF2ZUBtYWlsbWFuMy5vcmcmIzEzOzx icj4K aHR0cHM6Ly9saXN0cy5tYWlsbWFuMy5vcmcvbWFpbG1hbjMvbGlzdHMvbWFpbG1hbi11c2V ycy5t YWlsbWFuMy5vcmcvJiMxMzs8YnI+CjwvcD4KPC9ibG9ja3F1b3RlPjwvZGl2Pjxicj48L2R pdj4=
Rob's reply in this thread was garbled due to <https://gitlab.com/mailman/mailman/issues/616>. The text of the reply is quoted below.
On 7/27/19 12:54 AM, Rob Lister wrote:
One thing that does occur to me is that my exim header fudge will probably cause the Message-ID header to move around from its original position, and if someone is doing hashing or actually validating (gpg signing?) the headers haven't been tampered with, then it might complain, since although the message-id hasn't changed, it might be in a different place in the headers.
Then again, mailman can make plenty of other modifications and additions to the headers anyway, so I'm not sure it matters.
Probably the more correct way to do it is only add a new header if it's missing, and if it's there, leave it alone.
This is probably possible in exim with some more variables and a lot of nested curly braces...
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mark Sapiro writes:
Rob's reply in this thread was garbled due to <https://gitlab.com/mailman/mailman/issues/616>. The text of the reply is quoted below.
On 7/27/19 12:54 AM, Rob Lister wrote:
One thing that does occur to me is that my exim header fudge will probably cause the Message-ID header to move around from its original position, and if someone is doing hashing or actually validating (gpg signing?) the headers haven't been tampered with, then it might complain, since although the message-id hasn't changed, it might be in a different place in the headers.
I don't think this is a problem. GPG or S/MIME signing with any software I know of only signs the message body (or a MIME part). DKIM message signing isn't going to care about the order of fields because DKIM canonicalizes the order (at least for the unique fields) according to the specification in the signature field, and that will be verified on the way in. Of course adding a Message-ID will likely break DKIM for the outgoing post, but Mailman as normally used breaks DKIM anyway. This can be partly mitigated by using ARC (authenticated received chain), which is supported by Google already, and possibly some of the other usual DMARC suspects (Yahoo!, AOL).
Probably the more correct way to do it is only add a new header if it's missing, and if it's there, leave it alone.
True, and I think this does it:
headers_add = ${if def:h_message-id:{fail}{Message-ID: ... id generator code ...}}
If you like that and test it, I would really appreciate feedback so we can update the documentation.
One thing *I* would like to add to this recipe is logging. I'd be curious to see how frequently it happens outside of spam and junior high school hackers using netcat as their MUA. I don't see how to do this in Exim, though.
Another is using a prefix other than "E" to indicate this isn't standard Exim behavior.
Steve
participants (4)
-
Mark Sapiro -
Rob Lister -
Robert Lister
-
Stephen J. Turnbull