All addresses in a bounced list posting seem to be considered as bouncing
Hi,
I noticed this happened (bounce.log):
Jul 09 12:46:51 2022 (71506) Member xxxxx@xxxxx.force9.co.uk already scored a bounce on list users.mailman.bitfolk.com today. Jul 09 12:46:51 2022 (71506) Bounce message for a non subscriber: Email users-leave@mailman.bitfolk.com is not a subcriber of users.mailman.bitfolk.com Jul 09 12:46:51 2022 (71506) Member andy@bitfolk.com on list users.mailman.bitfolk.com, bounce score = 1. Jul 09 12:46:51 2022 (71506) Bounce message for a non subscriber: Email users@mailman.bitfolk.com is not a subcriber of users.mailman.bitfolk.com
What happened here is that a reply to one of my posts was rejected by subscriber xxxxx@xxxxx.force9.co.uk at smtp time (Exim mainlog):
2022-07-09 12:46:13 1oA9qc-007qOx-0T ** xxxxx@xxxxx.force9.co.uk R=dnslookup T=remote_smtp H=mx.avasin.plus.net [212.159.8.200] X=TLS1.2:ECDHE_X25519__RSA_PSS_RSAE_SHA256__AES_256_GCM:256 CV=yes DN="businessCategory=Private Organization,jurisdictionOfIncorporationCountryName=GB,serialNumber=03279013,C=GB,L=Sheffield,O=Plusnet plc,CN=mx.avasin.plus.net": SMTP error from remote mail server after end of data: 552 Spam Message Rejected
The message contained my email address (andy@bitfolk.com) in quoted text, since it was a reply to a message of mine, and also contained addresses from the list's footer, which accounts for the log messages about users-leave@mailman.bitfolk.com and users@mailman.bitfolk.com.
I'm concerned that Mailman seems to have scored my address as bouncing even though it isn't, merely because it appears inside a message that was rejected.
Is that expected? Is there any way to avoid it?
I see that Mailman has a bunch of heuristics to parse bounce messages. In this case though, there shouldn't have been a bounce message received to users-bounces@, as this was an SMTP-time rejection. Mailman seems to have parsed addresses out of the actual mail it was sending, rather than only scoring the subscriber address it was trying to send a list posting to.
Cheers, Andy
On Sat, Jul 09, 2022 at 01:35:52PM +0000, Andy Smith wrote:
The message contained my email address (andy@bitfolk.com) in quoted text, since it was a reply to a message of mine
I've just realised that in the list posting involved, my address was also present as a Cc: header, so probably that is where Mailman got it from rather than the body. Even so, is that expected and desirable? It doesn't seem correct, especially for an SMTP-time reject.
Thanks, Andy
On 7/9/22 6:35 AM, Andy Smith wrote:
I'm concerned that Mailman seems to have scored my address as bouncing even though it isn't, merely because it appears inside a message that was rejected.
Is that expected? Is there any way to avoid it?
No, it's not and it should not ever happen.
I see that Mailman has a bunch of heuristics to parse bounce messages. In this case though, there shouldn't have been a bounce message received to users-bounces@, as this was an SMTP-time rejection. Mailman seems to have parsed addresses out of the actual mail it was sending, rather than only scoring the subscriber address it was trying to send a list posting to.
Yes, flufl.bounce is used to parse messages delivered to the -bounces. There are a lot of hueristic recognizers, but RFC 3464 compliant DSNs should be recognized by the DSN detector which is tried first and which gets the failed address from the report. See https://gitlab.com/warsaw/flufl.bounce/-/blob/master/flufl/bounce/_scan.py#L...
However, SMTP failures do not result in DSNs to the -bounces address. If recipients are refused at SMTP time, this should result in a message in Mailman's smtp.log of the form
<message_id> recipients refused: <mta's message>
and we create a fake DSN (see https://gitlab.com/mailman/mailman/-/blob/master/src/mailman/runners/outgoin...) to be processed as a bounce for the failed recipient. In this case, no parsing of the failed message should occur.
I can't explain what happened in your case. It seems that somehow the original post was delivered to the -bounces address and parsed by one of the heuristic recognizers (probably simplematch), but how could that happen?
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mark Sapiro writes:
I can't explain what happened in your case. It seems that somehow the original post was delivered to the -bounces address and parsed by one of the heuristic recognizers (probably simplematch), but how could that happen?
People (or their homebrew MUAs) occasionally think the bounces address is the right place to post. We've seen that before, maybe that happened this time. Or it could be a deliberate attack.
Steve
Hi Mark,
On Sat, Jul 09, 2022 at 09:06:55AM -0700, Mark Sapiro wrote:
I can't explain what happened in your case. It seems that somehow the original post was delivered to the -bounces address and parsed by one of the heuristic recognizers (probably simplematch), but how could that happen?
This seems to have been the DSN that caused that bounce score (Exim mainlog):
2022-07-09 12:46:13 1oA9qf-007qVv-2R <= <> R=1oA9qc-007qOx-0T U=Debian-exim P=local S=44941 2022-07-09 12:46:13 1oA9qf-007qVv-2R => users-bounces@mailman.bitfolk.com R=mailman3_router T=mailman3_transport H=127.0.0.1 [127.0.0.1] C="250 Ok" 2022-07-09 12:46:13 1oA9qf-007qVv-2R Completed
The P=local is a local submission, so I think that was generated by Mailman in response to the SMTP reject.
In the smtp.log, mail from list poster came in:
Jul 09 12:46:04 2022 (71509) ('127.0.0.1', 45530) handling connection Jul 09 12:46:04 2022 (71509) ('127.0.0.1', 45530) Data: b'LHLO lists0.bitfolk.com' Jul 09 12:46:04 2022 (71509) ('127.0.0.1', 45530) Data: b'MAIL FROM:<xxxxx@xxxxx.com>' Jul 09 12:46:04 2022 (71509) ('127.0.0.1', 45530) sender: xxxxx@xxxxx.com Jul 09 12:46:04 2022 (71509) ('127.0.0.1', 45530) Data: b'RCPT TO:<users@mailman.bitfolk.com>' Jul 09 12:46:04 2022 (71509) ('127.0.0.1', 45530) recip: users@mailman.bitfolk.com Jul 09 12:46:04 2022 (71509) ('127.0.0.1', 45530) Data: b'DATA' Jul 09 12:46:04 2022 (71509) ('127.0.0.1', 45530) Data: b'QUIT' Jul 09 12:46:04 2022 (71509) ('127.0.0.1', 45530) connection lost Jul 09 12:46:04 2022 (71509) Connection lost during _handle_client()
Over in Exim's mainlog I then see that delivered out to list members. One of them rejects it at SMTP time, which I think causes this:
Jul 09 12:46:13 2022 (71509) Available AUTH mechanisms: LOGIN(builtin) PLAIN(builtin) Jul 09 12:46:13 2022 (71509) Peer: ('127.0.0.1', 35440) Jul 09 12:46:13 2022 (71509) ('127.0.0.1', 35440) handling connection Jul 09 12:46:13 2022 (71509) ('127.0.0.1', 35440) Data: b'LHLO lists0.bitfolk.com' Jul 09 12:46:13 2022 (71509) ('127.0.0.1', 35440) Data: b'MAIL FROM:<>' Jul 09 12:46:13 2022 (71509) ('127.0.0.1', 35440) sender: <> Jul 09 12:46:13 2022 (71509) ('127.0.0.1', 35440) Data: b'RCPT TO:<users-bounces@mailman.bitfolk.com>' Jul 09 12:46:13 2022 (71509) ('127.0.0.1', 35440) recip: users-bounces@mailman.bitfolk.com Jul 09 12:46:13 2022 (71509) ('127.0.0.1', 35440) Data: b'DATA' Jul 09 12:46:13 2022 (71509) ('127.0.0.1', 35440) Data: b'QUIT' Jul 09 12:46:13 2022 (71509) ('127.0.0.1', 35440) connection lost Jul 09 12:46:13 2022 (71509) Connection lost during _handle_client()
…and I think that bounce message somehow incremented the bounce score for andy@bitfolk.com even though that address was only present in the original post's Cc: and body.
Is there a good way to further debug this? Maybe by being able to see the full content of the DSNs that Mailman generates?
I think I've seen this before in the last couple of weeks so I'm sure it'll crop up again soon. I've seen it mention about bounce score for users-leave before, but I only really paid attention today when it also listed my own address.
Thanks, Andy
On 7/9/22 9:51 AM, Andy Smith wrote:
Over in Exim's mainlog I then see that delivered out to list members. One of them rejects it at SMTP time, which I think causes this:
Jul 09 12:46:13 2022 (71509) Available AUTH mechanisms: LOGIN(builtin) PLAIN(builtin) Jul 09 12:46:13 2022 (71509) Peer: ('127.0.0.1', 35440) Jul 09 12:46:13 2022 (71509) ('127.0.0.1', 35440) handling connection Jul 09 12:46:13 2022 (71509) ('127.0.0.1', 35440) Data: b'LHLO lists0.bitfolk.com' Jul 09 12:46:13 2022 (71509) ('127.0.0.1', 35440) Data: b'MAIL FROM:<>' Jul 09 12:46:13 2022 (71509) ('127.0.0.1', 35440) sender: <> Jul 09 12:46:13 2022 (71509) ('127.0.0.1', 35440) Data: b'RCPT TO:<users-bounces@mailman.bitfolk.com>' Jul 09 12:46:13 2022 (71509) ('127.0.0.1', 35440) recip: users-bounces@mailman.bitfolk.com Jul 09 12:46:13 2022 (71509) ('127.0.0.1', 35440) Data: b'DATA' Jul 09 12:46:13 2022 (71509) ('127.0.0.1', 35440) Data: b'QUIT' Jul 09 12:46:13 2022 (71509) ('127.0.0.1', 35440) connection lost Jul 09 12:46:13 2022 (71509) Connection lost during _handle_client()
The SMTP rejection should not cause that. An SMTP recipient reject is processed internally and does not result in anything sent via SMTP to the list-bounces address.
…and I think that bounce message somehow incremented the bounce score for andy@bitfolk.com even though that address was only present in the original post's Cc: and body.
If the original post was the message delivered to users-bounces@mailman.bitfolk.com, it is possible that simplematch or another recognizer parsed those addresses from the message body.
Is there a good way to further debug this? Maybe by being able to see the full content of the DSNs that Mailman generates?
As I posted previously, the DSN that Mailman generates is defined at https://gitlab.com/mailman/mailman/-/blob/master/src/mailman/runners/outgoin..., but this message is not sent anywhere. It is only saved for possible inclusion in an admin notification or a probe message.
If you send me the post as received from the list, I can run it through flufl.bounce and see what that produces.
You might also be able to find the fake dsn and/or the message received
via SMTP to users-bounces@mailman.bitfolk.com in Mailman's var/messages/
directory, although these are pickles and need to be examined with
mailman qfile.
I think I've seen this before in the last couple of weeks so I'm sure it'll crop up again soon. I've seen it mention about bounce score for users-leave before, but I only really paid attention today when it also listed my own address.
users-leave probably is parsed out of the message footer.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Hi Mark,
On Sat, Jul 09, 2022 at 12:05:32PM -0700, Mark Sapiro wrote:
If the original post was the message delivered to users-bounces@mailman.bitfolk.com, it is possible that simplematch or another recognizer parsed those addresses from the message body.
For now I've configured a copy of all mails to -bounces addresses, so if/when it happens again I can inspect the actual email that triggered it.
In case anyone else ever wants to do that, this is one way to do it on Exim:
# This goes in your "routers" section. # Add a router, before every other router that you would want to # tap messages for. traffic_tap: unseen no_expn no_verify transport = local_copy_outgoing driver = accept # "mm_domains" is a list of domain names for mailman, e.g. # "lists.example.com"; it's set as part of the usual Mailman install on # Debian. domains = +mm_domains # Only do this for bounce addresses. So between this and the above, this # router will only happen for *-bounces*@your-mailman. local_part_suffix = -bounces : -bounces+*
# This goes in your "transports" section. # Add a transport for taking a local copy. Mail to Mailman is "outgoing" as it # is delivered by LMTP over a socket. But the name is not important. local_copy_outgoing: driver = appendfile # Simply store it all in a maildir. If you're expecting a lot of # messages then you'll want a directory structure by date and/or # recipient or something. directory = /var/mail/archive/.outgoing/ delivery_date_add envelope_to_add return_path_add # If not on Debian then your exim probably runs as a different user/.group. group = Debian-exim user = Debian-exim mode = 0660 maildir_format = true create_directory = true
Cheers, Andy
On Mon, Jul 11, 2022 at 02:44:59AM +0000, Andy Smith wrote:
Hi Mark,
On Sat, Jul 09, 2022 at 12:05:32PM -0700, Mark Sapiro wrote:
If the original post was the message delivered to users-bounces@mailman.bitfolk.com, it is possible that simplematch or another recognizer parsed those addresses from the message body.
For now I've configured a copy of all mails to -bounces addresses, so if/when it happens again I can inspect the actual email that triggered it.
This just happened again. As a refresher, the issue I'm seeing is that Mailman registers a bounce against addresses that aren't subscribed, including addresses for the list itself and for the -leave address that is in some footer text.
Here's the bounce.log:
Sep 30 17:12:25 2022 (8428) Member [elided1]@[elided1].force9.co.uk on list users.mailman.bitfolk.com, bounce score = 1. Sep 30 17:12:25 2022 (8428) Bounce message for a non subscriber: Email users@mailman.bitfolk.com is not a subcriber of users.mailman.bitfolk.com Sep 30 17:12:25 2022 (8428) Bounce message for a non subscriber: Email mailto:[elided2]@gmail.com is not a subcriber of users.mailman.bitfolk.com Sep 30 17:12:25 2022 (8428) Member [elided2]@gmail.com on list users.mailman.bitfolk.com, bounce score = 1. Sep 30 17:12:25 2022 (8428) Bounce message for a non subscriber: Email users-leave@mailman.bitfolk.com is not a subcriber of users.mailman.bitfolk.com Sep 30 17:12:25 2022 (8428) Member [elided3]@[elided3].org.uk on list users.mailman.bitfolk.com, bounce score = 1.
I have elided parts of the email addresses of real subscribers.
Attached is the actual non-delivery report from the MTA that triggered this. In it I have done my best to elide the names and addresses of real people, but nothing else. They match the same elisions from the bounce log.
Any ideas why Mailman is interpreting this NDR so over-zealously? It seems like a pretty standard one. [elided1].force9.co.uk is the subscriber who rejected the mail, yet also users@mailman.bitfolk.com, [elided2]@gmail.com, users-leave@mailman.bitfolk.com and [elided3]@[elided3].org.uk (sender) were punished.
Thanks, Andy
On 9/30/22 12:41, Andy Smith wrote:
This just happened again. As a refresher, the issue I'm seeing is that Mailman registers a bounce against addresses that aren't subscribed, including addresses for the list itself and for the -leave address that is in some footer text.
Here's the bounce.log:
Sep 30 17:12:25 2022 (8428) Member [elided1]@[elided1].force9.co.uk on list users.mailman.bitfolk.com, bounce score = 1. Sep 30 17:12:25 2022 (8428) Bounce message for a non subscriber: Email users@mailman.bitfolk.com is not a subcriber of users.mailman.bitfolk.com Sep 30 17:12:25 2022 (8428) Bounce message for a non subscriber: Email mailto:[elided2]@gmail.com is not a subcriber of users.mailman.bitfolk.com Sep 30 17:12:25 2022 (8428) Member [elided2]@gmail.com on list users.mailman.bitfolk.com, bounce score = 1. Sep 30 17:12:25 2022 (8428) Bounce message for a non subscriber: Email users-leave@mailman.bitfolk.com is not a subcriber of users.mailman.bitfolk.com Sep 30 17:12:25 2022 (8428) Member [elided3]@[elided3].org.uk on list users.mailman.bitfolk.com, bounce score = 1.
I have elided parts of the email addresses of real subscribers.
Attached is the actual non-delivery report from the MTA that triggered this. In it I have done my best to elide the names and addresses of real people, but nothing else. They match the same elisions from the bounce log.
What is your version of flufl.bounce?
I made one change to your attached DSN. In the message/selivery-status part, I changed
Final-Recipient: rfc822;[elided1]@[elided1].force9.co.uk
to
Final-Recipient: rfc822;elided1@elided1.force9.co.uk
and processed the message with flufl.bounce.scan_message() which returned the single address elided1@elided1.force9.co.uk.
I suspect this is https://gitlab.com/warsaw/flufl.bounce/-/issues/12 which was fixed in flufl.bounce 4.0 and upgrading to 4.0 will fix this.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Hi Mark,
On Fri, Sep 30, 2022 at 01:16:10PM -0700, Mark Sapiro wrote:
What is your version of flufl.bounce?
I'm using Debian stable, and this is what is packaged:
$ dpkg -l | grep flufl ii python3-flufl.bounce 3.0.1-1 all email bounce detectors (Python 3)
I suspect this is https://gitlab.com/warsaw/flufl.bounce/-/issues/12 which was fixed in flufl.bounce 4.0 and upgrading to 4.0 will fix this.
So that probably answers that then. Hmm, I wonder about the best way to proceed. The next stable release of Debian will have something that calls itself 4.0-2 and looks like it is fixed there:
https://salsa.debian.org/python-team/packages/flufl.bounce/-/blob/master/flufl/bounce/_scan.pyso maybe I can ask for that package to be put in stable-backports.
I think the other option might be to remove system package python3-flufl.bounce, add it back with "equivbs" so that its system dependencies think it is present and then install the latest one from pip.
I'll report the bug in Debian anyway. Thanks!
Cheers, Andy
Hello,
On Fri, Sep 30, 2022 at 08:53:38PM +0000, Andy Smith wrote:
On Fri, Sep 30, 2022 at 01:16:10PM -0700, Mark Sapiro wrote:
I suspect this is https://gitlab.com/warsaw/flufl.bounce/-/issues/12 which was fixed in flufl.bounce 4.0 and upgrading to 4.0 will fix this.
So that probably answers that then. Hmm, I wonder about the best way to proceed. The next stable release of Debian will have something that calls itself 4.0-2 and looks like it is fixed there:
https://salsa.debian.org/python-team/packages/flufl.bounce/-/blob/master/flufl/bounce/_scan.py
It seems that this package version can be downloaded and installed in Debian stable (bullseye):
https://packages.debian.org/bookworm/all/python3-flufl.bounce/downloadIt isn't recommended to mix packages from different releases of course, but the changes seem quite minimal and it fixes the problem for me.
Cheers, Andy
participants (3)
-
Andy Smith -
Mark Sapiro -
Stephen J. Turnbull