February 2025 - Mailman-users - lists.mailman3.org

Mailman shutting down Hyperkitty: Request too large
by Arte Chambers Feb. 2, 2025

Feb. 2, 2025

Hello I need help troubleshooting a problem with MM. The service is stopping and throwing this error in the mailman log "Exception in the HyperKitty archiver: 413 Request Entity Too Large" I'm not sure what my next steps should be to troubleshoot this issue. System CTL has this message and i'm not sure if it's relevant to the issue. ``` >sudo systemctl status mailman3.service systemd[1]: mailman3.service: Can't open PID file /opt/mailman/mm/var/master.pid (yet?) after start: No such file or directory ``` The MM Service will run for a while then shuts down. /mailman.log ``` Jan 30 05:22:20 2025 (1547) HyperKitty failure on https://list.louisvillecommunitygrocery.com/archives/api/mailman/archive: <html> <head><title>413 Request Entity Too Large</title></head> <body> <center><h1>413 Request Entity Too Large</h1></center> <hr><center>nginx/1.24.0 (Ubuntu)</center> </body> </html> (413) Jan 30 05:22:20 2025 (1547) Could not archive the message with id < CAPesOD3EWaFzfVZAbUzFm3Vk3Xnqcf9KodVic6agaT27Kmhy0w(a)mail.gmail.com> Jan 30 05:22:20 2025 (1547) archiving failed, re-queuing (mailing-list lcg-board-private.list.louisvillecommunitygrocery.com, message < CAPesOD3EWaFzfVZAbUzFm3Vk3Xnqcf9KodVic6agaT27Kmhy0w(a)mail.gmail.com>) Jan 30 05:22:20 2025 (1547) Exception in the HyperKitty archiver: <html> <head><title>413 Request Entity Too Large</title></head> <body> <center><h1>413 Request Entity Too Large</h1></center> <hr><center>nginx/1.24.0 (Ubuntu)</center> </body> </html> Jan 30 05:22:20 2025 (1547) Traceback (most recent call last): File "/opt/mailman/venv/lib/python3.12/site-packages/mailman_hyperkitty/__init__.py", line 158, in _archive_message url = self._send_message(mlist, msg) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib/python3.12/site-packages/mailman_hyperkitty/__init__.py", line 228, in _send_message raise ValueError(result.text) ValueError: <html> <head><title>413 Request Entity Too Large</title></head> <body> <center><h1>413 Request Entity Too Large</h1></center> <hr><center>nginx/1.24.0 (Ubuntu)</center> </body> </html> Jan 30 05:22:21 2025 (1547) HyperKitty archived message <CAPesOD25frujqYsAXU3kk8MUOY=h3ZV4cgzUXSi9L0c3K2W0NQ(a)mail.gmail.com> to https://list.louisvillecommunitygrocery.com/archives/list/testing@list.loui… [30/Jan/2025:05:22:43 +0000] "GET /3.1/users/lcgcoop(a)gmail.com HTTP/1.1" 200 428 "-" "GNU Mailman REST client v3.3.5" [30/Jan/2025:05:22:43 +0000] "GET /3.1/users/a6af988d603c49c9ac61a3a41562d132/addresses HTTP/1.1" 200 927 "-" "GNU Mailman REST client v3.3.5" [30/Jan/2025:05:22:43 +0000] "POST /3.1/members/find HTTP/1.1" 200 3407 "-" "GNU Mailman REST client v3.3.5" [30/Jan/2025:05:22:43 +0000] "POST /3.1/members/find HTTP/1.1" 200 4705 "-" "GNU Mailman REST client v3.3.5" [30/Jan/2025:05:22:43 +0000] "GET /3.1/addresses/lcgcoop(a)gmail.com HTTP/1.1" 200 401 "-" "GNU Mailman REST client v3.3.5" [30/Jan/2025:05:22:43 +0000] "GET /3.1/lists/ testing(a)list.louisvillecommunitygrocery.com HTTP/1.1" 200 482 "-" "GNU Mailman REST client v3.3.5" [30/Jan/2025:05:22:43 +0000] "GET /3.1/lists/ testing.list.louisvillecommunitygrocery.com/roster/owner HTTP/1.1" 200 763 "-" "GNU Mailman REST client v3.3.5" [30/Jan/2025:05:22:44 +0000] "GET /3.1/addresses/lcgcoop(a)gmail.com HTTP/1.1" 200 401 "-" "GNU Mailman REST client v3.3.5" [30/Jan/2025:05:23:53 +0000] "GET /3.1/lists/ testing.list.louisvillecommunitygrocery.com HTTP/1.1" 200 482 "-" "GNU Mailman REST client v3.3.5" [30/Jan/2025:05:23:53 +0000] "GET /3.1/lists/ testing.list.louisvillecommunitygrocery.com HTTP/1.1" 200 482 "-" "GNU Mailman REST client v3.3.5" [30/Jan/2025:05:23:53 +0000] "GET /3.1/lists/ testing@list.louisvillecommunitygrocery.com/requests/count?token_owner=moderator HTTP/1.1" 200 73 "-" "GNU Mailman REST client v3.3.5" [30/Jan/2025:05:23:53 +0000] "GET /3.1/lists/ testing(a)list.louisvillecommunitygrocery.com/held/count HTTP/1.1" 200 73 "-" "GNU Mailman REST client v3.3.5" [30/Jan/2025:05:24:15 +0000] "GET /3.1/lists/ testing(a)list.louisvillecommunitygrocery.com HTTP/1.1" 200 482 "-" "GNU Mailman REST client v3.3.5" Jan 30 05:42:54 2025 (1550) ACCEPT: <008401db72ce$cb6d50d0$6247f270$@ iglou.com> Jan 30 05:42:55 2025 (1547) HyperKitty failure on https://list.louisvillecommunitygrocery.com/archives/api/mailman/archive: <html> <head><title>413 Request Entity Too Large</title></head> <body> <center><h1>413 Request Entity Too Large</h1></center> <hr><center>nginx/1.24.0 (Ubuntu)</center> </body> </html> (413) Jan 30 05:42:55 2025 (1547) Could not archive the message with id < CAPesOD3EWaFzfVZAbUzFm3Vk3Xnqcf9KodVic6agaT27Kmhy0w(a)mail.gmail.com> Jan 30 05:42:55 2025 (1547) archiving failed, re-queuing (mailing-list lcg-board-private.list.louisvillecommunitygrocery.com, message < CAPesOD3EWaFzfVZAbUzFm3Vk3Xnqcf9KodVic6agaT27Kmhy0w(a)mail.gmail.com>) Jan 30 05:42:55 2025 (1547) Exception in the HyperKitty archiver: <html> <head><title>413 Request Entity Too Large</title></head> <body> <center><h1>413 Request Entity Too Large</h1></center> <hr><center>nginx/1.24.0 (Ubuntu)</center> </body> </html> Jan 30 05:42:55 2025 (1547) Traceback (most recent call last): File "/opt/mailman/venv/lib/python3.12/site-packages/mailman_hyperkitty/__init__.py", line 158, in _archive_message url = self._send_message(mlist, msg) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib/python3.12/site-packages/mailman_hyperkitty/__init__.py", line 228, in _send_message raise ValueError(result.text) ValueError: <html> <head><title>413 Request Entity Too Large</title></head> <body> <center><h1>413 Request Entity Too Large</h1></center> <hr><center>nginx/1.24.0 (Ubuntu)</center> </body> </html> Jan 30 05:42:57 2025 (1547) HyperKitty archived message <008401db72ce$cb6d50d0$6247f270$(a)iglou.com> to https://list.louisvillecommunitygrocery.com/archives/list/lcg-board-private… [2025-01-30 05:46:02 +0000] [1554] [ERROR] Worker (pid:1586) was sent SIGKILL! Perhaps out of memory? [2025-01-30 05:46:02 +0000] [2227] [INFO] Booting worker with pid: 2227 Jan 30 05:46:09 2025 (1553) pipeline runner caught SIGTERM. Stopping. Jan 30 05:46:09 2025 (1553) pipeline runner exiting. Jan 30 05:46:09 2025 (1551) lmtp runner caught SIGTERM. Stopping. Jan 30 05:46:09 2025 (1551) lmtp runner exiting. Jan 30 05:46:09 2025 (1558) digest runner caught SIGTERM. Stopping. Jan 30 05:46:09 2025 (1558) digest runner exiting. Jan 30 05:46:09 2025 (1557) virgin runner caught SIGTERM. Stopping. Jan 30 05:46:09 2025 (1557) virgin runner exiting. Jan 30 05:46:09 2025 (1556) task runner caught SIGTERM. Stopping. Jan 30 05:46:09 2025 (1556) task runner exiting. Jan 30 05:46:09 2025 (1541) Master watcher caught SIGTERM. Exiting. Jan 30 05:46:09 2025 (1555) retry runner caught SIGTERM. Stopping. Jan 30 05:46:09 2025 (1555) retry runner exiting. Jan 30 05:46:09 2025 (1552) out runner caught SIGTERM. Stopping. Jan 30 05:46:09 2025 (1552) out runner exiting. Jan 30 05:46:09 2025 (1549) command runner caught SIGTERM. Stopping. Jan 30 05:46:09 2025 (1549) command runner exiting. Jan 30 05:46:09 2025 (1548) bounces runner caught SIGTERM. Stopping. Jan 30 05:46:09 2025 (1548) bounces runner exiting. Jan 30 05:46:09 2025 (1547) archive runner caught SIGTERM. Stopping. Jan 30 05:46:09 2025 (1547) archive runner exiting. Jan 30 05:46:09 2025 (1550) in runner caught SIGTERM. Stopping. Jan 30 05:46:09 2025 (1550) in runner exiting. [2025-01-30 05:46:09 +0000] [1554] [INFO] Handling signal: term [2025-01-30 05:46:09 +0000] [2227] [INFO] Worker exiting (pid: 2227) [2025-01-30 05:46:09 +0000] [1585] [INFO] Worker exiting (pid: 1585) Jan 30 05:46:10 2025 (1541) Master watcher caught SIGTERM. Exiting. [2025-01-30 05:46:10 +0000] [1554] [ERROR] Worker (pid:1585) was sent SIGKILL! Perhaps out of memory? [2025-01-30 05:46:10 +0000] [1554] [ERROR] Worker (pid:2227) was sent SIGTERM! [2025-01-30 05:46:10 +0000] [1554] [INFO] Shutting down: Master Jan 30 05:46:11 2025 (1541) Master stopped ``` Thank you, Paul 'Arte Chambers' Robey 502-408-6922

3 13

The paradox of VERP probe messages
by Jeremy Stanley Feb. 2, 2025

Feb. 2, 2025

tl;dr: Is there a way to have VERP probe messages not include copies of the bounce message that triggered them? Longer form... Some months ago I turned on bounce processing for a fairly large list with a long history, because I could see from the MTA logs that there were easily hundreds of old subscribers whose addresses had become defunct and this seemed like a great way to clean that up and make the server more conscientious at not repeatedly trying to deliver messages to sites that were just going to reject them. This worked really well. Until legitimate subscribers also started to get removed, that is, due primarily to false positives from overzealous spam filters. I had previously read about MM3's VERP probe message feature, which seemed to solve the main reason I had kept bounce processing disabled in the MM2 days, and thought it was on by default but clearly it wasn't. So I found the necessary config option and switched that on. This worked really well. Until legitimate subscribers started to get removed again, that is, due primarily to false positives from overzealous spam filters. See, I realized that the VERP probes were helpfully including copies of the most recent bounce for that user, and those NDRs quite often helpfully included their own copies of the messages which had been bounced. So if a subscriber's MTA was rejecting a message because the content looked like spam, then there was a good chance it would also reject the VERP probe message because it included the same content which had caused the earlier rejection. The VERP probe feature was essentially defeating its own purpose as a neutral confirmation that the subscriber hadn't been rejecting messages due to their content. This behavior raises a more significant concern as well. The MTA on the Mailman server in this case (Exim if it matters) wants to perform batch deliveries when there are multiple subscribers who all share the same remote MTAs. If multiples of those reject a list post, then the NDR back to Mailman aggregates those multiple rejections into a single message. If that incident triggers a VERP probe, the bulk NDR enumerating the addresses of all the subscribers who were rejected gets forwarded along as part of the VERP probe, disclosing a potentially sensitive litany of random subscriber addresses to other subscribers within that batch. I don't know whether this is a misconfiguration, or an unintended consequence of the intersection of Mailman and MTA features. While I understand that the inclusion of a bounce sample within a VERP probe may be a way to take some workload off list admins in that users can possibly see a message/rejection to explain the reason for the probe, the downsides of that choice make the feature unfortunately unusable for some lists that could otherwise have benefited from it. Am I possibly missing an existing switch that could make VERP probe messages *not* include copies of the bounces that triggered them? This is in some ways similar to the problems I've noticed with message hold notifications to moderators including copies of the held messages, since these are (at least in the case of the lists I manage) often spam, so has a tendency to cause the mail providers for the list moderators to start considering the mailing list server a spam source. If there were a switch to make message hold notifications to moderators *not* include copies of the messages themselves, I'd start allowing moderator notifications again. Suggestions welcome! Also willing to try to submit patches to enable some of these solutions if they don't already exist. And thanks to anyone who read this far! -- Jeremy Stanley

2 4

Outgoing runner lockup? + "slice inspector" tool
by Stephen J. Turnbull Feb. 1, 2025

Feb. 1, 2025

Hi, I'm advising a user on a system which experiences occasional lockups in an outgoing runner. The runner contacts a remote smarthost, and occasionally locks up with the smtp connection still open according to "lsof -i :25". This state seems to be permanent: the backlog for that slice starts to grow without bound, for at least an hour. When this was happening once a month, they would just restart Mailman core, and the backlog would clear in minutes. But recently it's been happening daily which is distracting their staff, and worries me. Another thing that is strange about this site is that it should be possible to hit that runner with a SIGUSR1 and restart it. This works for me, but on that system the stuck runner exits, but does not restart. Since normally it only happens to one runner of several, I wanted to identify the runner and process. I attach the tool I developed, for anyone who might have configured multiple runners and is interested to see the distribution across runners. Has anybody seen an outgoing runner lock up with an open smtp session? Any ideas on why? My analysis so far: - Because a restart works every time, I'm pretty sure it doesn't have anything to do with message content. - I believe both the Mailman host and the outgoing smarthost are Linodes, in the same datacenter. The problematic Mailman system is Mailman 3.3.6, Python 3.10 on Ubuntu 16.xx LTS. I believe the smarthost is a more recent Ubuntu LTS, probably running Postfix 3.7 as the MTA. (Yes, they have sufficiently paranoid security and QA teams. ;-) - We're using smtplib, which as far as I can tell basically has a 60s timeout for each command. Thus you'd think it would time out. I guess it could be inflooping on timeout, retry, timeout, retry, but I don't know how to check that. Maybe ss(8) will serve? - My system where SIGUSR1 works as documented is Python 3.11.2 on a Digital Ocean droplet with Debian 12.9, Linux 6.1.0. - I haven't tried to reproduce the Python 3.10 + Mailman 3.3.6 configuration and test SIGUSR1 in that configuration yet. Seems unlikely, waiting for the proverbial "round tuit". - It has occurred to us to use the local Postfix as the relay MTA. I'm waiting on a report whether that alleviates the problem. Even if so, I want to fix the underlying defect if possible. Any ideas would be welcome, including general debugging advice. Here's the promised slice_inspector tool. (Patches and suggestions welcome, though I don't promise to implement any time soon.) The tool imports some files from Mailman core, click, and psutil (not psutils!) I believe the latter modules are required by Mailman, so you should be able to run it as is under the 'mailman' user (or perhaps 'list' on Debian) in the environment Mailman itself uses. It defaults to assuming that mailman.cfg is /etc/mailman3/mailman.cfg. You'll probably need to fix the shebang if you want to chmod +x. The '--help' should be pretty self-explanatory. Steve

2 1

messages with attachments, without text discarded silently
by Schulz Feb. 1, 2025

Feb. 1, 2025

Using the current mailman version, I can reproduce the following: - send a text-formatted email with attachment but without text body to a list. - the msg will be received by mailman - the msg shows up in shunt, the log shows: ile "/usr/lib/python3.12/email/message.py", line 343, in set_payload payload = payload.encode(charset.output_charset, 'surrogateescape') ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ UnicodeEncodeError: 'ascii' codec can't encode characters in position 10-17: ordinal not in range(128) an attempt to unpickle the file leads to the same error. I know it's uncommon to send emails without txt but with attachments - but it may happen; at least we would welcome an error message.

2 3