June 2023 - Mailman-users - lists.mailman3.org

Digests not working correctly
by Joel Lord 23 Jun '23

23 Jun '23

So I seem to have problems with both periodic AND size-triggered digests. Size first, this is simpler. Just got one sent out, clearly triggered by a new email getting added to the queued up messages, but the setting for the list is the default 30KB and the digest came out at 23.2 KB. Is it measuring the accumulated queue as size on disk including metadata and filesystem block sizes or something, rather than the size of what the emailed digest will be? Periodic is much more complicated, even to explain. I have the cron job running nightly at 11:00 PM and it certainly seems to be working, except for one really odd thing. The list also definitely has periodic digests enabled in the web UI. This is a very bursty list, so it will go mad with traffic for a few weeks and then will go mostly idle for a month or so. The first digest of a burst is definitely size-triggered, but pretty reliably it has the last few messages from the previous burst in there. So the one I got today had three messages from today and one message from May 4th. But that May 4th message should have gone out at 11:00 PM on May 4th with the daily digest and then gotten flushed from the pending digest queue, right? -- Joel Lord

2 12

Importing mbox file into MM3
by Mark 23 Jun '23

23 Jun '23

Hi All I'm trying to import a large mbox file from a MM2 list into a new list on MM3. The new MM3 list has 3 test emails in its archive. # cd /usr/share/mailman3-web/ # python3 manage.py hyperkitty_import -l LISTNAME@DOMAIN /home/mark/tmp/LISTNAME.mbox Importing from mbox file /home/mark/tmp/LISTNAME.mbox to LISTNAME@DOMAIN Computing thread structure Synchronizing properties with Mailman Warming up cache The full-text search index is not updated for this list. It will not be updated by the 'minutely' incremental update job. To update the index for this list, run the Django admin command with arguments 'update_index_one_list LISTNAME@DOMAIN'. That all seems pretty good. Then I attempt to rebuild the archive. # python3 manage.py update_index_one_list LISTNAME@DOMAIN Indexing 3 emails The import or rebuild has failed as it's just rebuilding the existing archive of the 3 test messages. What am I doing wrong? Thanks Mark

2 2

Where in database are subject prefix sequences stored?
by Thomas Ward 22 Jun '23

22 Jun '23

I am using numeric prefixes for a list, such that it's `[ListName-000012]`. Where would that sequence indicator be stored in the database? Postorious' database? Hyperkitty's database? I ask because I'm rebuilding a home-grown listserv solution using Mailman3 and the main requirement is the thread IDs being *properly sequential* with the current implementation, so I have to set the "next ID in sequence" to a start value. I don't see this clearly in the DB where the sequence value would be stored, so if anyone knows that'd be great to know so I can get this working. Thomas

2 2

Help needed to create filters that were working in Mailman 2.
by mrl＠psfc.mit.edu 22 Jun '23

22 Jun '23

How do I make a filter that if it matches, it accepts the message and doesn't check the remaining filters? I could do this using the anti-spam filters in mailman 2, as there was a Accept option. But there is no Accept option in Mailman 3 (that I can find). I tried using "Default antispam action", but that didn't stop the rest of the filters from being applied. Thanks for any suggestions. - Mark

3 4

toward improving correctness when running on single core system ...
by Nelson Strother 21 Jun '23

21 Jun '23

I recognize that most real computer systems now are equipped with multiple CPUs, usually with each CPU having multiple cores or hardware threads. But when one obtains a computer system from a service provider with a small expected load, it is likely that a virtual computer system will provide only a single core. Yes, one can obtain a multi-core virtual system, maybe for only twice the cost; but it would seem wasteful to make that choice solely to support starting and stopping the Mailman3 program when the system load at all other times is satisfied by a single core system. From my observations, mailman 3.3.8 is unreliable when running on a single core system. When `mailman start` is issued, there is a moderately high probability that one or more of the thirteen runner processes will not survive this startup phase. Depending upon the current usage of the mailman installation, one may or may not notice this fact. In either case, `mailman status` provides the same information, with no hint of missing or damaged capabilities when one or more of the runner processes are stillborn. [1] One potential improvement would be to increase the lifetime of the locks used by each of the runner processes during `mailman start`. [2] Based on a small sample of runs of `mailman start`, this change may provide correctness (i.e. I have yet to see a runner process go MIA). But it still seems unsatisfactory for availability / performance, as the elapsed wall clock time for the system to settle down and be functional and responsive is approximately 13 minutes. While this is an improvement from the 20 minute or greater delays observed before making this change, it still differs greatly from the nearly instantaneous startup time experienced on multicore / multiprocessor systems. The best "performance" I have yet obtained on this single core system is WITHOUT the change in [2] below, but by crossing abstraction boundaries and reducing the amount of time this single CPU spends futilely busy waiting for itself when it has not provided an opportunity for another process (maybe even the process currently holding the desired lock) to make any progress. [3] With this change, "only" 9 minutes elapsed wall clock time is wasted (with maybe only the middle 4 minutes at 100% CPU utilization) before the system settles down and is functional and responsive. However, ample room for improvement remains. [4] Please suggest better ideas or corrections. Nelson [1] For a few examples of these failures, see: https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/thread/… [2] Here is a diff from the original mailman/database/factory.py (or .../mailman/.local/lib/python3.9/site-packages/mailman/database/factory.py if you wish to apply this to an installed system): > 25a26 > > from datetime import timedelta > 53c54,55 > < with Lock(os.path.join(config.LOCK_DIR, 'dbcreate.lck')): > --- > > with Lock(os.path.join(config.LOCK_DIR, 'dbcreate.lck'), > > timedelta(seconds=30)): which increases the delay from the default of 15 seconds to 30 seconds before another process will be empowered to break the lock being held by one of mailman's runner processes, which leads to the death of a process. [3] Here is a diff from the original .../mailman/.local/lib/python3.9/site-packages/flufl/lock/_lockfile.py : > 7a8 > > import subprocess > 338a340,344 > > if sys.platform.startswith('linux') and \ > > (len(subprocess.check_output(["lscpu", "-p"]).splitlines()) == 5): > > # there is only one processor [four lines of comments, and only one line for one CPU] > > # let another process run before joining the traffic jam for this lock > > os.system('sleep 20s') For possible similar benefits on other platforms, one should also include variations on this theme using something like e.g.: > > if sys.platform.startswith('win32') and \ > > subprocess.check_output('wmic path win32_Processor get NumberOfLogicalProcessors').strip().endswith(b'1'): I seriously doubt all (or even any?) other users of the flufl.lock package would benefit from this exact change ... but many may benefit from sleeping for e.g. a half-second when on a single core? In the case of Mailman3, sleeping for a fixed duration shorter than 20 seconds retains the original problem of broken locks and runner process death, while sleeping for a fixed duration longer than 20 seconds leads to an increase in both the wall clock delay time as well as the CPU time consumed during `mailman start`. I have not experimented with random or staggered delays, which may further reduce the number of simultaneously blocked waiters for this lock. [4] I have seen `mailman start`complete in less than 1 second wall clock time on a multicore / multiprocessor system. All of these observations made with mailman 3.3.8 as installed via pip.

5 8

How to tell who moderated a post?
by bryan.kartzman＠yu.edu 21 Jun '23

21 Jun '23

Is there any way to tell which moderator of a list approved a particular post to that list?

2 3

Re: mailman3 auto-disabling of subscriptions
by Mark Sapiro 19 Jun '23

19 Jun '23

On 6/17/23 17:02, Mike Wertheim wrote: > I set `remove_dkim_headers: yes` in the [mta] section of mailman.cfg. > > I'm seeing a bunch of these errors in the logs: > 2023-06-17 18:57:11 1qAbMA-005Y7W-HP ** mike.wertheim(a)gmail.com > <mailto:mike.wertheim@gmail.com> R=dnslookup_single_domain > T=remote_smtp_single_domain H=alt2.gmail-smtp-in.l.google.com > <http://alt2.gmail-smtp-in.l.google.com> [172.253.62.26]: SMTP error > from remote mail server after end of data: 550-5.7.26 Unauthenticated > email from gmail.com <http://gmail.com> is not accepted due to > domain's\n550-5.7.26 DMARC policy. Please contact the administrator of > gmail.com <http://gmail.com> domain\n550-5.7.26 if this was a legitimate > mail. Please visit\n550-5.7.26 > https://support.google.com/mail/answer/2451690 > <https://support.google.com/mail/answer/2451690> to learn about the\n550 > 5.7.26 DMARC initiative. > w13-20020a05622a190d00b003fde5babbf5si1170418qtc.14 - gsmtp Your issue is mail From: the gmail.com domain to the list and sent to gmail users is being bounced because it fails DMARC. I.e. Gmail applies a DMARC policy of p=reject to mail From: gmail.com even though it's published DMARC policy is p=none. You need to apply DMARC mitigations to this mail. Unfortunately, this requires setting DMARC Mitigate unconditionally to Yes. Mailman 2.1 has a feature to apply DMARC mitigation to messages From: specific domains regardless of any domain specific DMARC Policy. Unfortunately this hasn't been implemented in Mailman 3 yet. I just filed https://gitlab.com/mailman/mailman/-/issues/1084 for that. -- Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

2 1

Where does sender postorius@ come from?
by Jens Günther 19 Jun '23

19 Jun '23

I just realized, that mails for registering an account and/or resetting ones password are obviously sent from postorius@SERVE_FROM_DOMAIN. Why are they not sent from MAILMAN_ADMIN_EMAIL? The problem is: - an account is trying to register with a mailaddress from a HostEurope-server - HostEurope-Servers are doing Sender-Callouts (bad practise :-D) to verify that a mailaccount exists - mail is then bouncing, b/c those are not answered (shoud I?): status=bounced (host [80.237.138.5] said: 550-Sender verification is required but failed. - bounce then can't be processed, since mailaccount does not exist :-D status=bounced (mail for SERVE_FROM_DOMAIN loops back to myself) Am I missing something are shouldn't it be better to send those mails from MAILMAN_ADMIN_EMAIL? (I don't want to mess with these sender callouts or change anything, HostEurope has been a little bit of a pain in the ... with mailinglists and not much help at all). Thanks in advance, Jens.

3 8

Proxy error.
by Mark London 18 Jun '23

18 Jun '23

My server has several aliases. And Chrome's cookies sometimes get messed up because of it. Several users get a proxy error when trying to connect to the mailman website. One person just fixed it by clearling cookies, although this is not a great solution. Error log says "You may need to add 'psfcmail2.psfc.mit.edu' to ALLOWED_HOSTS". But it is already there in /etc/mailman/settings.py Or should it go elsewhere? Thanks. - Mark Chrome: Proxy Error The proxy server received an invalid response from an upstream server. The proxy server could not handle the request Reason: Error reading from remote server Apache/2.4.52 (Ubuntu) Server at lists.psfc.mit.edu Port 443Error log: ERROR 2023-06-18 19:11:11,568 2114 django.security.DisallowedHost Invalid HTTP_HOST header: 'psfcmail2.psfc.mit.edu'. You may need to add 'psfcmail2.psfc.mit.edu' to ALLOWED_HOSTS. Traceback (most recent call last): File "/opt/mailman/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 56, in inner response = get_response(request) File "/opt/mailman/venv/lib/python3.10/site-packages/django/utils/deprecation.py", line 135, in __call__ response = self.process_request(request) File "/opt/mailman/venv/lib/python3.10/site-packages/django/middleware/common.py", line 48, in process_request host = request.get_host() File "/opt/mailman/venv/lib/python3.10/site-packages/django/http/request.py", line 152, in get_host raise DisallowedHost(msg) django.core.exceptions.DisallowedHost: Invalid HTTP_HOST header: 'psfcmail2.psfc.mit.edu'. You may need to add 'psfcmail2.psfc.mit.edu' to ALLOWED_HOSTS

1 1

mailman3 auto-disabling of subscriptions
by Mike Wertheim 17 Jun '23

17 Jun '23

I'm running an email list on mailman3. My users are getting inundated by messages saying that their subscriptions have been disabled due to their bounce score exceeding the mailing list's bounce_score_threshold. But there is nothing wrong with these users' email accounts. They are all perfectly fine. Apparently the issue is somehow related to spam. Whenever I (as admin) get these emails, I just unsubscribe the user and then re-subscribe them. But also, the user gets confused and emails me about it, which unnecessarily takes up their time and mine. The relevant list settings are currently... bounce score threshold: 5 bounce info stale after: 7d bounce disabled warning interval: 7d bounce disable warnings: 5 What's the best way to fix this? I'm considering reducing "bounce info stale after" to "2d" and/or increasing "bounce disabled warning interval" to "90d".

3 5