- Mailman-users - lists.mailman3.org

re-injecting emails that were not archived
by Ken Alker June 25, 2023

June 25, 2023

I have just migrated from a mailman 3.2.2 Debian package installation to a mailman 3.3.8 source install (in a Python virtual environment as mailman user) on an Ubuntu 20.04.6 LTS system. Several months ago, the system had been upgraded from mailman V2 to mailman V3.2.2. I noticed when I came onto the project last week that there were 140 messages stuck in the "shunt" queue for the two lists hosted by this server (I have no idea why and the old logs were not available and I don't recall the results of "mailman qfile", assuming that had shed some light on the reason why they were shunted). In any event, I unshunted these messages and they flowed out to the subscriber's email addresses successfully. What I was not expecting is that the system did not inject any of these emails into the archive. AFAICT, the messages are no longer anywhere on the system (but I could be mistaken). Here are my questions: 1) Why would these emails not have been injected into the archive? 2) Is there some way for me to re-inject these messages into the two lists? The only place I (THINK I) have them is in my own personal IMAP box since I'm a subscriber to the lists. 3) Can this be done without the emails being re-delivered to subscribers? Thank you, Ken

2 1

Re: migratng from 3.2.2 Debian packages to 3.3.8 source install - web server question
by Ken Alker June 25, 2023

June 25, 2023

As an aside, for anyone else who is trying to do the same (and to give some background into my procedures so far), I have been following these references: <https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/thread/…> <https://docs.mailman3.org/en/latest/install/virtualenv.html#setup-virtualenv> My goal is to create the least change to file locations and configuration files as possible (I may move things into more "source-installed centric" locations once I have things working again), thus, many files are still in the original Debian package-allocated directories. That said, I was forced to move mailman3web.db to /opt/mailman/web as sqlite (and, hence, mailman-web) simply could not open the database as user mailman from my virtual environment (even with mailman3web.db file permissions set to 777 and owner.group set to mailman.mailman). So far I've gotten mailman running properly and handling the mailing lists, however, I have not gotten the web interface working again.

2 6

Importing MBOX from custom listserv to MM3?
by Thomas Ward June 25, 2023

June 25, 2023

We are looking at moving from a home grown listserv system at a client to potentially MailMan3 for the lists platform instead. We have historic thread data from the homegrown systems in mbox form. Is there a way to import that into MM3 for indexing in the archiver there so we dont need to keep the (literally massive) MBOX file? Sent from my Galaxy

2 2

Importing mbox file into MM3
by Mark June 23, 2023

June 23, 2023

Hi All I'm trying to import a large mbox file from a MM2 list into a new list on MM3. The new MM3 list has 3 test emails in its archive. # cd /usr/share/mailman3-web/ # python3 manage.py hyperkitty_import -l LISTNAME@DOMAIN /home/mark/tmp/LISTNAME.mbox Importing from mbox file /home/mark/tmp/LISTNAME.mbox to LISTNAME@DOMAIN Computing thread structure Synchronizing properties with Mailman Warming up cache The full-text search index is not updated for this list. It will not be updated by the 'minutely' incremental update job. To update the index for this list, run the Django admin command with arguments 'update_index_one_list LISTNAME@DOMAIN'. That all seems pretty good. Then I attempt to rebuild the archive. # python3 manage.py update_index_one_list LISTNAME@DOMAIN Indexing 3 emails The import or rebuild has failed as it's just rebuilding the existing archive of the 3 test messages. What am I doing wrong? Thanks Mark

2 2

Where in database are subject prefix sequences stored?
by Thomas Ward June 22, 2023

June 22, 2023

I am using numeric prefixes for a list, such that it's `[ListName-000012]`. Where would that sequence indicator be stored in the database? Postorious' database? Hyperkitty's database? I ask because I'm rebuilding a home-grown listserv solution using Mailman3 and the main requirement is the thread IDs being *properly sequential* with the current implementation, so I have to set the "next ID in sequence" to a start value. I don't see this clearly in the DB where the sequence value would be stored, so if anyone knows that'd be great to know so I can get this working. Thomas

2 2

Help needed to create filters that were working in Mailman 2.
by mrl＠psfc.mit.edu June 22, 2023

June 22, 2023

How do I make a filter that if it matches, it accepts the message and doesn't check the remaining filters? I could do this using the anti-spam filters in mailman 2, as there was a Accept option. But there is no Accept option in Mailman 3 (that I can find). I tried using "Default antispam action", but that didn't stop the rest of the filters from being applied. Thanks for any suggestions. - Mark

3 4

toward improving correctness when running on single core system ...
by Nelson Strother June 21, 2023

June 21, 2023

I recognize that most real computer systems now are equipped with multiple CPUs, usually with each CPU having multiple cores or hardware threads. But when one obtains a computer system from a service provider with a small expected load, it is likely that a virtual computer system will provide only a single core. Yes, one can obtain a multi-core virtual system, maybe for only twice the cost; but it would seem wasteful to make that choice solely to support starting and stopping the Mailman3 program when the system load at all other times is satisfied by a single core system. From my observations, mailman 3.3.8 is unreliable when running on a single core system. When `mailman start` is issued, there is a moderately high probability that one or more of the thirteen runner processes will not survive this startup phase. Depending upon the current usage of the mailman installation, one may or may not notice this fact. In either case, `mailman status` provides the same information, with no hint of missing or damaged capabilities when one or more of the runner processes are stillborn. [1] One potential improvement would be to increase the lifetime of the locks used by each of the runner processes during `mailman start`. [2] Based on a small sample of runs of `mailman start`, this change may provide correctness (i.e. I have yet to see a runner process go MIA). But it still seems unsatisfactory for availability / performance, as the elapsed wall clock time for the system to settle down and be functional and responsive is approximately 13 minutes. While this is an improvement from the 20 minute or greater delays observed before making this change, it still differs greatly from the nearly instantaneous startup time experienced on multicore / multiprocessor systems. The best "performance" I have yet obtained on this single core system is WITHOUT the change in [2] below, but by crossing abstraction boundaries and reducing the amount of time this single CPU spends futilely busy waiting for itself when it has not provided an opportunity for another process (maybe even the process currently holding the desired lock) to make any progress. [3] With this change, "only" 9 minutes elapsed wall clock time is wasted (with maybe only the middle 4 minutes at 100% CPU utilization) before the system settles down and is functional and responsive. However, ample room for improvement remains. [4] Please suggest better ideas or corrections. Nelson [1] For a few examples of these failures, see: https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/thread/… [2] Here is a diff from the original mailman/database/factory.py (or .../mailman/.local/lib/python3.9/site-packages/mailman/database/factory.py if you wish to apply this to an installed system): > 25a26 > > from datetime import timedelta > 53c54,55 > < with Lock(os.path.join(config.LOCK_DIR, 'dbcreate.lck')): > --- > > with Lock(os.path.join(config.LOCK_DIR, 'dbcreate.lck'), > > timedelta(seconds=30)): which increases the delay from the default of 15 seconds to 30 seconds before another process will be empowered to break the lock being held by one of mailman's runner processes, which leads to the death of a process. [3] Here is a diff from the original .../mailman/.local/lib/python3.9/site-packages/flufl/lock/_lockfile.py : > 7a8 > > import subprocess > 338a340,344 > > if sys.platform.startswith('linux') and \ > > (len(subprocess.check_output(["lscpu", "-p"]).splitlines()) == 5): > > # there is only one processor [four lines of comments, and only one line for one CPU] > > # let another process run before joining the traffic jam for this lock > > os.system('sleep 20s') For possible similar benefits on other platforms, one should also include variations on this theme using something like e.g.: > > if sys.platform.startswith('win32') and \ > > subprocess.check_output('wmic path win32_Processor get NumberOfLogicalProcessors').strip().endswith(b'1'): I seriously doubt all (or even any?) other users of the flufl.lock package would benefit from this exact change ... but many may benefit from sleeping for e.g. a half-second when on a single core? In the case of Mailman3, sleeping for a fixed duration shorter than 20 seconds retains the original problem of broken locks and runner process death, while sleeping for a fixed duration longer than 20 seconds leads to an increase in both the wall clock delay time as well as the CPU time consumed during `mailman start`. I have not experimented with random or staggered delays, which may further reduce the number of simultaneously blocked waiters for this lock. [4] I have seen `mailman start`complete in less than 1 second wall clock time on a multicore / multiprocessor system. All of these observations made with mailman 3.3.8 as installed via pip.

5 8

How to tell who moderated a post?
by bryan.kartzman＠yu.edu June 21, 2023

June 21, 2023

Is there any way to tell which moderator of a list approved a particular post to that list?

2 3

Re: mailman3 auto-disabling of subscriptions
by Mark Sapiro June 19, 2023

June 19, 2023

On 6/17/23 17:02, Mike Wertheim wrote: > I set `remove_dkim_headers: yes` in the [mta] section of mailman.cfg. > > I'm seeing a bunch of these errors in the logs: > 2023-06-17 18:57:11 1qAbMA-005Y7W-HP ** mike.wertheim(a)gmail.com > <mailto:mike.wertheim@gmail.com> R=dnslookup_single_domain > T=remote_smtp_single_domain H=alt2.gmail-smtp-in.l.google.com > <http://alt2.gmail-smtp-in.l.google.com> [172.253.62.26]: SMTP error > from remote mail server after end of data: 550-5.7.26 Unauthenticated > email from gmail.com <http://gmail.com> is not accepted due to > domain's\n550-5.7.26 DMARC policy. Please contact the administrator of > gmail.com <http://gmail.com> domain\n550-5.7.26 if this was a legitimate > mail. Please visit\n550-5.7.26 > https://support.google.com/mail/answer/2451690 > <https://support.google.com/mail/answer/2451690> to learn about the\n550 > 5.7.26 DMARC initiative. > w13-20020a05622a190d00b003fde5babbf5si1170418qtc.14 - gsmtp Your issue is mail From: the gmail.com domain to the list and sent to gmail users is being bounced because it fails DMARC. I.e. Gmail applies a DMARC policy of p=reject to mail From: gmail.com even though it's published DMARC policy is p=none. You need to apply DMARC mitigations to this mail. Unfortunately, this requires setting DMARC Mitigate unconditionally to Yes. Mailman 2.1 has a feature to apply DMARC mitigation to messages From: specific domains regardless of any domain specific DMARC Policy. Unfortunately this hasn't been implemented in Mailman 3 yet. I just filed https://gitlab.com/mailman/mailman/-/issues/1084 for that. -- Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

2 1

Where does sender postorius@ come from?
by Jens Günther June 19, 2023

June 19, 2023

I just realized, that mails for registering an account and/or resetting ones password are obviously sent from postorius@SERVE_FROM_DOMAIN. Why are they not sent from MAILMAN_ADMIN_EMAIL? The problem is: - an account is trying to register with a mailaddress from a HostEurope-server - HostEurope-Servers are doing Sender-Callouts (bad practise :-D) to verify that a mailaccount exists - mail is then bouncing, b/c those are not answered (shoud I?): status=bounced (host [80.237.138.5] said: 550-Sender verification is required but failed. - bounce then can't be processed, since mailaccount does not exist :-D status=bounced (mail for SERVE_FROM_DOMAIN loops back to myself) Am I missing something are shouldn't it be better to send those mails from MAILMAN_ADMIN_EMAIL? (I don't want to mess with these sender callouts or change anything, HostEurope has been a little bit of a pain in the ... with mailinglists and not much help at all). Thanks in advance, Jens.

3 8