I'm working on importing nearly 2000 lists and have been doing some testing prior to migrating from Mailman2.
I'm using the mailman create command via for loop. It takes quite a while to do all the work including updating MySQL and re-generating the LMTP files for each list. As the number of lists increases the process gets slower.
Do you have any recommendations for speeding up this process? I am running the system in AWS using EFS for storage of the LMTP files.
Thanks!
-- Dan Caballero Caltech
On Thu, Oct 7, 2021, at 11:23 AM, dancab@caltech.edu wrote:
I'm working on importing nearly 2000 lists and have been doing some testing prior to migrating from Mailman2.
I'm using the mailman create command via for loop. It takes quite a while to do all the work including updating MySQL and re-generating the LMTP files for each list. As the number of lists increases the process gets slower.
Do you have any recommendations for speeding up this process? I am running the system in AWS using EFS for storage of the LMTP files.
There isn't a way to disable creation of LMTP files, but if this instance you are running on isn't live, then you can configure Mailman with NullMTA so it won't try to generate LMTP files.
[mta]
# NullMTA is just implementing the interface and thus satisfying Mailman
# without doing anything fancy
outgoing: mailman.mta.null.NullMTAYou need to add ^ to mailman.cfg, commenting out the _real_ outgoing MTA configuration.
Once you are done with the migration, you can configure the right MTA
and run mailman aliases command to generate them.
-- thanks, Abhilash Raj (maxking)
Thank you. I will test this on my "dev/test" system to see how the performance improves for the mass import. We are also using the API for creating "new" lists and notice there's significant delay for creating just one list due to the nearly 2000 existing lists.
We have noticed similar slowness when importing/creating/deleting lists. We have ~2600 lists and the bottleneck does seem to be the MTA alias code that reads from backend (sorts by domain and then listnames?) and subsequently writes the (lmtp,vmap,etc) files.
We're running AWS aurora serverless DB on the backend and have split core/web hosts. Our setup is a bit different than most on this list probably, but for now we've resorted to setting:
incoming: mailman.mta.null.NullMTA
on the web boxes (not core) and actually rebuild the aliases every X minutes (X=15 for now) in a rundeck job. Testing the creation/deletion of lists now take ~5 seconds vs 3+minutes when Mailman was trying to re-generate the aliases on every create/delete. An obvious downside is the X delta out of sync-ness between Mailman and your MTA.
Bryan Loniewski Rutgers University OIT:EAS:Messaging and Collaboration Services System Administrator
From: dancab@caltech.edu <dancab@caltech.edu> Sent: Tuesday, October 19, 2021 1:41 PM To: mailman-users@mailman3.org Subject: [MM3-users] Re: Mailman import of 2000 lists
Thank you. I will test this on my "dev/test" system to see how the performance improves for the mass import. We are also using the API for creating "new" lists and notice there's significant delay for creating just one list due to the nearly 2000 existing lists.
Mailman-users mailing list -- mailman-users@mailman3.org To unsubscribe send an email to mailman-users-leave@mailman3.org https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.mail...
Bryan Loniewski writes:
We have noticed similar slowness when importing/creating/deleting lists. We have ~2600 lists and the bottleneck does seem to be the MTA alias code that reads from backend (sorts by domain and then list names?) and subsequently writes the (lmtp,vmap,etc) files.
Is there a GitLab issue for this? If not, will somebody file one for us? I'd do it but I don't really understand the issue and diagnosis except for S-L-O-W, that part I get.
We're running AWS aurora serverless DB on the backend and have split core/web hosts.
For example, "is this relevant to the problem, or just to your workaround?" is a question I can't answer myself without a lot more work than I can afford to do for a week or so.
On Oct 19, 2021, at 7:37 PM, Stephen J. Turnbull <stephenjturnbull@gmail.com> wrote:
Bryan Loniewski writes:
We have noticed similar slowness when importing/creating/deleting lists. We have ~2600 lists and the bottleneck does seem to be the MTA alias code that reads from backend (sorts by domain and then list names?) and subsequently writes the (lmtp,vmap,etc) files.
Is there a GitLab issue for this? If not, will somebody file one for us? I'd do it but I don't really understand the issue and diagnosis except for S-L-O-W, that part I get.
I didn’t find any issue, so I created one
https://gitlab.com/mailman/mailman/-/issues/953
My understanding of the issue is that Mailman tries to create the LMTP files whenever a new list is created and in their setup, the LMTP files are on a remote volume. So the latency to write to LMTP file is significant. And when 2600 of those happen, it takes a lot of time.
I don’t think we can do anything about the MySQL latency though, since we have to create the lists. Although, these lists are separate rows in database, so it should be possible to do multiple imports in parallel by invoking the commands simultaneously.
We should be able to fix this by adding a flag that skips creation of LMTP files after creation/import of a list. Once the imports are complete, a single run of "mailman aliases” command should generate everything. Hyperkitty has a similar mechanism to avoid indexing emails when bulk importing emails from mboxes.
-- thanks, Abhilash Raj (maxking)
"We should be able to fix this by adding a flag that skips creation of LMTP files after creation/import of a list."
Would that flag exist in the API? It would be good to have that flexibility. I don't mind managing mailman aliases via cron if needed.
participants (4)
-
Abhilash Raj -
Bryan Loniewski -
dancab＠caltech.edu -
Stephen J. Turnbull