moving lists to another server
I have both my mailman lists and django config in postgres databases. If I want to move my lists to another server that is running mailman 3, aside from copying the databases to the other server, what other things (perhaps all files under /var/lib/mailman3?, etc) would I need to bring over to the other server?
On Tue, Nov 7, 2023 at 12:28 AM <bryan.kartzman@yu.edu> wrote:
I have both my mailman lists and django config in postgres databases. If I want to move my lists to another server that is running mailman 3, aside from copying the databases to the other server, what other things (perhaps all files under /var/lib/mailman3?, etc) would I need to bring over to the other server?
For Mailman3, here we advocate for the virtualenv[0] method of installation, which uses /opt/mailman. The configs are in /etc/mailman3 and the data is mostly in /opt/mailman/mm/var so you'd need this directory moved over, IIRC.
[0] https://docs.mailman3.org/en/latest/install/virtualenv.html#virtualenv-insta...
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 "Oh, the cruft.", egrep -v '^$|^.*#' ¯\_(ツ)_/¯ :-) [How to ask smart questions: http://www.catb.org/~esr/faqs/smart-questions.html]
Thanks for the response, Odhiambo. I think the equivalent of /opt/mailman/mm/var for debian is the /var/lib/mailman3 directory. And of couirse the config files under /etc/mailman3
From: Odhiambo Washington <odhiambo@gmail.com> Sent: Monday, November 6, 2023 11:36 PM To: Bryan Kartzman <bryan.kartzman@yu.edu> Cc: mailman-users@mailman3.org <mailman-users@mailman3.org> Subject: [EXT] - Re: [MM3-users] moving lists to another server
**External Email**
On Tue, Nov 7, 2023 at 12:28 AM <bryan.kartzman@yu.edu<mailto:bryan.kartzman@yu.edu>> wrote: I have both my mailman lists and django config in postgres databases. If I want to move my lists to another server that is running mailman 3, aside from copying the databases to the other server, what other things (perhaps all files under /var/lib/mailman3?, etc) would I need to bring over to the other server?
For Mailman3, here we advocate for the virtualenv[0] method of installation, which uses /opt/mailman. The configs are in /etc/mailman3 and the data is mostly in /opt/mailman/mm/var so you'd need this directory moved over, IIRC.
[0] https://docs.mailman3.org/en/latest/install/virtualenv.html#virtualenv-insta...
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 "Oh, the cruft.", egrep -v '^$|^.*#' ¯\_(ツ)_/¯ :-) [How to ask smart questions: http://www.catb.org/~esr/faqs/smart-questions.html]
bryan.kartzman@yu.edu writes:
I have both my mailman lists and django config in postgres databases.
These are Mailman 3 lists? What Django config? Do you mean Postorius and HyperKitty? Do you intend to use Postorius and HyperKitty on the new server?
If I want to move my lists to another server that is running mailman 3, aside from copying the databases to the other server,
Do you mean the other server is already actively distributing Mailman list traffic, managing users and archives? Or is it fresh instance?
what other things (perhaps all files under /var/lib/mailman3?, etc) would I need to bring over to the other server?
That's going to vary depending on the way your existing server is set up. A Mailman 3 instance is a complex system of cooperating applications. Assuming you have not patched or added any Python code, typical configuration and customizations include
- Your webserver (Apache, nginx are most common) needs configuration to handle WSGI applications and may need modules enabled or installed for the WSGI server. This is usually installed as an OS distro package, and the configurations will live in /etc/apache2 or similar for another webserver. Depending on other content served by the two hosts you may have to edit the configuration on the new host rather than copy it from the old one.
- The WSGI server is typically a separate application, although there is an Apache module for this. Gunicorn is a Python application usually installed via pip. I like to keep its Django configuration, which I call gunicorn.conf.py, in /etc/mailman3. (Mailman core also uses gunicorn, but that is transparent to you, it should not need custom configuration.) uwsgi is a C or C++ application normally supplied as a distro package. Don't know where it keeps its config files but since it's a distro package my first guess is /etc/uwsgi.
- Your MTA, usually Postfix or Exim4, maybe Sendmail or qmail needs to be configured to route mail to Mailman, and apply any milters, signatures, and perhaps anti-yahoo configurations such as ARC.
- Your database server, usually PostgreSQL or MySQL.
- Mailman core has a single configuration file mailman.cfg, which I keep in /etc/mailman3. It also has a crontab, which in Debian- derived systems typically lives in the file /etc/cron.d/mailman3. Note that this could be implmented as a systemd timer, but I don't know if any distros do that.
- Django, Postorius, and HyperKitty together constitute the mailman-web subsystem. I strongly recommend you use the mailmanweb Python package to manage Django and its applications. In any case I am not familiar with other organizations of those packages, so I'm going to assume it. If you are not using mailmanweb, you will need to be careful because most subpackages in the Django subsystem name their configuration files settings.py, urls.py, and views.py just like top-level, and distinguish them by using Python's attribute notation (eg, postorius.urls). a. Django application configuration for Postorius and HyperKitty goes in settings.py, which I put in /etc/mailman3. b. Django site configuration for Postorius and HyperKitty goes in urls.py (the routing configuration mapping URLs to views) and views.py (mapping views to Python code). I put these in /etc/mailmn3. c. Full-text indexing for HyperKitty. Default is Whoosh (a Python application) whose configuration goes in Django's settings.py. I prefer Xapian, which requires a separate application (Debian and Ubuntu have packages), but the needed configuration can be done in the top-level settings.py.
- Template customization. If you have any customized templates for Mailman's email or Django's views, you'll need to copy and move them too. I haven't customized email, but Django's page templates can be kept separately with a path variable set in settings.py. I put them in a hierarchy under /etc/mailman3/templates. Maybe Mailman core has a similar setting?
- Transient files. Aside from code and configuration, everything in Django is in the RDBMS, there are be no random files strewn around as far as I know. Mailman core is another matter. You can hope that all Mailman transient files are in its "var" hierarchy. $var_dir (the setting in mailman.cfg or inherited from schema.cfg) is the root of this hierarchy. I set var_dir=/opt/mailman/var, I think Mark may use var_dir=/opt/mailman/mm/var. I would guess that FHS-conformant distros would likely use var_dir=/var/lib/mailman3. a. The queue directories $var_dir/queue/* should be empty. Shut down the incoming mail and let mailman run until it's clear. (Normally takes < 5 seconds.) The exceptions are the shunt and bad queues, which are messages that Mailman doesn't know what to do with. With the venv active, run "mailman unshunt" to attempt to clear the shunt queue. Move anything that's left in shunt as well as any .bak or .tmp files into the bad queue for later treatment. These can be copied to the bad queue in the new instance. Use "mailman qfile <file>" to examine them to see if they can be deleted. b. I think the only stuff that ends up in the $var_dir/lists/* directories are digests being built for later distribution. Use the usual tools to send them off or copy them over to the new instance. c. There should be nothing in $var_dir/archives/hyperkitty/spool, which is temporary storage for messages to be archived. That is, it should get cleared up by the few seconds while incoming mail is stopped and outgoing messages are being processed. If there's anything still there, stash it away for later examination using "mailman qfile". (Don't put it in queue/bad or queue/archive though, this is a different process from the normal pipeline.)
I think that's about it.
Steve
Wow, thanks for the very thorough response, Steve.
I should have started by saying that when we built the current production server, we cloned it off the test server and we were thinking now to move the data onto another clone of the old test server. So, yes we would have to apply all the changes we made since we cloned the current production server, but everything else as far as the OS is concerned should be comparable (webserver, MTA, Mailman core, Django, Hyperkitty, Postorius). That's why I was just asking about the "data" aspects of mailman/hyperkitty/postorius. What would I have to move to replicate all the lists on the current production server to a new clone of the original test server once we've applied any changes we made to production server since the clone.
From: Stephen J. Turnbull <turnbull.stephen.fw@u.tsukuba.ac.jp> Sent: Tuesday, November 7, 2023 1:07 PM To: Bryan Kartzman <bryan.kartzman@yu.edu> Cc: mailman-users@mailman3.org <mailman-users@mailman3.org> Subject: [EXT] - [MM3-users] moving lists to another server
**External Email**
bryan.kartzman@yu.edu writes:
I have both my mailman lists and django config in postgres databases.
These are Mailman 3 lists? What Django config? Do you mean Postorius and HyperKitty? Do you intend to use Postorius and HyperKitty on the new server?
If I want to move my lists to another server that is running mailman 3, aside from copying the databases to the other server,
Do you mean the other server is already actively distributing Mailman list traffic, managing users and archives? Or is it fresh instance?
what other things (perhaps all files under /var/lib/mailman3?, etc) would I need to bring over to the other server?
That's going to vary depending on the way your existing server is set up. A Mailman 3 instance is a complex system of cooperating applications. Assuming you have not patched or added any Python code, typical configuration and customizations include
- Your webserver (Apache, nginx are most common) needs configuration to handle WSGI applications and may need modules enabled or installed for the WSGI server. This is usually installed as an OS distro package, and the configurations will live in /etc/apache2 or similar for another webserver. Depending on other content served by the two hosts you may have to edit the configuration on the new host rather than copy it from the old one.
- The WSGI server is typically a separate application, although there is an Apache module for this. Gunicorn is a Python application usually installed via pip. I like to keep its Django configuration, which I call gunicorn.conf.py, in /etc/mailman3. (Mailman core also uses gunicorn, but that is transparent to you, it should not need custom configuration.) uwsgi is a C or C++ application normally supplied as a distro package. Don't know where it keeps its config files but since it's a distro package my first guess is /etc/uwsgi.
- Your MTA, usually Postfix or Exim4, maybe Sendmail or qmail needs to be configured to route mail to Mailman, and apply any milters, signatures, and perhaps anti-yahoo configurations such as ARC.
- Your database server, usually PostgreSQL or MySQL.
- Mailman core has a single configuration file mailman.cfg, which I keep in /etc/mailman3. It also has a crontab, which in Debian- derived systems typically lives in the file /etc/cron.d/mailman3. Note that this could be implmented as a systemd timer, but I don't know if any distros do that.
- Django, Postorius, and HyperKitty together constitute the mailman-web subsystem. I strongly recommend you use the mailmanweb Python package to manage Django and its applications. In any case I am not familiar with other organizations of those packages, so I'm going to assume it. If you are not using mailmanweb, you will need to be careful because most subpackages in the Django subsystem name their configuration files settings.py, urls.py, and views.py just like top-level, and distinguish them by using Python's attribute notation (eg, postorius.urls). a. Django application configuration for Postorius and HyperKitty goes in settings.py, which I put in /etc/mailman3. b. Django site configuration for Postorius and HyperKitty goes in urls.py (the routing configuration mapping URLs to views) and views.py (mapping views to Python code). I put these in /etc/mailmn3. c. Full-text indexing for HyperKitty. Default is Whoosh (a Python application) whose configuration goes in Django's settings.py. I prefer Xapian, which requires a separate application (Debian and Ubuntu have packages), but the needed configuration can be done in the top-level settings.py.
- Template customization. If you have any customized templates for Mailman's email or Django's views, you'll need to copy and move them too. I haven't customized email, but Django's page templates can be kept separately with a path variable set in settings.py. I put them in a hierarchy under /etc/mailman3/templates. Maybe Mailman core has a similar setting?
- Transient files. Aside from code and configuration, everything in Django is in the RDBMS, there are be no random files strewn around as far as I know. Mailman core is another matter. You can hope that all Mailman transient files are in its "var" hierarchy. $var_dir (the setting in mailman.cfg or inherited from schema.cfg) is the root of this hierarchy. I set var_dir=/opt/mailman/var, I think Mark may use var_dir=/opt/mailman/mm/var. I would guess that FHS-conformant distros would likely use var_dir=/var/lib/mailman3. a. The queue directories $var_dir/queue/* should be empty. Shut down the incoming mail and let mailman run until it's clear. (Normally takes < 5 seconds.) The exceptions are the shunt and bad queues, which are messages that Mailman doesn't know what to do with. With the venv active, run "mailman unshunt" to attempt to clear the shunt queue. Move anything that's left in shunt as well as any .bak or .tmp files into the bad queue for later treatment. These can be copied to the bad queue in the new instance. Use "mailman qfile <file>" to examine them to see if they can be deleted. b. I think the only stuff that ends up in the $var_dir/lists/* directories are digests being built for later distribution. Use the usual tools to send them off or copy them over to the new instance. c. There should be nothing in $var_dir/archives/hyperkitty/spool, which is temporary storage for messages to be archived. That is, it should get cleared up by the few seconds while incoming mail is stopped and outgoing messages are being processed. If there's anything still there, stash it away for later examination using "mailman qfile". (Don't put it in queue/bad or queue/archive though, this is a different process from the normal pipeline.)
I think that's about it.
Steve
Bryan Kartzman writes:
Wow, thanks for the very thorough response, Steve.
I should have started by saying that when we built the current production server, we cloned it off the test server and we were thinking now to move the data onto another clone of the old test server. So, yes we would have to apply all the changes we made since we cloned the current production server, but everything else as far as the OS is concerned should be comparable (webserver, MTA, Mailman core, Django, Hyperkitty, Postorius). That's why I was just asking about the "data" aspects of mailman/hyperkitty/postorius. What would I have to move to replicate all the lists on the current production server to a new clone of the original test server once we've applied any changes we made to production server since the clone.
If you're using the same instance of the database, and the transient queues are clean, then once you've applied those changes you just point Mailman core, HyperKitty, and Postorius at the database. If you need to copy the database into a different instance that's already running, *pro-ba-bly* you can just dump the 'mailman' database (and if you have a separate 'mailmanweb' database for Postorius and HyperKitty, that one too) as SQL code. Then load them into that database (make sure there's nothing in the SQL that messes with any existing databases in the RDBMS). If you have a fresh new instance of the database it's the same but no worries.
Steve
On Tue, Nov 7, 2023 at 2:07 PM Stephen J. Turnbull < turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:
bryan.kartzman@yu.edu writes:
I have both my mailman lists and django config in postgres databases.
These are Mailman 3 lists? What Django config? Do you mean Postorius and HyperKitty? Do you intend to use Postorius and HyperKitty on the new server?
If I want to move my lists to another server that is running mailman 3, aside from copying the databases to the other server,
Do you mean the other server is already actively distributing Mailman list traffic, managing users and archives? Or is it fresh instance?
Hello Stephen et al,
Since this topic is relevant to something I am faced with at the moment, I believe it's okay if I piggyback on the thread. I have a server that is already actively distributing Mailman list traffic, and managing users and archives. I have offered to take over another community ML (also using MM3) from another server and host the list on my server. I am scratching my head bald trying to figure out how to bring in this other MM3 site and list into my existing server, especially with regard to the DB backend.
The list - let's call it users@domainX.name was using MySQL backend and had a separate DB for Mailman core and Mailman web. I believe combining these into one isn't a problem. But getting to merge this DB into my current one is what I am not sure about.
Any pointers on how to bring in this new site into my current setup??
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 "Oh, the cruft.", egrep -v '^$|^.*#' ¯\_(ツ)_/¯ :-) [How to ask smart questions: http://www.catb.org/~esr/faqs/smart-questions.html]
Hey, I also would love to know how to export and import setting from one MM§-instance to another.
We have a running MM3-instance, that is already distributing mails for serveral domains, and want to import some more lists from another server without implementing all setting by hand for every list.
Kind regards, Jens.
Am 17.11.23 um 09:51 schrieb Odhiambo Washington:
On Tue, Nov 7, 2023 at 2:07 PM Stephen J. Turnbull < turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:
Hello Stephen et al,
Since this topic is relevant to something I am faced with at the moment, I believe it's okay if I piggyback on the thread. I have a server that is already actively distributing Mailman list traffic, and managing users and archives. I have offered to take over another community ML (also using MM3) from another server and host the list on my server. I am scratching my head bald trying to figure out how to bring in this other MM3 site and list into my existing server, especially with regard to the DB backend.
The list - let's call it users@domainX.name was using MySQL backend and had a separate DB for Mailman core and Mailman web. I believe combining these into one isn't a problem. But getting to merge this DB into my current one is what I am not sure about.
Any pointers on how to bring in this new site into my current setup??
On 11/17/23 00:51, Odhiambo Washington wrote:
I am scratching my head bald trying to figure out how to bring in this other MM3 site and list into my existing server, especially with regard to the DB backend.
The list - let's call it users@domainX.name was using MySQL backend and had a separate DB for Mailman core and Mailman web. I believe combining these into one isn't a problem. But getting to merge this DB into my current one is what I am not sure about.
This is https://gitlab.com/mailman/postorius/-/issues/13 which is a real need, but doesn't yet exist.
I think the easiest thing for the archives is to forget the Mailman web DB and just export the archive as a mbox from the other site (if it's large, exporting the entire archive will time out so you have to do it in pieces). Then you can use hyperkitty_import to import it.
For the list data, it's probably easiest to just create the list manually and go through the settings in Postorius and make them match, although you could dump the mailinglist table entry where list_id = 'users@domainx.name' and load that, but you'd need to edit the id field in the dump to match the id on your server. That leaves the members. Doing this with a DB dump/load would be very complex because while the member table has a list_id field which is the id of the list's entry in the mailinglist table and can be used to select the relevant members, these records also link to entries in the address, preferences and user tables and matching them would be an issue.
I think a custom script would be required. If I were doing this, I would make a separate database on my server with a copy of the data from the other server and then try to make a script that would extract data from the other database and add it to mine.
Alternatively you could just dump the members from the other servers
with the mailman members command perhaps separately for regular and
the different digest types and then add them with the mailman addmembers command. This wouldn't get all their attributes, but it
could work.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 11/17/23 14:42, Mark Sapiro wrote:
although you could dump the mailinglist table entry where list_id = 'users@domainx.name' and load that, but you'd need to edit the id field
That should be 'users.domainx.name'.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On Sat, Nov 18, 2023 at 1:42 AM Mark Sapiro <mark@msapiro.net> wrote:
On 11/17/23 00:51, Odhiambo Washington wrote:
I am scratching my head bald trying to figure out how to bring in this other MM3 site and list into my existing server, especially with regard to the DB backend.
The list - let's call it users@domainX.name was using MySQL backend and had a separate DB for Mailman core and Mailman web. I believe combining these into one isn't a problem. But getting to merge this DB into my current one is what I am not sure about.
This is https://gitlab.com/mailman/postorius/-/issues/13 which is a real need, but doesn't yet exist.
I think the easiest thing for the archives is to forget the Mailman web DB and just export the archive as a mbox from the other site (if it's large, exporting the entire archive will time out so you have to do it in pieces). Then you can use hyperkitty_import to import it.
What is the command to export the archive as a mbox from the DB? I can't figure out which option it is by looking at 'mailman-web -h'.
For the list data, it's probably easiest to just create the list manually and go through the settings in Postorius and make them match,
I want to go with the above.
although you could dump the mailinglist table entry where list_id =
'users@domainx.name' and load that, but you'd need to edit the id field in the dump to match the id on your server.
Not going with that.
That leaves the members.
Given that I have a dump of the two DBs (mailman and mailman-web) I just created one DB and imported them. I have been able to dump the members into a text file. Now importing is not a problem. BTW, how will ' mailman addmembers' treat a file containing:
Firstname Lastname <email@address>
Onlyname <email2@address>
?
Doing this with a DB dump/load would be very complex because while the member table has a list_id field which is the id of the list's entry in the mailinglist table and can be used to select the relevant members, these records also link to entries in the address, preferences and user tables and matching them would be an issue.
I am also not taking that path.
I think a custom script would be required. If I were doing this, I would
make a separate database on my server with a copy of the data from the other server and then try to make a script that would extract data from the other database and add it to mine.
Alternatively you could just dump the members from the other servers
with the
mailman memberscommand perhaps separately for regular and the different digest types and then add them with themailman addmemberscommand. This wouldn't get all their attributes, but it could work.
I have opted for the simple way to do this:
- Create the site in Django Admin and create/configure the list in Postorius
- Add the members from the file dump, with delivery set to regular.
- Send the members an email informing them that they need to set their delivery preferences.
I am stuck on the issue of exporting the archives to mbox file as already stated above.
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 "Oh, the cruft.", egrep -v '^$|^.*#' ¯\_(ツ)_/¯ :-) [How to ask smart questions: http://www.catb.org/~esr/faqs/smart-questions.html]
On 11/20/23 00:58, Odhiambo Washington wrote:
What is the command to export the archive as a mbox from the DB? I can't figure out which option it is by looking at 'mailman-web -h'.
Normally you do this on the old server via the HyperKitty web UI which has download buttons which do things like
https://example.com/archives/list/list@example.com/export/list@example.com.m...
or for a partial archive
However, in your case I don't think you have the old installation working any more so the above won't help.
BTW, how will ' mailman addmembers' treat a file containing:
Firstname Lastname <email@address> Onlyname <email2@address>
Correctly. It will import email@address with display name = 'Firstname Lastname' and email2@address with display name = 'Onlyname'.
I have opted for the simple way to do this:
- Create the site in Django Admin and create/configure the list in Postorius
- Add the members from the file dump, with delivery set to regular.
- Send the members an email informing them that they need to set their delivery preferences.
I am stuck on the issue of exporting the archives to mbox file as already stated above.
Perhaps you can export the mbox via the web UI on the old server or maybe temporarily create a second mailman-web on your server with the old mailman-web database and use that web UI. Otherwise, if you have a working mailman-web that can access this database, you could try something like
$ mailman-web shell
>>> from hyperkitty.lib.compat import get_list_by_name
>>> mlist = get_list_by_name('list_name_only', 'list_domain')
>>> with open('output.mbox', 'wb') as fp:
... for email in mlist.emails.order_by("archived_date").all():
... fp.write(email.as_bytes() + b'\n')
...
>>>
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On Mon, Nov 20, 2023 at 8:53 PM Mark Sapiro <mark@msapiro.net> wrote:
On 11/20/23 00:58, Odhiambo Washington wrote:
What is the command to export the archive as a mbox from the DB? I can't figure out which option it is by looking at 'mailman-web -h'.
Normally you do this on the old server via the HyperKitty web UI which has download buttons which do things like
https://example.com/archives/list/list@example.com/export/list@example.com.m...
or for a partial archive
However, in your case I don't think you have the old installation working any more so the above won't help.
I had a dump of the old installation actually - both databases. The above has actually helped and I obtained the archive.
BTW, how will ' mailman addmembers' treat a file containing:
Firstname Lastname <email@address> Onlyname <email2@address>Correctly. It will import email@address with display name = 'Firstname Lastname' and email2@address with display name = 'Onlyname'.
Super!
I have opted for the simple way to do this:
- Create the site in Django Admin and create/configure the list in Postorius
- Add the members from the file dump, with delivery set to regular.
- Send the members an email informing them that they need to set their delivery preferences.
I am stuck on the issue of exporting the archives to mbox file as already stated above.
Perhaps you can export the mbox via the web UI on the old server or maybe temporarily create a second mailman-web on your server with the old mailman-web database and use that web UI.
That's exactly how I did it and got the archives.
Otherwise, if you have a working mailman-web that can access this database, you could try something like
$ mailman-web shell >>> from hyperkitty.lib.compat import get_list_by_name >>> mlist = get_list_by_name('list_name_only', 'list_domain') >>> with open('output.mbox', 'wb') as fp: ... for email in mlist.emails.order_by("archived_date").all(): ... fp.write(email.as_bytes() + b'\n') ... >>>
I tested the above out of curiosity. It failed. There are 9 lists in the setup, but I am only interested in 1.
(venv) mailman@debian12:~$ mailman-web shell
Python 3.11.2 (main, Mar 13 2023, 12:18:29) [GCC 12.2.0]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.17.2 -- An enhanced Interactive Python. Type '?' for help.
In [1]: from hyperkitty.lib.compat import get_list_by_name
In [2]: mlist = get_list_by_name('list_name_only', 'list_domain')
---------------------------------------------------------------------------
Http404 Traceback (most recent call last)
Cell In[2], line 1
----> 1 mlist = get_list_by_name('list_name_only', 'list_domain')
File ~/venv/lib/python3.11/site-packages/hyperkitty/lib/compat.py:37, in
get_list_by_name(list_name, domain)
35 matching =
list(MailingList.objects.filter(name__startswith=list_name+"@"))
36 if len(matching) == 0: # no candidate found
---> 37 raise Http404("No archived mailinglist by that name")
38 if len(matching) == 1: # only one candidate
39 return matching[0]
Http404: No archived mailinglist by that name
Was I supposed to substitute something for the real listname of the list I wanted?
Another thing - totally unrelated - that surprised me: While using the commented-out block below, I could not click and get anything beyond http://http://list.name/mailman3 as anything to do with /postorius/xxxxx gave "Page not found" error. However, with the second config block that uses the UNIX socket, I am able to access everything
#<IfModule mod_proxy.c>
# ProxyPreserveHost On
# ProxyPass "/mailman3" "http://127.0.0.1:8010/mailman3"
# ProxyPass "/archives" "http://127.0.0.1:8010/archives"
# ProxyPass "/accounts" "http://127.0.0.1:8010/accounts"
# ProxyPass "/admin" "http://127.0.0.1:8010/admin"
# ProxyPass "/user-profile" "http://127.0.0.1:8010/user-profile"
# </IfModule>
ProxyPassMatch ^/static/ !
ProxyPassMatch ^/favicon.ico !
ProxyPass / unix:/opt/mailman/mm/var/uwsgi.sock|uwsgi://localhost/
ProxyPassReverse / unix:/opt/mailman/mm/var/uwsgi.sock|uwsgi://localhost/
Why is it that the UNIX socket option worked flawlessly while the other option failed???
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 "Oh, the cruft.", egrep -v '^$|^.*#' ¯\_(ツ)_/¯ :-) [How to ask smart questions: http://www.catb.org/~esr/faqs/smart-questions.html]
On 11/21/23 04:32, Odhiambo Washington wrote:
I tested the above out of curiosity. It failed. There are 9 lists in the setup, but I am only interested in 1.
(venv) mailman@debian12:~$ mailman-web shell Python 3.11.2 (main, Mar 13 2023, 12:18:29) [GCC 12.2.0] Type 'copyright', 'credits' or 'license' for more information IPython 8.17.2 -- An enhanced Interactive Python. Type '?' for help. In [1]: from hyperkitty.lib.compat import get_list_by_name In [2]: mlist = get_list_by_name('list_name_only', 'list_domain')
You can't use literally ('list_name_only', 'list_domain'). If the list of interest is for example alist@example.com, the above should be
mlist = get_list_by_name('alist', 'example.com')
...
Was I supposed to substitute something for the real listname of the list I wanted?
Yes.
Another thing - totally unrelated - that surprised me: While using the commented-out block below, I could not click and get anything beyond http://http://list.name/mailman3 as anything to do with /postorius/xxxxx gave "Page not found" error. However, with the second config block that uses the UNIX socket, I am able to access everything
#<IfModule mod_proxy.c> # ProxyPreserveHost On # ProxyPass "/mailman3" "http://127.0.0.1:8010/mailman3" # ProxyPass "/archives" "http://127.0.0.1:8010/archives" # ProxyPass "/accounts" "http://127.0.0.1:8010/accounts" # ProxyPass "/admin" "http://127.0.0.1:8010/admin" # ProxyPass "/user-profile" "http://127.0.0.1:8010/user-profile" # </IfModule> ProxyPassMatch ^/static/ ! ProxyPassMatch ^/favicon.ico ! ProxyPass / unix:/opt/mailman/mm/var/uwsgi.sock|uwsgi://localhost/ ProxyPassReverse / unix:/opt/mailman/mm/var/uwsgi.sock|uwsgi://localhost/Why is it that the UNIX socket option worked flawlessly while the other option failed???
The first option does not have ProxyPass for urls matching /postorius or /hyperkitty so such urls don't work. The second option passes everything (ProxyPass / ) to uwsgi via the socket so all urls that uwsgi can handle will work.
You could do something similar with the first option with simply
ProxyPass "/" "http://127.0.0.1:8010/"
assuming uwsgi is listening on port 8010.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Odhiambo Washington writes:
Hello Stephen et al,
Since this topic is relevant to something I am faced with at the moment, I believe it's okay if I piggyback on the thread.
I think continuing the thread is appropriate, but I personally would change the Subject to reflect the specifics.
I have a server that is already actively distributing Mailman list traffic, and managing users and archives. I have offered to take over another community ML (also using MM3) from another server and host the list on my server. I am scratching my head bald trying to figure out how to bring in this other MM3 site and list into my existing server, especially with regard to the DB backend.
The list - let's call it users@domainX.name was using MySQL backend and had a separate DB for Mailman core and Mailman web. I believe combining these into one isn't a problem. But getting to merge this DB into my current one is what I am not sure about.
Any pointers on how to bring in this new site into my current setup??
I don't have pointers, in the sense of concrete advice I would not be horrified if you followed it to the letter. Here's how I would attack the problem.
Decide the List-Id and List-Post questions. I disagree with Mark about List-Id (or maybe he mistyped List-Post?) -- the List-Id should NOT be changed. I don't know how many users are taking advantage of List-Id (my filters do!), but they will thank you for keeping it the same.
The List-Post question is contingent. Will you be serving their domain, or will you change the List-Post address? Note that with some additional configuration to Mailman and your MTA they could set up a forwarding configuration for their lists to your server, Or if they don't have other email needs, set up an MX. Or they can transfer their domain name's A and MX records to your host.
Alternatively you can change the List-Post address.to your domain. In that case you will have to edit the List-Post address in your database.
There are similar issues for mailman-web (especially host names). I have not thought about them at all, so I'll leave that for the interested reader. ;-)
(If the DB engines are different) Dump both databases in text format (ie, scripts of SQL commands). Check to see how they differ.
Adapt the foreign database to its new environment with search and replace. (Be verbose about the search terms, include lots of context to minimize the chance of inadvertantly replacing something in the wrong context that you missed in step 3.) This is where you would change the List-Post address, as well. You will also want to change the 'mailmanweb' database to just 'mailman'.
(Optional if your name is "Daredevil") Create a new Mailman suite install using your current configs, preferably in a new VM or set of containers. Configure the MTA to save all outgoing mail to a file, or maybe firewall off outgoing network traffic to port 25 and leave all the posts in the MTA queues. Create a new 'mailman-test' database. Copy all the dumps to the VM, and edit the names of the databases in the SQL to 'mailman-test'. Restore the dump of your system into the database, and test. Then do the same, restoring the mailman and mailmanweb databases from the foreign system into this new database.
Adjust to address any issues discovered in 5. If none, go to 7, Otherwise go back to 5.
Apply any changes made in 6 to your configs, and read the foreign dumps into your system, and you should be ready to go for the distribution part. Not sure about host and webserver configuration for mailman-web, though.
Steve
On Sat, Nov 18, 2023 at 11:48 AM Stephen J. Turnbull < turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:
Odhiambo Washington writes:
Hello Stephen et al,
Since this topic is relevant to something I am faced with at the moment, I believe it's okay if I piggyback on the thread.
I think continuing the thread is appropriate, but I personally would change the Subject to reflect the specifics.
I have a server that is already actively distributing Mailman list traffic, and managing users and archives. I have offered to take over another community ML (also using MM3) from another server and host the list on my server. I am scratching my head bald trying to figure out how to bring in this other MM3 site and list into my existing server, especially with regard to the DB backend.
The list - let's call it users@domainX.name was using MySQL backend and had a separate DB for Mailman core and Mailman web. I believe combining these into one isn't a problem. But getting to merge this DB into my current one is what I am not sure about.
Any pointers on how to bring in this new site into my current setup??
I don't have pointers, in the sense of concrete advice I would not be horrified if you followed it to the letter. Here's how I would attack the problem.
Decide the List-Id and List-Post questions. I disagree with Mark about List-Id (or maybe he mistyped List-Post?) -- the List-Id should NOT be changed. I don't know how many users are taking advantage of List-Id (my filters do!), but they will thank you for keeping it the same.
The List-Post question is contingent. Will you be serving their domain, or will you change the List-Post address?
The idea is to (if possible) not change anything at all. I will be serving the list using the original domain. I am hoping for a seamless migration where no one notices any change except for the IP address where the MM3 is running.
Note that with some additional configuration to Mailman and your MTA they could set up a forwarding configuration for their lists to your server, Or if they don't have other email needs, set up an MX. Or they can transfer their domain name's A and MX records to your host.
As it is, I have been given full control of the domain for this purpose so I can point A and MX records to my server. The ML itself is currently down. I have a dump of the /opt/mailman as well as the DBs used by Core and Web.
Alternatively you can change the List-Post address.to yourdomain. In that case you will have to edit the List-Post address in your database.
I am hoping that should not be necessary. I actually had thought it wouldn't be complex doing this until I sat down and said wait - there is a DB involved :)
- There are similar issues for mailman-web (especially host names). I have not thought about them at all, so I'll leave that for the interested reader. ;-)
Hehee.. Wrapping my head around this has not been easy, to say the least.
- (If the DB engines are different) Dump both databases in text
format (ie, scripts of SQL commands). Check to see how they differ.
The DB is MySQL. My running setup uses MariaDB. I am not getting what you mean here by how they differ.
- Adapt the foreign database to its new environment with search and
replace. (Be verbose about the search terms, include lots of context to minimize the chance of inadvertantly replacing something in the wrong context that you missed in step 3.) This is where you would change the List-Post address, as well. You will also want to change the 'mailmanweb' database to just 'mailman'.
If going with what I have mentioned above, is this step still necessary?
- (Optional if your name is "Daredevil") Create a new Mailman suite install using your current configs, preferably in a new VM or set of containers. Configure the MTA to save all outgoing mail to a file, or maybe firewall off outgoing network traffic to port 25 and leave all the posts in the MTA queues. Create a new 'mailman-test' database. Copy all the dumps to the VM, and edit the names of the databases in the SQL to 'mailman-test'. Restore the dump of your system into the database, and test. Then do the same, restoring the mailman and mailmanweb databases from the foreign system into this new database.
I was thinking about this as one way of testing the situation, much as it is quite involving.
- Adjust to address any issues discovered in 5. If none, go to 7,
Otherwise go back to 5.
- Apply any changes made in 6 to your configs, and read the foreign dumps into your system, and you should be ready to go for the distribution part. Not sure about host and webserver configuration for mailman-web, though.
Suppose I just do: that I am yet to understand here around SITE_ID when you have multiple
- Create the new list on my server using the inherited domain and configure it appropriately,
- Configure Django to handle the different domains (there is something
sites!) 3. Extract the users from the list and subscribe them afresh (I can shoot an email to let them know the ML had to be moved so they can set their preferences again) 4. Import the archives of the list I should more or less achieve the migration, no?
What about setting up on /opt/mailman2 a complete replica of the list? That'd be the real "Daredevil" :)
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 "Oh, the cruft.", egrep -v '^$|^.*#' ¯\_(ツ)_/¯ :-) [How to ask smart questions: http://www.catb.org/~esr/faqs/smart-questions.html]
Odhiambo Washington writes:
The idea is to (if possible) not change anything at all. I will be serving the list using the original domain.
Fine. I just didn't want to assume anything.
I am hoping that should not be necessary.
It's not. That's one of the major things that MM3 does that MM2 does not: handle multiple domains properly.
I actually had thought it wouldn't be complex doing this until I sat down and said wait - there is a DB involved :)
- (If the DB engines are different) Dump both databases in text
format (ie, scripts of SQL commands). Check to see how they differ.The DB is MySQL. My running setup uses MariaDB. I am not getting what you mean here by how they differ.
You didn't say that before. I tend to assume PostgreSQL because that's what I use.
I don't know if MySQL and MariaDB have different features that have different syntax, etc. That's what you're looking for. Any commands that are *not SQL* used to configure the database engine itself (for example the utf8mb4 parameter which MySQL has but PostgreSQL does not, does MariaDB need it?) Also any non-standard SQL stuff (mostly around varchar/text types I think).
- Adapt the foreign database to its new environment with search and
replace.If going with what I have mentioned above, is this step still necessary?
Isn't one installation using 'mailman' and 'mailmanweb' databases, while the other only 'mailman'? You'll need to deal with that at least I suppose.
- (Optional if your name is "Daredevil") Create a new Mailman suite install using your current configs, preferably in a new VM or set of containers.
I was thinking about this as one way of testing the situation, much as it is quite involving.
Fred Brooks famously put it this way: "Build one to throw away. You will anyway. The only question is whether you deliver the first version to the customer." ;-)
Suppose I just do: that I am yet to understand here around SITE_ID when you have multiple
- Create the new list on my server using the inherited domain and configure it appropriately,
- Configure Django to handle the different domains (there is something
sites!)
You just add them in the Django admin page. Problems come in when you delete the original example.com placeholder domain. I think you probably need to add SITE_ID=0 to settings.py so the Django figures out the site from the HTTP header rather than just assuming it's the same Postorius site for all lists.
- Extract the users from the list and subscribe them afresh (I can shoot an email to let them know the ML had to be moved so they can set their preferences again)
- Import the archives of the list I should more or less achieve the migration, no?
I think that should work.
What about setting up on /opt/mailman2 a complete replica of the list? That'd be the real "Daredevil" :)
This will also work, I'm sure. But we'd be embarrassed after going to the trouble of making Mailman 3 handle multiple domains. Also you'd need separate instances of Mailman running, with their own /etc/mailman3-1 and /etc/mailman3-2, and more important you'd double the RSS and VM requirements by running two instances.
On Mon, Nov 20, 2023 at 7:56 AM Stephen J. Turnbull < turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:
Odhiambo Washington writes:
The idea is to (if possible) not change anything at all. I will be serving the list using the original domain.
Fine. I just didn't want to assume anything.
I am hoping that should not be necessary.
It's not. That's one of the major things that MM3 does that MM2 does not: handle multiple domains properly.
I actually had thought it wouldn't be complex doing this until I sat down and said wait - there is a DB involved :)
- (If the DB engines are different) Dump both databases in text
format (ie, scripts of SQL commands). Check to see how they differ.The DB is MySQL. My running setup uses MariaDB. I am not getting what you mean here by how they differ.
You didn't say that before. I tend to assume PostgreSQL because that's what I use.
I don't know if MySQL and MariaDB have different features that have different syntax, etc. That's what you're looking for. Any commands that are *not SQL* used to configure the database engine itself (for example the utf8mb4 parameter which MySQL has but PostgreSQL does not, does MariaDB need it?) Also any non-standard SQL stuff (mostly around varchar/text types I think).
I use the utf8mb4 character set as default, unless something wants a different one. MySQL and MariaDB are almost similar. I have encountered very few significant differences since I switched to MariaDB.
- Adapt the foreign database to its new environment with search and
replace.If going with what I have mentioned above, is this step still necessary?
Isn't one installation using 'mailman' and 'mailmanweb' databases, while the other only 'mailman'? You'll need to deal with that at least I suppose.
This is true. However, I was hoping that I could merge the two foreign DBs into one. I am yet to test it though.
- (Optional if your name is "Daredevil") Create a new Mailman suite install using your current configs, preferably in a new VM or set of containers.
I was thinking about this as one way of testing the situation, much as it is quite involving.
Fred Brooks famously put it this way: "Build one to throw away. You will anyway. The only question is whether you deliver the first version to the customer." ;-)
I had to Google who he was :-)
Suppose I just do:
- Create the new list on my server using the inherited domain and configure it appropriately,
- Configure Django to handle the different domains (there is something that I am yet to understand here around SITE_ID when you have multiple sites!)
You just add them in the Django admin page. Problems come in when you delete the original example.com placeholder domain. I think you probably need to add SITE_ID=0 to settings.py so the Django figures out the site from the HTTP header rather than just assuming it's the same Postorius site for all lists.
In my case, I did not delete the original example.com. I finally get to use SITE_ID=0. I suppose that it should be accompanied by FILTER_VHOST=True
- Extract the users from the list and subscribe them afresh (I can shoot an email to let them know the ML had to be moved so they can set their preferences again)
- Import the archives of the list I should more or less achieve the migration, no?
I think that should work.
I am going to try this method. It looks safer for me.
What about setting up on /opt/mailman2 a complete replica of the list? That'd be the real "Daredevil" :)
This will also work, I'm sure. But we'd be embarrassed after going to the trouble of making Mailman 3 handle multiple domains. Also you'd need separate instances of Mailman running, with their own /etc/mailman3-1 and /etc/mailman3-2, and more important you'd double the RSS and VM requirements by running two instances.
I also will need to have extra macros, routers, and transports for Exim. This method is untenable as it complicates life.
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 "Oh, the cruft.", egrep -v '^$|^.*#' ¯\_(ツ)_/¯ :-) [How to ask smart questions: http://www.catb.org/~esr/faqs/smart-questions.html]
Odhiambo Washington writes:
I use the utf8mb4 character set as default, unless something wants a different one. MySQL and MariaDB are almost similar. I have encountered very few significant differences since I switched to MariaDB.
OK, then that would not be a problem.
On Mon, Nov 20, 2023 at 7:56?AM Stephen J. Turnbull < turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:
Isn't one installation using 'mailman' and 'mailmanweb' databases, while the other only 'mailman'? You'll need to deal with that at least I suppose.
This is true. However, I was hoping that I could merge the two foreign DBs into one. I am yet to test it though.
Yes. The point is that when you dump 'mailman' and 'mailmanweb' on the old host the name of the database will occur in at least the SQL CREATE DATABASE and CONNECT commands. You will need to edit that for mailmanweb.
I had to Google who he was :-)
One of my favorites!
In my case, I did not delete the original example.com. I finally get to use SITE_ID=0. I suppose that it should be accompanied by FILTER_VHOST=True
Yes, I think so.
I am going to try [the create and configure lists manually and add subscribers] method. It looks safer for me.
If there are only a few, it's little more time spent than writing scripts. And you can be sure you'll have basically working lists that you can fix any inadvertant configuration changes later. The only thing I would double and triple check is privacy settings (reply-to munging, archives, and subscriber rosters) and DMARC settings. In the MTA of course you want to be very careful about getting SPF and DKIM right for the new domain.
Steve
On Mon, Nov 20, 2023 at 12:14 PM Stephen J. Turnbull < turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:
Odhiambo Washington writes:
I use the utf8mb4 character set as default, unless something wants a different one. MySQL and MariaDB are almost similar. I have encountered very few significant differences since I switched to MariaDB.
OK, then that would not be a problem.
On Mon, Nov 20, 2023 at 7:56?AM Stephen J. Turnbull < turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:
Isn't one installation using 'mailman' and 'mailmanweb' databases, while the other only 'mailman'? You'll need to deal with that at least I suppose.
This is true. However, I was hoping that I could merge the two foreign DBs into one. I am yet to test it though.
Yes. The point is that when you dump 'mailman' and 'mailmanweb' on the old host the name of the database will occur in at least the SQL CREATE DATABASE and CONNECT commands. You will need to edit that for mailmanweb.
Actually, that was the easier part.
[snip]
If there are only a few, it's little more time spent than writing scripts. And you can be sure you'll have basically working lists that you can fix any inadvertant configuration changes later. The only thing I would double and triple check is privacy settings (reply-to munging, archives, and subscriber rosters) and DMARC settings. In the MTA of course you want to be very careful about getting SPF and DKIM right for the new domain.
This is another easy part.
How do I dump the archives into an mbox file?
Thanks for all the support.
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 "Oh, the cruft.", egrep -v '^$|^.*#' ¯\_(ツ)_/¯ :-) [How to ask smart questions: http://www.catb.org/~esr/faqs/smart-questions.html]
Stephen J. Turnbull writes:
- There are similar issues for mailman-web (especially host names). I have not thought about them at all, so I'll leave that for the interested reader. ;-)
Mark's right about this. I think Postorius and other configs will take care of themselves except for the users' tables which can be handled as Mark suggests, and the HyperKitty archives themselves are most easily handled by exporting to mbox and importing into the new site.
- Apply any changes made in 6 to your configs, and read the foreign dumps into your system, and you should be ready to go for the distribution part. Not sure about host and webserver configuration for mailman-web, though.
I just remembered one thing (among many, I'm sure): you probably need to create directories under $var_dir/lists for each foreign list. I'm pretty sure this is done by the "mailman create" function, not on the fly. The source for "mailman create" would be another good place to look for tasks you (or your script) need to perform and changes needed to database values in the SQL dump scripts.
Steve
On Sat, Nov 18, 2023 at 12:41 PM Stephen J. Turnbull < turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:
Stephen J. Turnbull writes:
- There are similar issues for mailman-web (especially host names). I have not thought about them at all, so I'll leave that for the interested reader. ;-)
Mark's right about this. I think Postorius and other configs will take care of themselves except for the users' tables which can be handled as Mark suggests, and the HyperKitty archives themselves are most easily handled by exporting to mbox and importing into the new site.
This one is noted.
- Apply any changes made in 6 to your configs, and read the foreign dumps into your system, and you should be ready to go for the distribution part. Not sure about host and webserver configuration for mailman-web, though.
I just remembered one thing (among many, I'm sure): you probably need to create directories under $var_dir/lists for each foreign list. I'm pretty sure this is done by the "mailman create" function, not on the fly. The source for "mailman create" would be another good place to look for tasks you (or your script) need to perform and changes needed to database values in the SQL dump scripts.
I am holding on for more input after my previous response. Then I can consider this if I must go this route.
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 "Oh, the cruft.", egrep -v '^$|^.*#' ¯\_(ツ)_/¯ :-) [How to ask smart questions: http://www.catb.org/~esr/faqs/smart-questions.html]
participants (6)
-
Bryan Kartzman -
bryan.kartzman＠yu.edu
-
Jens Günther -
Mark Sapiro -
Odhiambo Washington -
Stephen J. Turnbull