Hello Mailman users,
I know how to migrate mailman 2 lists to a new mailman 3 server. However I do not know how to migrate a mailman 3 list from one mailman 3 server to another one. Are there any instructions or protocol on how to accomplish this? The Mailman 3 server I want to migrate *from* was setup using Docker. The Mailman 3 server I want to migrate these lists *to* was setup using a python virtual environment via pip.
-- Please let me know if you need further assistance.
Thank you for your business. We appreciate our clients. Brian Carpenter EMWD.com
-- EMWD's Knowledgebase: https://clientarea.emwd.com/index.php/knowledgebase
EMWD's Community Forums http://discourse.emwd.com/
Hi Brian, It would be great if you share the information about "How to migrate mailman 2 lists to a new mailman 3 server". It is what I am currently having issues with, maybe I am missing some parts .
Thanks Mohsen
-----Original Message----- From: Brian Carpenter <brian_carpenter@emwd.com> Sent: Monday, April 27, 2020 11:52 AM To: Mailman users <mailman-users@mailman3.org> Subject: [MM3-users] Migrating Mailman 3 lists
[EXTERNAL EMAIL]
Hello Mailman users,
I know how to migrate mailman 2 lists to a new mailman 3 server. However I do not know how to migrate a mailman 3 list from one mailman 3 server to another one. Are there any instructions or protocol on how to accomplish this? The Mailman 3 server I want to migrate *from* was setup using Docker. The Mailman 3 server I want to migrate these lists *to* was setup using a python virtual environment via pip.
-- Please let me know if you need further assistance.
Thank you for your business. We appreciate our clients. Brian Carpenter EMWD.com
-- EMWD's Knowledgebase: https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fclientarea...
EMWD's Community Forums https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdiscourse.e...
Mailman-users mailing list -- mailman-users@mailman3.org To unsubscribe send an email to mailman-users-leave@mailman3.org https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.mail...
On 4/27/20 11:57 AM, Mohsen Masoudfar wrote:
Hi Brian, It would be great if you share the information about "How to migrate mailman 2 lists to a new mailman 3 server". It is what I am currently having issues with, maybe I am missing some parts .
Thanks Mohsen
My process may differ from yours depending on what server setup you have for Mailman 3. For instance the following tasks will need some additional steps if you are using a Docker installation of Mailman 3.
Steps to Import a Mailman 2.1 List into Mailman 3
Reference URL: https://docs.mailman3.org/en/latest/migration.html
SFTP from Mailman 3 server into Mailman 2 server to pick up config.pck and listname.mbox files. The rest of the steps will be performed on the Mailman 3 server.
Switch to user mailman and go into the /opt/mailman/mm/ directory.
Enter the virtual environment: source /opt/mailman/mm/venv/bin/activate
Make sure the config.pck and listname.mbox file is located in the /opt/mailman/mm/ directory before performing next step.
mailman import21 mailman3-list@example.com config.pck (This is to migrate the list members and settings)
/opt/mailman/mm/bin/django-admin hyperkitty_import -l mailman3-list@example.com listname.mbox (This is to migrate the list archives)
/opt/mailman/mm/bin/django-admin update_index_one_list mailman3-list@example.com
-- Please let me know if you need further assistance.
Thank you for your business. We appreciate our clients. Brian Carpenter EMWD.com
-- EMWD's Knowledgebase: https://clientarea.emwd.com/index.php/knowledgebase
EMWD's Community Forums http://discourse.emwd.com/
Il 27/04/20 18:32, Brian Carpenter ha scritto:
. . .
- mailman import21 mailman3-list@example.com config.pck (This is to migrate the list members and settings)
This was ok.
- /opt/mailman/mm/bin/django-admin hyperkitty_import -l mailman3-list@example.com listname.mbox (This is to migrate the list archives)
This doesn'7 work on a debian 10 system where mm3 was installed via apt-get
django admin says:
No Django settings specified. Unknown command: 'hyperkitty_import' Type 'django-admin help' for usage.
Where am I wrong? I'm not able to find a document for debian packages installation.
Thank You
-- Gelpi ing. Andrea
It took the computing power of three C-64s to fly to the Moon. It takes a 486 to run Windows 95. Something is wrong here.
On 2020-04-28 13:17, Gelpi Andrea wrote:
Il 27/04/20 18:32, Brian Carpenter ha scritto:
. . .
- mailman import21 mailman3-list@example.com config.pck (This is to migrate the list members and settings)
This was ok.
- /opt/mailman/mm/bin/django-admin hyperkitty_import -l mailman3-list@example.com listname.mbox (This is to migrate the list archives)
This doesn'7 work on a debian 10 system where mm3 was installed via apt-get
django admin says:
No Django settings specified. Unknown command: 'hyperkitty_import' Type 'django-admin help' for usage.
Where am I wrong? I'm not able to find a document for debian packages installation.
I did install MM3 months ago and did import MM2 maioingarchives.
mailman import21 abc /local/mailman/lists/abc/config.pck
did worked fairly well.
Also the impoort of mails: python3 manage.py hyperkitty_import -l abc /local/mailman/archives/private/abc.mbox/abc.mbox
did worked quite well. On a debian buster ssystem with installed mailman3 packages.
Thank You
MfG, Lars Schimmer
TU Graz, Institut für ComputerGraphik & WissensVisualisierung Tel: +43 316 873-5405 E-Mail: l.schimmer@cgv.tugraz.at Fax: +43 316 873-5402 PGP-Key-ID: 0x4A9B1723
Gelpi Andrea wrote:
[...]
django admin says:
No Django settings specified. Unknown command: 'hyperkitty_import' Type 'django-admin help' for usage.
When I was playing about with this package I had to run something similar to the following:
django-admin help --pythonpath /usr/share/mailman3-web --settings settings
I don't know whether there is a script installed anywhere now that makes this easier on the Mailman3 packages in Debian as I have rolled my own install, it may give you something to look at though.
Andrew.
Il 28/04/20 15:28, Andrew Hodgson ha scritto:
Gelpi Andrea wrote:
[...]
django admin says:
No Django settings specified. Unknown command: 'hyperkitty_import' Type 'django-admin help' for usage.
When I was playing about with this package I had to run something similar to the following:
django-admin help --pythonpath /usr/share/mailman3-web --settings settings
I don't know whether there is a script installed anywhere now that makes this easier on the Mailman3 packages in Debian as I have rolled my own install, it may give you something to look at though.
Andrew.
Thnak You so much, it works.
django-admin hyperkitty_import --pythonpath /usr/share/mailman3-web --settings settings -l list@example.com /mailman-dir/private/list.mbox/list.mbox
and
django-admin update_index_one_list --pythonpath /usr/share/mailman3-web --settings settings list@example.com
-- Gelpi ing. Andrea
It took the computing power of three C-64s to fly to the Moon. It takes a 486 to run Windows 95. Something is wrong here.
On Mon, Apr 27, 2020, at 8:52 AM, Brian Carpenter wrote:
Hello Mailman users,
I know how to migrate mailman 2 lists to a new mailman 3 server. However I do not know how to migrate a mailman 3 list from one mailman 3 server to another one. Are there any instructions or protocol on how to accomplish this? The Mailman 3 server I want to migrate *from* was setup using Docker. The Mailman 3 server I want to migrate these lists *to* was setup using a python virtual environment via pip.
If you can backup and migrate the database, that should do all of the Web stuff (except search index & logs) and most of the Core data. You'd need to additionally move the var/ directory if you want to digests/logs/pending emails.
If you don't need any of the digests/logs/pending emails in various queue, you should be able to migrate just by moving the database. There was an issue1 that was created recently where the digests would fail if the directory structure in the var/ directory wasn't re-created automatically, so you might have to preserve that data structure.
-- Please let me know if you need further assistance.
Thank you for your business. We appreciate our clients. Brian Carpenter EMWD.com
-- EMWD's Knowledgebase: https://clientarea.emwd.com/index.php/knowledgebase
EMWD's Community Forums http://discourse.emwd.com/
Mailman-users mailing list -- mailman-users@mailman3.org To unsubscribe send an email to mailman-users-leave@mailman3.org https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
-- thanks, Abhilash Raj (maxking)
On Apr 27, 2020, at 8:58 AM, Abhilash Raj <maxking@asynchronous.in> wrote:
I know how to migrate mailman 2 lists to a new mailman 3 server. However I do not know how to migrate a mailman 3 list from one mailman 3 server to another one. Are there any instructions or protocol on how to accomplish this? The Mailman 3 server I want to migrate *from* was setup using Docker. The Mailman 3 server I want to migrate these lists *to* was setup using a python virtual environment via pip.
If you can backup and migrate the database, that should do all of the Web stuff (except search index & logs) and most of the Core data. You'd need to additionally move the
var/directory if you want to digests/logs/pending emails.
At some point I would like to redo my MM3 instance to correct 2 bad decisions I made when I installed it: 1) using sqlite3 in production, and 2) using the Ubuntu packages, which are way behind current stable (~1 year).
I’ve seen mentioned that it’s not as simple as dumping the sqlite3 database and restoring it to, say, Postgres because some of the data types are different between them (which seems like an odd design choice).
Anyone sorted this kind of migration out yet?
- Mark
mark@pdc-racing.net | 408-348-2878
Mark Dadgar writes:
I’ve seen mentioned that it’s not as simple as dumping the sqlite3 database and restoring it to, say, Postgres because some of the data types are different between them (which seems like an odd design choice).
It's not really a design "choice". It's implied by using a object-relational manager (ORM, in the case of Mailman SQLAlchemy) for the database.
As for helping with the problem, I'm not a real SQL expert, but I think it's probably possible to come really close by dumping the database and restoring it to PostgreSQL.
Also, here's an interesting project:
https://pgloader.readthedocs.io/en/latest/
It's just the first open source one I looked at after searching for "migrate sqlite3 to postgres". I have no recommendation, but I'll note that this particular software does log errors of various kinds, so it's not just a quick hack -- they've thought about some of these issues. There may be better ones, or you may want to go with a commercial product.
Steve
On Apr 28, 2020, at 6:49 AM, Stephen J. Turnbull <turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:
Mark Dadgar writes:
I’ve seen mentioned that it’s not as simple as dumping the sqlite3 database and restoring it to, say, Postgres because some of the data types are different between them (which seems like an odd design choice).
It's not really a design "choice". It's implied by using a object-relational manager (ORM, in the case of Mailman SQLAlchemy) for the database.
As for helping with the problem, I'm not a real SQL expert, but I think it's probably possible to come really close by dumping the database and restoring it to PostgreSQL.
Also, here's an interesting project:
https://pgloader.readthedocs.io/en/latest/
It's just the first open source one I looked at after searching for "migrate sqlite3 to postgres". I have no recommendation, but I'll note that this particular software does log errors of various kinds, so it's not just a quick hack -- they've thought about some of these issues. There may be better ones, or you may want to go with a commercial product.
I finally found some time to try migrating my sqlite3 dbs to postgres using pgload (which is a pretty cool tool, frankly).
I get a bunch of these errors because identifiers in postgres are limited to 63 characters and mm3 creates some in sqlite that are longer than that:
2020-05-20T02:16:20.574000Z WARNING PostgreSQL warning: identifier "idx_44731_auth_user_user_permissions_user_id_permission_id_14a6b632_uniq" will be truncated to "idx_44731_auth_user_user_permissions_user_id_permission_id_14a6" 2020-05-20T02:16:20.583000Z WARNING PostgreSQL warning: identifier "idx_44719_auth_group_permissions_group_id_permission_id_0cd325b0_uniq" will be truncated to "idx_44719_auth_group_permissions_group_id_permission_id_0cd325b" 2020-05-20T02:16:20.644000Z WARNING PostgreSQL warning: identifier "idx_44821_socialaccount_socialtoken_app_id_account_id_fca4e0ac_uniq" will be truncated to "idx_44821_socialaccount_socialtoken_app_id_account_id_fca4e0ac_" 2020-05-20T02:16:20.662000Z WARNING PostgreSQL warning: identifier "idx_44839_socialaccount_socialaccount_provider_uid_fc810c6e_uniq" will be truncated to "idx_44839_socialaccount_socialaccount_provider_uid_fc810c6e_uni" 2020-05-20T02:16:20.711000Z WARNING PostgreSQL warning: identifier "idx_44764_auth_permission_content_type_id_codename_01ab375a_uniq" will be truncated to "idx_44764_auth_permission_content_type_id_codename_01ab375a_uni"
etc.
I killed the job before it finished and have not actually tried running against this database, but my question is this:
How does mailman3 handle this limitation with postgres if I set up an instance from scratch using postgres? Does it just truncate the identifiers identically, in which case migrating via pgloader should work?
For reference my pgloader script looks like:
load database from sqlite:///var/lib/dbconfig-common/sqlite3/mailman3-web/mailman3web.db into postgresql://mailman3:XXREDACTEDXX@localhost/mailman
with include drop, create tables, create indexes, reset sequences, no truncate
Thanks!
- Mark
mark@pdc-racing.net | 408-348-2878
On 5/19/20 7:25 PM, Mark Dadgar wrote:
I get a bunch of these errors because identifiers in postgres are limited to 63 characters and mm3 creates some in sqlite that are longer than that:
2020-05-20T02:16:20.574000Z WARNING PostgreSQL warning: identifier "idx_44731_auth_user_user_permissions_user_id_permission_id_14a6b632_uniq" will be truncated to "idx_44731_auth_user_user_permissions_user_id_permission_id_14a6"
...
etc.
I killed the job before it finished and have not actually tried running against this database, but my question is this:
How does mailman3 handle this limitation with postgres if I set up an instance from scratch using postgres? Does it just truncate the identifiers identically, in which case migrating via pgloader should work?
Mailman doesn't make those names. Those are names in table index entries. Mailman specifies the columns in a table which are indexed, but the DBM itself manages the index.
For reference my pgloader script looks like:
load database from sqlite:///var/lib/dbconfig-common/sqlite3/mailman3-web/mailman3web.db into postgresql://mailman3:XXREDACTEDXX@localhost/mailman
with include drop, create tables, create indexes, reset sequences, no truncate
I'm not sure why this doesn't work. A brief look at the pgloader docs seems to say it should.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mark Sapiro wrote:
On 5/19/20 7:25 PM, Mark Dadgar wrote:
I get a bunch of these errors because identifiers in postgres are limited to 63 characters and mm3 creates some in sqlite that are longer than that:
2020-05-20T02:16:20.574000Z WARNING PostgreSQL warning: identifier "idx_44731_auth_user_user_permissions_user_id_permission_id_14a6b632_uniq" will be truncated to "idx_44731_auth_user_user_permissions_user_id_permission_id_14a6"
[...]
For reference my pgloader script looks like:
load database from sqlite:///var/lib/dbconfig-common/sqlite3/mailman3-web/mailman3web.db into postgresql://mailman3:XXREDACTEDXX@localhost/mailman
with include drop, create tables, create indexes, reset sequences, no truncate
Have you tried this without indexes and then run the migrate script? It may re-create them as part of the schema upgrade check.
Andrew.
Andrew Hodgson writes:
Mark Sapiro wrote:
On 5/19/20 7:25 PM, Mark Dadgar wrote:
For reference my pgloader script looks like:
load database from sqlite:///var/lib/dbconfig-common/sqlite3/mailman3-web/mailman3web.db into postgresql://mailman3:XXREDACTEDXX@localhost/mailman
with include drop, create tables, create indexes, reset sequences, no truncate
Have you tried this without indexes and then run the migrate script? It may re-create them as part of the schema upgrade check.
That was my immediate guess. I'm not an SQL expert, but I believe that sequences and indexes are different entities, they're both optimizations that you don't need (ie, ignoring performance), and implementations differ on how they're implemented. Eg, if sqlite implements them as tables, while PostgreSQL has a special data structure, you could run into this kind of behavior. I'm pretty sure that these will be recreated appropriate to the new database when the schema validator is done.
But don't trust me, and *do* keep a backup. ;-)
Steve
On May 24, 2020, at 7:24 AM, Stephen J. Turnbull <turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:
Mark Sapiro wrote:
On 5/19/20 7:25 PM, Mark Dadgar wrote:
For reference my pgloader script looks like:
load database from sqlite:///var/lib/dbconfig-common/sqlite3/mailman3-web/mailman3web.db into postgresql://mailman3:XXREDACTEDXX@localhost/mailman
with include drop, create tables, create indexes, reset sequences, no truncate
Have you tried this without indexes and then run the migrate script? It may re-create them as part of the schema upgrade check.
That was my immediate guess. I'm not an SQL expert, but I believe that sequences and indexes are different entities, they're both optimizations that you don't need (ie, ignoring performance), and implementations differ on how they're implemented. Eg, if sqlite implements them as tables, while PostgreSQL has a special data structure, you could run into this kind of behavior. I'm pretty sure that these will be recreated appropriate to the new database when the schema validator is done.
But don't trust me, and *do* keep a backup. ;-)
😁
I did try it without indexes and it started showing the same warnings.
I will give this another try and let it run to completion. I was hesitant to do that because the core database is 7.5 GB and the web database is 35 GB so there are only so many copies I can keep hanging around on the cloud volume.
Since the data is mostly text, I am hoping that postgres will compress the bejeesus out of it.
- Mark
mark@pdc-racing.net | 408-348-2878
Brian Carpenter wrote:
I know how to migrate mailman 2 lists to a new mailman 3 server. However I do not know how to migrate a mailman 3 list from one mailman 3 server to another one. Are there any instructions or protocol on how to accomplish this? >The Mailman 3 server I want to migrate *from* was setup using Docker. The Mailman 3 server I want to migrate these lists *to* was setup using a python virtual environment via pip.
I did this the other week, the difference I think we've got here is that I migrated the whole instance (Mailman Core's var files plus complete Postgresql database) over to the new server rather than trying to add the lists to an existing setup, which I couldn't work out either which is what prompted me to complete the server migration before adding more lists.
The only issue I ran into was not copying the Mailman Core var files to the new server, this actually gave me a bit of trouble later on as firstly Mailman doesn't create the lists directory which is what I was checking with Exim for list existence, and also I had an issue where I was missing a moderated message from the messages directory which stopped the admin interface from displaying for that list.
However if anything would help on my experience would be happy to share more info.
Andrew.
On Mon, Apr 27, 2020, at 9:05 AM, Andrew Hodgson wrote:
Brian Carpenter wrote:
I know how to migrate mailman 2 lists to a new mailman 3 server. However I do not know how to migrate a mailman 3 list from one mailman 3 server to another one. Are there any instructions or protocol on how to accomplish this? >The Mailman 3 server I want to migrate *from* was setup using Docker. The Mailman 3 server I want to migrate these lists *to* was setup using a python virtual environment via pip.
I did this the other week, the difference I think we've got here is that I migrated the whole instance (Mailman Core's var files plus complete Postgresql database) over to the new server rather than trying to add the lists to an existing setup, which I couldn't work out either which is what prompted me to complete the server migration before adding more lists.
The only issue I ran into was not copying the Mailman Core var files to the new server, this actually gave me a bit of trouble later on as firstly Mailman doesn't create the lists directory which is what I was checking with Exim for list existence, and also I had an issue where I was missing a moderated message from the messages directory which stopped the admin interface from displaying for that list.
var/ directory does store some state for Mailman, specifically queued messages. Optionally, also templates. Directory structure is something that could be fixed with this issue1, but given that the messages are still stored on the filesystem instead of database, you'd need to move var/ directory too.
It might help to document the process somewhere. I'll create a bug report.
However if anything would help on my experience would be happy to share more info.
Andrew.
Mailman-users mailing list -- mailman-users@mailman3.org To unsubscribe send an email to mailman-users-leave@mailman3.org https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
-- thanks, Abhilash Raj (maxking)
Abhilash Raj wrote:
On Mon, Apr 27, 2020, at 9:05 AM, Andrew Hodgson wrote:
[Moving Mailman to new server]
I did this the other week, the difference I think we've got here is that I migrated the whole instance (Mailman Core's var files plus complete Postgresql database) over to the new server rather than trying to add the lists to an existing setup, which I couldn't work out either which is what prompted me to complete the server migration before adding more lists.
The only issue I ran into was not copying the Mailman Core var files to the new server, this actually gave me a bit of trouble later on as firstly Mailman doesn't create the lists directory which is what I was checking with Exim for list existence, and also I had an issue where I was missing a moderated message from the messages directory which stopped the admin interface from displaying for that list.
var/ directory does store some state for Mailman, specifically queued messages. Optionally, also templates. Directory structure is something that could be fixed with this issue[1], but given that the messages are still stored on the >filesystem instead of database, you'd need to move var/ directory too.
I would say that the var directory structure is essential to keep if possible between servers. I don't really want to fix the issue where I couldn't access a list because of old pending messages that were no longer available. I fixed it in the end by creating pickles with the same name, and then just deleting the messages from the web interface. The problem was happening because Postorius was trying to get the list of held messages to show on the admin interface, this caused Mailman REST interface to throw an exception.
I was going to post on here but figured it out in the end.
Another issue though stemming from this is that when I was moving the data across I had several messages in the var/messages directory structure and my incorrect assumption was that they were old held messages which weren't deleted. There were around 60 messages in that folder structure and I actually only needed to recover 2 of them to get the list on track, I forgot to check that the user had cleared their held messages. Even now when I deleted the fake messages I put there in the web interface, the files are still now on the server in that structure.
Andrew.
participants (9)
-
Abhilash Raj -
Andrew Hodgson -
Brian Carpenter -
Gelpi Andrea -
Lars Schimmer -
Mark Dadgar -
Mark Sapiro -
Mohsen Masoudfar -
Stephen J. Turnbull