On 8/7/24 10:52, IEM Network Operation Center (IOhannes m zmölnig) wrote:

...

With the dump and other utilities from your database manager. Almost everything is in the database. There are some things in Mailman's var/ directory, but those can simply be moved.

sorry for keeping you all busy, but: which "things in Mailman's var/" need to be copied over to the new host?

in my old instance (from Debian packages) i find in /var/lib/mailman3/:

• ./archives: dirtree without files
• ./cache: (afaict) currently active templates for mailinglist mails (hashed filenames) - i assume that these can safely be ignored and will be restored from the db?
• ./data: empty
• ./lists: empty directories named after each list (i think these are used by the MTA to determine whether it should hand over a mail to the mm3-ltmp)
• ./locks: plenty of lock-files, most of them rather stale (as in: 3 years old)
• ./messages: a couple of old messages (i think all of them have been held for approval)
• ./queue: dirtree without files
• ./templates: more templates for the existing mailinglists, though they don't seem to be used (are these the fallbacks when I remove a template via the webinterface?)
• ./web/static/*: seem to be copied over from various Debian packages (dependencies of mm3), with minor differences in icons and jquery (I assume these files get copied over when the mm3 instance is created, and do not get updated when the corresponding Debian packages get updated)
• ./web/static/CACHE - can these be ignored?
• ./web/fulltext_index: are these files regenerated when running "update_index"?

so i think the only stuff that needs to get recreated from mailman's var is the ./lists directory. is this correct?

mgfdasr IOhannes

Die Inhaltsfilterung von Mailman hat die folgenden MIME-Teile aus dieser Nachricht entfernt.

Content-Type: application/pgp-keys Name: OpenPGP_0xB65019C47F7A36F8.asc

On 8/7/24 12:55, Odhiambo Washington via Mailman-users wrote:

On Wed, Aug 7, 2024 at 1:05 PM IEM Network Operation Center (IOhannes m zmölnig) <noc@iem.at> wrote:

...

• ./data: empty

With a Postfic MTA situation, that would have had some files. So you don't use Postfix.

...

• ./lists: empty directories named after each list (i think these are used by the MTA to determine whether it should hand over a mail to the mm3-ltmp)

True, at least according to Exim MTA.

indeed, i'm using exim. i guess these directories are named like they are for backwards compatibility. if the sysadmin is supposed to ever do something with them, it would be nice if there was some (discoverable) documentation (like a README in Mailman's var; or the yet-to-be-written mm3-to-mm3 migration guide; even if those files/directories are mentioned in the Bible for installing Mailman, i don't think this can be found by a normal administrator (it's not reasonable to assume that one should look up each file/directory in Mailman's var to see if it is needed).

...

• ./locks: plenty of lock-files, most of them rather stale (as in: 3 years old)

So MM3 on that host hasn't been running or?

it *is* running. i do not know why it didn't cleanup.

• ./messages: a couple of old messages (i think all of them have been

...

held for approval)

• ./queue: dirtree without files
• ./templates: more templates for the existing mailinglists, though they don't seem to be used (are these the fallbacks when I remove a template via the webinterface?)

Yes.

so can i just ignore them?

So out here we focus so much on the following as the Bible for installing and running MM3: https://docs.mailman3.org/en/latest/install/virtualenv.html

probably. sorry for being obtuse.

afaict, the documentation is written for people setting things up from scratch, where the paths are mostly setup automatically, and you only need to configure them manually for interoperability with other software (e.g. when using postfix as the MTA, the docs tell the admin to ensure that both mm3 and postfix communicate via the same paths). i guess i'm confused because it is unclear to me, which of these paths hold persistent data, and which are just for ephemeral things (like sockets).

mgfasdr IOhannes

Die Inhaltsfilterung von Mailman hat die folgenden MIME-Teile aus dieser Nachricht entfernt.

Content-Type: application/pgp-keys Name: OpenPGP_0xB65019C47F7A36F8.asc

On Wed, Aug 7, 2024 at 11:52 AM IEM Network Operation Center (IOhannes m zmölnig) <noc@iem.at> wrote:

thanks both Mark and Odhiambo (who replied off-list) for your answers.

On 8/6/24 18:40, Mark Sapiro wrote:

...

On 8/6/24 01:43, noc@iem.at wrote:

...

but how can i export an entire MM3 list (including 20 years of archives) to some serialization format, and import it again in another instance?

With the dump and other utilities from your database manager. Almost everything is in the database. There are some things in Mailman's var/ directory, but those can simply be moved.

urgh. while this will (hopefully) work with a fresh installation, i'm somewhat afraid whether this will also work for:

• migrating a single list from one instance to another (at least, if you (like me) do not have have an intimate knowledge about the db layout)

Hmm.. I think I did something like this in the recent past. I need to remember how I went about it though, but it could be something like: (Assuming a virtualenv install) I believe you could: there are no conflicts with the sites

1. Make a full backup of the old MM2 install - I mean the DB and the MM3 files.
2. update the install to the latest version corresponding to the one on the new server.
3. Delete all lists with their archives EXCEPT that one list you want to migrate.
4. Export the DB(s)
5. !! Backup the DBs on the new server (just in case something bad happens!!
6. Import the DB dumps from the old server into the new one. Hopefully,

• importing lists into an instance that is already populated with other

lists (things like changed db schemas come to my mind)

Here again, you should have both sites running at the same software level.

right now my task is to move both some MM3 lists (from an old MM3

instance) and some MM2 lists onto a single server with a fresh installation, so i guess i will be fine.

Okay.

my worries are mostly about future migrations.

Sit down and write down your plans. Always come back here for 'sanity checks' on those plans.

is a feature-request warranted, or am i worrying too much?

A feature to automate your migrations? Have you consulted ChatGPT yet? :-)

-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 In an Internet failure case, the #1 suspect is a constant: DNS. "Oh, the cruft.", egrep -v '^$|^.*#' ¯\_(ツ)_/¯ :-) [How to ask smart questions: http://www.catb.org/~esr/faqs/smart-questions.html]

On Thu, Aug 8, 2024 at 11:19 AM IEM Network Operation Center (IOhannes m zmölnig) <noc@iem.at> wrote:

On 8/7/24 12:27, Odhiambo Washington via Mailman-users wrote:

...

On Wed, Aug 7, 2024 at 11:52 AM IEM Network Operation Center (IOhannes m zmölnig) <noc@iem.at> wrote:

...

is a feature-request warranted, or am i worrying too much?

A feature to automate your migrations? Have you consulted ChatGPT yet? :-)

should i ?-P

anyhow, i'm happy to say that my migrations have completed by now, and my new instance is now running with mailinglists previously hosted on either an MM2 instance or an older MM3 instance. so far i have not discovered any problems (well, apart from xapian eating all my (rather small, as this is only a VM) partition; but that is of course not related to anything we discussed here).

what i did was in a nutshell: thanks again, Odhiambo and Mark for your hints!

• create a dump of the database of the old mm3 instance (e.g. 'pg_dump -Fc mailman3web')
• drop any existing databases from the new mm3 instance
• import the dump into the new instance
• run db migrations
• import pck files from the old mm2 instance
• import archives from the old mm2 instance

however, i can report that my fears about migrating between mm3 instances by means of DB dumps have (at least partially) become true. 0. i'm using mailman3 as packaged in Debian, and PostgreSQL as the backend. before running the actual migrations, i've experimented a bit on the new instance (creating and deleting test lists), to see if the core configuration is working as expected. so i already had the databases up and running, filled with a bit of data, before attempting the migrations.

1. for whatever reasons, the db-names have changed between 3.2.1 ("mailman3", "mailman3web") and 3.3.8 ("mailman3.db", "mailman3web.db"). i strongly suspect that this is a problem of the Debian packaging of mailman3 (i'm pretty sure that i only ever accepted the defaults when installing the mailman3 Debian packages).

As you've rightly discovered, that issue can only be addressed by the Debian package maintainers.

2. since the db names changed, i tried to be clever (but i'm not a knowledgeable postgreSQL admin) and tried to only import the data-part of the dump ("pg_restore --data"), re-using the existing tables. this of course miserably failed.

I suspect failed because of the schema differences from 3.2.1 to 3.3.8. Well, I am also NOT experienced with PgSQL administration, but I think that for data to be imported from one table to another, the column definitions MUST match.

3. my next attempt to import the data into an existing DB, was to rename

the new db to the old name, import and rename back. this also failed miserably, most notably because the import insisted on assigning key IDs that were already in use. (that is: because i had already created and tested test-lists, the new instance already contained an email with hyperkitty_email.id=1; the dump from the old instance also contained an email with hyperkitty_email.id=1; so the import failed. this example is made up from memory, as i have not kept all the error logs). so this also failed.

That is expected. Maybe you could have vacuumed the tables completely.

4. i ended up, deleting the entire database, and import it from scratch.

after that, ran 'mailman-web migrate' to update the DB schema to the current one. this again failed with one hyperkitty-related migration that would create new tables. it turned out, that the new hosts uses postgreSQL-15.7.0 (the old one uses postgresql-11), and afaik since psql15 only a database *owner* can create new tables. after i made my mailman3-db-user the owner, all migrations succeeded.

That's a known issue here already. Something to do with access privileges to the SCHEMA public.

5. finally i imported my mm2 mailinglists and archives, which went rather smoothly.

Sure. The only issue people have had with that was to do with some small inconsistencies in the archives. There is a tool that can be used to check the archives and fix such.

some of my problems obviously come from me not being a DB-manager by

profession (e.g. new constraints in psql15,...) but others, like "3.", seem to hint at a core problem in the suggested migration strategy: if the to-be-imported db contains unique-keys that are already taken, the import simply fails (or at best, you get a partial import). i do not see how such a migration can ever work if you try to merge to mm3 instances where the dbs have a considerable amount of entries in the same tables (as you will always get constraint conflicts).

And that is the point that I did not reach/go beyond when I talked about having consistency in the DBs. But if the destination server had no lists, then it would probably not have been too difficult. Again, that's stuff for DB deep-dive.

to summarize, migrations must:

• *first* import old mm3 data into an *empty* database (so you cannot import multiple mm3 instances into a single new one)
• import mm2 data afterwards

I believe there is a way around that because I recently did import an ML from one site to a live site.

hence my idea for a special export/import format (and command), that

would only contain the actual structured data, but without all the extra db-specific things (like unique ids for referencing between tables).

Oh, yes! I am sure the Developers will be happy to get such a tool as a contribution from the community.

-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 In an Internet failure case, the #1 suspect is a constant: DNS. "Oh, the cruft.", egrep -v '^$|^.*#' ¯\_(ツ)_/¯ :-) [How to ask smart questions: http://www.catb.org/~esr/faqs/smart-questions.html]