moving list from mm3 -> mm3
Hey there, I'm still trying to figure out, how to move a bunch of lists from one mm3-instance to another. If somebody has a working solution, I would be really glad to hear from it.
I came across this thread, but I don't think it tackles my question: https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/thread/3... (or maybe I missed the right parts)
We have equal setups and versions:
- maxxking docker install 0.4.5
- Core 3.3.8
- API 3.1
- Postorius 1.3.8
Config
So I found the possibility to get the config of a list via REST-API:
curl -u restadmin:restpass localhost:8001/3.1/lists/$short.$domain/config | jq > short.domain
This is a nice collection of the configuration of a certain list. Does anybody know how I would get this data back to the newly created list in the second working mm3-instance? Shouldn't it be possible via POST command? I thought of something like:
curl -u restadmin:restpass -d @short.domain -H "Content-Type: application/json" localhost:8001/3.1/lists/&short.&domain/config
But it returns {"title": "405 Method Not Allowed"} and I don't know where to look in the docs, since it shows almost always a way through mailman shell.
Here I found something about POST: https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/rest/docs/h... -> but how would it work exactly?
## Members, moderators, owners
To export all members I normally run within mm-core-container as user mailman the following command:
mailman members &short.&domain > &short.txt
To fill them back in, it can be done in Postorius (or is there a way through API?).
What I would love to know is how to get out Owners of the corresponding lists and reimport them in the new one (which will be the same at the end :-D but different instance).
## registered users
This part is somehow tricky, since I have no clue how to do it. Some of the mailinglist-admins don't want to bother their list-owners again with registering newly on the other system. Is there a way to export all users (I think I found them in the DB in the table auth_users, somehow in combination with account_emailaddress) and get them over to the new instance (actually I just need part of them, but that's another question that I might solve lateron on my own).
Maybe the answer is already here: https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/rest/docs/u... -> but can somebody help to get this working (import/export)?
Thank you in advance, I hope it's not too much or too nooby :-) Jens.
This here would be best if I could list them by Domain!? But seems to me a little nonsense ... but maybe it's registered, through which VHOST the user registered.
Am 08.12.23 um 17:01 schrieb Jens Günther:
## registered users
This part is somehow tricky, since I have no clue how to do it. Some of the mailinglist-admins don't want to bother their list-owners again with registering newly on the other system. Is there a way to export all users (I think I found them in the DB in the table auth_users, somehow in combination with account_emailaddress) and get them over to the new instance (actually I just need part of them, but that's another question that I might solve lateron on my own).
Maybe the answer is already here: https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/rest/docs/u... -> but can somebody help to get this working (import/export)?
On 12/8/23 08:01, Jens Günther wrote:
This is a nice collection of the configuration of a certain list. Does anybody know how I would get this data back to the newly created list in the second working mm3-instance? Shouldn't it be possible via POST command? I thought of something like:
curl -u restadmin:restpass -d @short.domain -H "Content-Type: application/json" localhost:8001/3.1/lists/&short.&domain/config
But it returns {"title": "405 Method Not Allowed"} and I don't know where to look in the docs, since it shows almost always a way through mailman shell.
You need a PUT command, not POST. Try the above with -X PUT.
## Members, moderators, owners
To export all members I normally run within mm-core-container as user mailman the following command:
mailman members &short.&domain > &short.txt
To fill them back in, it can be done in Postorius (or is there a way through API?).
You can use Postorius -> Mass operations -> Mass subscribe to add the members, but you can't do that for owners and moderators. Also you lose information about the members such as Delivery status, Delivery mode and other user settings.
What I would love to know is how to get out Owners of the corresponding lists and reimport them in the new one (which will be the same at the end :-D but different instance).
You can use the --role option with mailman members to get owners and
moderators, but short of a custom script, you have to add them manually.
## registered users
This part is somehow tricky, since I have no clue how to do it. Some of the mailinglist-admins don't want to bother their list-owners again with registering newly on the other system. Is there a way to export all users (I think I found them in the DB in the table auth_users, somehow in combination with account_emailaddress) and get them over to the new instance (actually I just need part of them, but that's another question that I might solve lateron on my own).
Maybe the answer is already here: https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/rest/docs/u... -> but can somebody help to get this working (import/export)?
That is not the answer. Those are Mailman core users, not Django (web login) users. You need to get the relevant entries from the auth_users and account_emailaddress tables and add them to the new DB.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Thanks Mark for our answers, I already got a bit further.
Am 09.12.23 um 00:27 schrieb Mark Sapiro:
But it returns {"title": "405 Method Not Allowed"} and I don't know where to look in the docs, since it shows almost always a way through mailman shell. You need a PUT command, not POST. Try the above with
-X PUT.
With the -X PUT-command, there is the following error:
{"title": "400 Bad Request", "description": "Unexpected parameters:
bounces_address, created_at, digest_last_sent_at, fqdn_listname,
http_etag, join_address, last_post_at, leave_address, list_name,
mail_host, next_digest_number, no_reply_address, owner_address, post_id,
posting_address, request_address, usenet_watermark, volume"}
-> I don't know why, since I tried it with the exact same example-list from where I got all these parameters (ex- and then import) ... do I still miss something? Or does it only work on "fresh" lists?
You can use Postorius -> Mass operations -> Mass subscribe to add the members, but you can't do that for owners and moderators. Also you lose information about the members such as Delivery status, Delivery mode and other user settings.
Is there another way to preserve these information about the members? (in general I don't think a lot of people put in effort to customize there memeberships ... and furthermore they would need to register, right?)
## registered users
That is not the answer. Those are Mailman core users, not Django (web login) users. You need to get the relevant entries from the auth_users and account_emailaddress tables and add them to the new DB.
Ok, that would be fine.
Are there any more relations to other tables/columns that need to be set? (I just realised, that there are 2 different id in each table ... the one in auth_users is the user_id in account_emailaddress :-) scary).
Do you have any suggestions on how to add them correctly?
Thank you in advance for now and another big one in general to all of you on this list that try to help continouisly with all these problems. It's a great pleasure to be able to ask for help on a list like that. Thank you!
Have a good rest of the year. Regards, Jens.
Jens Günther writes:
Am 09.12.23 um 00:27 schrieb Mark Sapiro:
You need a PUT command, not POST. Try the above with
-X PUT.With the
-X PUT-command, there is the following error: {"title": "400 Bad Request", "description": "Unexpected parameters: bounces_address, created_at, digest_last_sent_at, fqdn_listname, http_etag, join_address, last_post_at, leave_address, list_name, mail_host, next_digest_number, no_reply_address, owner_address, post_id, posting_address, request_address, usenet_watermark, volume"}
You should post the exact command or interactive conversation you're using.
My guess is that the REST API is expecting a single JSON object, and you sent it multiple parameters.
Have you played with the REST API interactively? The documentation at https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/rest/docs/r... is really quite good as a tutorial. It's not a full API reference, and not organized that way, although the table of contents is very useful for what it does mention. But I think what you need (if you want to explore it that deeply) is a tutorial to get a "feel" for how the REST API "likes to talk".
Is there another way to preserve these information about the members?
You can query the REST API (you might need to fetch the subscriber roster, then iterate over that to get the details), or you can dump the database table (or the whole database) using SQL-based utilities like psql (Postgres, I forget the MySQL equivalent).
(in general I don't think a lot of people put in effort to customize there memeberships ... and furthermore they would need to register, right?)
Not sure what you mean by "register", but every address that writes to Mailman gets a temporary user until a "real" user claims it via the mailbox ownership dance. If they receive mail in a mailbox, they've done the ownership dance for that mailbox. If you don't preserve that record, they'll have to do it again.
Steve
Hey Stephen, thank you for your answer.
Am 31.12.23 um 08:45 schrieb Stephen J. Turnbull:
My guess is that the REST API is expecting a single JSON object, and you sent it multiple parameters.
Have you played with the REST API interactively? The documentation at
https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/rest/docs/r...
is really quite good as a tutorial. It's not a full API reference, and not organized that way, although the table of contents is very useful for what it does mention. But I think what you need (if you want to explore it that deeply) is a tutorial to get a "feel" for how the REST API "likes to talk".
Thanks for this advice. I did read the doc and figured out, that PUT and PATCH for my concerns should be the right way to do it: https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/rest/docs/l... (PUT for all writable attributes, with PATCH, one can change one attribute at a time)
When playing around, I figured out that there are also Read-only attributes, that can't be set (just got that when Playing with PATCH-command), which are exactly the ones in the PUT-error.
So my main question is now: does anybody know of a possibility with this command to extract only writable attributes!? The ones that are too much are: "Unexpected parameters: bounces_address, created_at, digest_last_sent_at, fqdn_listname, http_etag, join_address, last_post_at, leave_address, list_name, mail_host, next_digest_number, no_reply_address, owner_address, post_id, posting_address, request_address, usenet_watermark, volume"
You should post the exact command or interactive conversation you're using.
Getting the config: curl -u restadmin:restpass localhost:8001/3.1/lists/listname.domain/config | jq > listname.domain
curl -u restadmin:restpass -d @listname.domain -H "Content-Type: application/json" -X PATCH localhost:8001/3.1/lists/listname.domain/config
-> one has to be careful when deleting attributes, at the end of an array, there must not be a "," (!) :-)
Is there another way to preserve these information about the members?
You can query the REST API (you might need to fetch the subscriber roster, then iterate over that to get the details), or you can dump the database table (or the whole database) using SQL-based utilities like psql (Postgres, I forget the MySQL equivalent).
The first point sounds reasonable and I get now with the doc, how it could work ... but maybe a little too much for now, dommage.
Just tried getting all the members: curl -u restadmin:restpass localhost:8001/3.1/lists/listname.domain/roster/member | jq > listname.domain.members
But putting them back seems to be a different thing: curl -u restadmin:restpass -d @listname.domain.members -H "Content-Type: application/json" -X PUT localhost:8001/3.1/lists/listname.domain/roster/members
(in general I don't think a lot of people put in effort to customize there memeberships ... and furthermore they would need to register, right?)
Not sure what you mean by "register", but every address that writes to Mailman gets a temporary user until a "real" user claims it via the mailbox ownership dance. If they receive mail in a mailbox, they've done the ownership dance for that mailbox. If you don't preserve that record, they'll have to do it again.
Yes. I was talking about "real" users that claim ownership by registering via their mailaddress.
Thanks for your answers again, regards.
Jens Günther writes:
Thanks for this advice. I did read the doc and figured out, that PUT and PATCH for my concerns should be the right way to do it:
I'm not real sure about the semantics of those verbs, but I imagine Barry did it right.
So my main question is now: does anybody know of a possibility with this command to extract only writable attributes!?
If you mean doing this so you can PUT all of those at once, I'm pretty sure it doesn't work that way. PUT creates or replaces a whole object. If there's an existing value with read-only attributes, that's not going to work. It may be possible to specify multiple values to PATCH, but that's beyond my knowledge.
Getting the config: curl -u restadmin:restpass localhost:8001/3.1/lists/listname.domain/config | jq > listname.domain
Do you need to use curl (or anything other than Python 3) to do your work for some reason? Life is a lot easier if you use mailmanclient.
-> one has to be careful when deleting attributes, at the end of an array, there must not be a "," (!) :-)
Yeah, JSON objects are not Python dicts, no, nope, everybody's been burned before. ;-)
Just tried getting all the members: curl -u restadmin:restpass localhost:8001/3.1/lists/listname.domain/roster/member | jq > listname.domain.members
But putting them back seems to be a different thing: curl -u restadmin:restpass -d @listname.domain.members -H "Content-Type: application/json" -X PUT localhost:8001/3.1/lists/listname.domain/roster/members
Do you mean the difference between roster/member and roster/members? Not sure why the would be, would have to look at the source.
Happy new year!
Steve
Hey Stephen, thanks again for your reply! And also a late Happy New Year :-)
Am 04.01.24 um 15:20 schrieb Stephen J. Turnbull:
So my main question is now: does anybody know of a possibility with this command to extract only writable attributes!?
If you mean doing this so you can PUT all of those at once, I'm pretty sure it doesn't work that way. PUT creates or replaces a whole object. If there's an existing value with read-only attributes, that's not going to work. It may be possible to specify multiple values to PATCH, but that's beyond my knowledge.
Mh. I tried it this way:
- I curled the config
- tried to PUT it back into another list (didn't work b/c of read-only attributes)
- I deleted all the attributes from the export
- then PUT this file and it worked
-> it only seems crazy to delete all the attributes myself, I would want to curl and save only writable attributes :-)
Getting the config: curl -u restadmin:restpass localhost:8001/3.1/lists/listname.domain/config | jq > listname.domain
Do you need to use curl (or anything other than Python 3) to do your work for some reason? Life is a lot easier if you use mailmanclient.
I don't need to use curl, no. This was just the only way I found till right now to export and reimport the config of a list.
How would this work with mailmanclient? I'v never used it before and didn't find a hop-in for the right usage right away. Could you point me to the right direction or even write some sample-commands to get export and import of a lists config? That would be awesome!
Just tried getting all the members: curl -u restadmin:restpass localhost:8001/3.1/lists/listname.domain/roster/member | jq > listname.domain.members
But putting them back seems to be a different thing: curl -u restadmin:restpass -d @listname.domain.members -H "Content-Type: application/json" -X PUT localhost:8001/3.1/lists/listname.domain/roster/members
Do you mean the difference between roster/member and roster/members? Not sure why the would be, would have to look at the source.
This was abviously my bad. It wasn't about members with s or not -> the right one would always be without: localhost:8001/3.1/lists/listname.domain/roster/member
-> I just tried to export and import members of a list via the API just as I tried with the config. If you know of another way or how this could work, I also would be really glad. (it works with "mailman members $list
textfile" and reimport them via Postorius, I just tried it like that, since on commandline it would be easier for a whole set of lists)
Thanks again and kind regards, Jens.
Jens Günther writes:
- I deleted all the attributes from the export
- then PUT this file and it worked
Does the updated list still work? ;-)
-> it only seems crazy to delete all the attributes myself, I would want to curl and save only writable attributes :-)
In the design, that's a feature. The intention was backup, and restore of whole configs into a blank server.
How would this work with mailmanclient? I'v never used it before and didn't find a hop-in for the right usage right away. Could you point me to the right direction or even write some sample-commands to get export and import of a lists config? That would be awesome!
The documentation is here:
Am 18.01.24 um 12:34 schrieb Stephen J. Turnbull:
- I deleted all the attributes from the export
- then PUT this file and it worked
Does the updated list still work? ;-)
Yes, it perfectly does.
-> it only seems crazy to delete all the attributes myself, I would want to curl and save only writable attributes :-)
In the design, that's a feature. The intention was backup, and restore of whole configs into a blank server.
Ok, that makes some sense. But it still doesn't work, because also on a new server, the read-only-attributes can't be written on :-p
I will just find a way to batch manipulate the exported files in a way, that I can import them again. Hope this then works (at least I have a proof of concept, that the list tsill works :-)
How would this work with mailmanclient? I'v never used it before and didn't find a hop-in for the right usage right away. Could you point me to the right direction or even write some sample-commands to get export and import of a lists config? That would be awesome!
The documentation is here:
Thank you for the pointer. Examples are here: https://docs.mailman3.org/projects/mailmanclient/en/latest/src/mailmanclient... -> I just don't see, how this would be easier :-D Read through it, but still don't know how to batch export and reimport config and/or users. -> maybe it's again a dumb question, but how would I start it? I installed the python3-mailmanclient-debian-package, but can't start the
-prompt -> it it's easier, is there an easy way to export and import config? (or members?)
Thank you! Jens.
Jens Günther writes:
-> I just don't see, how this would be easier :-D Read through it, but still don't know how to batch export and reimport config and/or users.
If you were expecting a single command, that would be available via the command line interface "mailman". There is no such thing yet.
What I meant by "easier" is that you can experiment with it and maintain state in the Python interpreter rather than having to store intermediate results in files and figure out how to read them back in. And you have all Python facilities (classes, lists, dicts) to organize data rather than needing to mess with text editing of JSON or trying to script things in exactly the right order (eg, to deal with read-only attributes).
Once you have an object, you can use the pickle module to save it (that's how Mailman 2 kept its configs, after all).
-> maybe it's again a dumb question, but how would I start it?
The easy way to experiment is
mailman shell
This starts a Python 3 interpreter with many of the base mailman classes and functions already imported. It's actually an alias for "mailman withlist", but I usually use "mailman shell" with no arguments and "mailman withlist -l $LIST" when I know what list I want to work with.
Once you know what you want to do, for list-oriented tasks you can use "mailman withlist -l $LIST -r $YOUR_SCRIPT" to do it repeatedly.
You'll need to be more "creative" with users and other non-list objects to write usable scripts, importing mailmanclient and any relevant core modules. See mailman/src/mailman/commands for many examples (including the source code for mailman shell and mailman withlist in the file cli_withlist.py. The "mailman" command line utility itself is in mailman/src/mailman/bin/mailman.py, but you don't need to look at that, it doesn't do much interesting for your purpose. And you can probably ignore all the @click decorators, they are used to get rid of boilerplate for managing command line arguments.
mailman shell and mailman withlist keep a history, I think it's configured in mailman.cfg or schema.cfg.
-> it it's easier, is there an easy way to export and import config? (or members?)
I don't understand what you're trying to do, so I can't say.
The usual use case is backing up a whole site either as a disaster recovery tool, or in order to move the site to a new host. For that an SQL database dump, and an archive of /etc/mailman3 and any changes you've made to templates or source files, is the way to go.
If you just want lists of addresses and the like, "mailman withlist" will do that for you.
But you seem to want self-contained pieces of the whole object- oriented database, and that's going to be harder because you need to maintain structure. There's no way around that.
Steve
participants (3)
-
Jens Günther -
Mark Sapiro -
Stephen J. Turnbull