Migration failure (2.1.29 to 3.3.8 on Debian testing)
Hello,
I have both the mailman and mailman3-full packages installed on my machine and I'm following the migration guide located here: https://docs.mailman3.org/en/latest/migration.html
Running the command { mailman import21 mylist@mysite.com /var/lib/mailman/lists/mylist/config.pck } completes once I create the list but states that it cannot convert the msg_footer and digest_footer mailing list attributes. (I might also point out that the { mailman create mylist@mysite.com } command isn't actually documented on that webpage.)
I can't seem to find the manage.py script in my $PATH and after searching, I find it as part of the mailman3-web package. When I run { sudo -H -u list python manage.py hyperkitty_import mylist@mysite.com /var/lib/mailman/archives/private/mylist.mbox/mylist.mbox }, it fails with "ImportError: No module named django.core.management".
If I run the script directly as { sudo -H -u list /usr/share/mailman3-web/manage.py ... }, I get "PermissionError: [Errno 13] Permission denied: '/usr/share/mailman3-web/settings_local.py'".
By default, settings_local.py's group is www-data, and not knowing where to go next, I decided to try changing that file's group to list. When I re-ran the command (as user list), I got a Python traceback that gives "PermissionError: [Errno 13] Permission denied: '/var/log/mailman3/web/mailman-web.log'".
Next, I changed the file's group back to list and decided to try running the manage.py script as the www-data user. This results in "/usr/lib/python3/dist-packages/django_q/conf.py:139: UserWarning: Retry and timeout are misconfigured." and "System check identified some issues:" followed by a list of "Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'" warnings for a number of Django tables. At the end of this list is "CommandError: The list address must be given on the command-line". But it *was* given.
My questions are:
- How do I undo any damage (if any) my efforts may have caused?
- What's the correct way to run manage.py?
On Wed, Jun 7, 2023 at 10:36 PM {16/7} <acc@hexadecagram.org> wrote:
Hello,
I have both the mailman and mailman3-full packages installed on my machine and I'm following the migration guide located here: https://docs.mailman3.org/en/latest/migration.html
Running the command { mailman import21 mylist@mysite.com /var/lib/mailman/lists/mylist/config.pck } completes once I create the list but states that it cannot convert the msg_footer and digest_footer mailing list attributes. (I might also point out that the { mailman create mylist@mysite.com } command isn't actually documented on that webpage.)
I can't seem to find the manage.py script in my $PATH and after searching, I find it as part of the mailman3-web package. When I run { sudo -H -u list python manage.py hyperkitty_import mylist@mysite.com /var/lib/mailman/archives/private/mylist.mbox/mylist.mbox }, it fails with "ImportError: No module named django.core.management".
If I run the script directly as { sudo -H -u list /usr/share/mailman3-web/manage.py ... }, I get "PermissionError: [Errno 13] Permission denied: '/usr/share/mailman3-web/settings_local.py'".
By default, settings_local.py's group is www-data, and not knowing where to go next, I decided to try changing that file's group to list. When I re-ran the command (as user list), I got a Python traceback that gives "PermissionError: [Errno 13] Permission denied: '/var/log/mailman3/web/mailman-web.log'".
Next, I changed the file's group back to list and decided to try running the manage.py script as the www-data user. This results in "/usr/lib/python3/dist-packages/django_q/conf.py:139: UserWarning: Retry and timeout are misconfigured." and "System check identified some issues:" followed by a list of "Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'" warnings for a number of Django tables. At the end of this list is "CommandError: The list address must be given on the command-line". But it *was* given.
Without using the recommended installation method, helping you is gonna be some effort! That recommended method is detailed at: https://docs.mailman3.org/en/latest/install/virtualenv.html#virtualenv-insta...
My questions are:
- How do I undo any damage (if any) my efforts may have caused?
You should not be worried so much because the Mailman3 confusion does not mess up your Mailman2.x install at all. If anything, I suppose just changing ownership of Mailman2.x files back to list:list should get Mailman2.x running.
- What's the correct way to run manage.py?
Use the recommended installation method: https://docs.mailman3.org/en/latest/install/virtualenv.html#virtualenv-insta... . Then from the virtualenv, you can migrate lists as follows:
- Create the list: mailman create listname@domain
- Import settings: mailman import21 listname@domains /path/to/lists/listname/config.pck
- Import archives: mailman-web hyperkitty_import -l listname@domain /path/to/archives/private/listname.mbox/listname.mbox # This can take long, if your archives are big
- Reindex the archives: mailman-web update_index_one_list listname@domain
This too can take long!
PS: You are right about migration. I have looked at https://docs.mailman3.org/en/latest/migration.html and it really does not to be updated to conform with the latest recommended documentation. Don't worry, we all went through that difficult process. You will make it.
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 "Oh, the cruft.", egrep -v '^$|^.*#' ¯\_(ツ)_/¯ :-) [How to ask smart questions: http://www.catb.org/~esr/faqs/smart-questions.html]
Hi Odhiambo, thank you for responding.
Let me first say that I am familiar with Python virtualenv, and while it may be the recommended **generic** installation procedure, there is no denying that Linux distros will offer software using their own packaging mechanisms. It would seem that moving this conversation over to the Debian community would be the best move to take then, if I continue to have issues and their packaging is to .
It turns out that I left the "-l" in the hyperkitty_import command. So I re-ran it with the "-l" included. I got the same warnings about primary keys but on first glance it appears to have successfully imported the entire set of archives. Something interesting to note: if I examine the mailman-web script, it attempts to run that same python script as the user www-data. This is just as I was doing (lucky guess!), but using su instead of sudo.
It seems like I've managed to complete the full process of migrating my list documented on that webpage (apart from rmlist'ing the old list).
FWIW, Here's the output:
$ sudo -H -u www-data /usr/share/mailman3-web/manage.py hyperkitty_import -l mylist@mysite.com /var/lib/mailman/archives/private/mylist.mbox/mylist.mbox ... warnings ... Importing from mbox file /var/lib/mailman/archives/private/mylist.mbox/mylist.mbox to mylist@mysite.com Computing thread structure Synchronizing properties with Mailman Warming up cache The full-text search index is not updated for this list. It will not be updated by the 'minutely' incremental update job. To update the index for this list, run the Django admin command with arguments 'update_index_one_list mylist@mysite.com'.
And here's the result of rebuilding the index.
$ sudo -H -u www-data /usr/share/mailman3-web/manage.py update_index_one_list mylist@mysite.com ... very similar warnings ... Indexing 1497 emails
On 6/7/23 11:58, {16/7} wrote:
I have both the mailman and mailman3-full packages installed on my machine and I'm following the migration guide located here: https://docs.mailman3.org/en/latest/migration.html
Which is the source of at least one of your issues. The Debian packages are behind and are missing multiple bug fixes and enhancements.
Running the command { mailman import21 mylist@mysite.com /var/lib/mailman/lists/mylist/config.pck } completes once I create the list but states that it cannot convert the msg_footer and digest_footer mailing list attributes. (I might also point out that the { mailman create mylist@mysite.com } command isn't actually documented on that webpage.)
import21 tries to make Mailman 3 compatible versions of your Mailman 2.1 msg_footer and digest_footer. It will put these in Mailman's var/templates/lists/<list-id>/<lc>/ directory as list:member:regular:footer.txt and list:member:digest:footer.txt respectively. You need to examine those files as they probably contain Mailman 2.1 style replacements that haven't been converted. See https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/rest/docs/t... for the Mailman 3 replacements which are encoded as dollar strings, e.g. $listname gets replaced with the list's posting address.
I can't seem to find the manage.py script in my $PATH and after searching, I find it as part of the mailman3-web package.
As you report in a followup, in the Debian package it's called mailman-web
Next, I changed the file's group back to list and decided to try running the manage.py script as the www-data user. This results in "/usr/lib/python3/dist-packages/django_q/conf.py:139: UserWarning: Retry and timeout are misconfigured." and "System check identified some issues:" followed by a list of "Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'" warnings for a number of Django tables.
Recent mailman installed following https://docs.mailman3.org/en/latest/install/virtualenv.html will include
DEFAULT_AUTO_FIELD = 'django.db.models.AutoField'
in Django's settings which will avoid that warning with Django>=3.2
My questions are:
- How do I undo any damage (if any) my efforts may have caused?
You probably didn't do any damage as the commands failed before doing anything.
- What's the correct way to run manage.py?
I see from your follow up that you figured this out.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Hey Mark, thanks for jumping in. I've noted everything you mentioned. Thank you for the pointers.
participants (3)
-
Mark Sapiro
-
Odhiambo Washington
-
{16/7}