Thanks for the al the help - much appreciated! Mailman3 is a HUGE step up from 2.1.x.
Running the commands as the correct user solved my issues. Specially, I ran the import21 command as the “list” user (created by the debian/ubuntu package) and the hyperkitty import command as the “www-data” user. I had to add login shells to the passwd entries for those users to make it happen as they get installed with nologin.
I imported a 24G archive inbox and something went wrong along the way - the archives for many months consist of just a single identical thread. Re-running the import results in the following in just a few seconds:
www-data@mail:/tmp$ python /usr/share/mailman3-web/manage.py hyperkitty_import -l trackjunkies@pdc-racing.net /backups/trackjunkies.mbox Importing from mbox file /backups/trackjunkies.mbox to trackjunkies@pdc-racing.net Computing thread structure Synchronizing properties with Mailman Warming up cache The full-text search index will be updated every minute. Run the 'manage.py runjob update_index' command to update it now.
Any thoughts on how I can re-run the full import to make sure that all the archives are present?
Thanks!
- Mark
mark@pdc-racing.net | 408-348-2878
On Oct 3, 2019, at 9:25 PM, Mark Sapiro <mark@msapiro.net> wrote:
On 10/3/19 5:17 PM, Mark Dadgar wrote:
I brought up a fresh install of Mailman3 via the current Debian package alongside my current 2.1 setup (installed and updated from source since 2002) on Ubuntu 18.04 LTS.
Importing lists from 2.1 has been a real problem.
I am trying to import a list with no archives to keep it simple while I prove out the process. If I run the mailman import21 comment as my user, I get the following errors:
— mark@mail:/tmp$ mailman import21 announce@pdc-racing.net ./announce-config.pck Traceback (most recent call last): File "/usr/bin/mailman", line 11, in <module> load_entry_point('mailman==3.1.1', 'console_scripts', 'mailman')() File "/usr/lib/python3/dist-packages/mailman/bin/mailman.py", line 94, in main initialize(config_path) File "/usr/lib/python3/dist-packages/mailman/core/initialize.py", line 189, in initialize initialize_2(propagate_logs=propagate_logs) File "/usr/lib/python3/dist-packages/mailman/core/initialize.py", line 152, in initialize_2 mailman.core.logging.initialize(propagate_logs) File "/usr/lib/python3/dist-packages/mailman/core/logging.py", line 157, in initialize _init_logger(propagate, sub_name, log, logger_config) File "/usr/lib/python3/dist-packages/mailman/core/logging.py", line 110, in _init_logger handler = ReopenableFileHandler(sub_name, path_abs) File "/usr/lib/python3/dist-packages/mailman/core/logging.py", line 50, in __init__ self._stream = self._open() File "/usr/lib/python3/dist-packages/mailman/core/logging.py", line 53, in _open return codecs.open(self.filename, 'a', 'utf-8') File "/usr/lib/python3.6/codecs.py", line 897, in open file = builtins.open(filename, mode, buffering) PermissionError: [Errno 13] Permission denied: '/var/log/mailman3/mailman.log'
This is the issue. I don't know how the Debian packages do this, but I recommend having a 'mailman' user that owns everything and running commands as that user.
Error in atexit._run_exitfuncs: Traceback (most recent call last): File "/usr/lib/python3.6/logging/__init__.py", line 1945, in shutdown h.flush() File "/usr/lib/python3/dist-packages/mailman/core/logging.py", line 56, in flush if self._stream: AttributeError: 'ReopenableFileHandler' object has no attribute '_stream'
This is due to the prior permissions error.
If I run it as root, it runs for a long time but spews a HUGE stream of these errors to /var/log/mailman3/mailman.log:
If there are a lot of members to add, that can take a long time. The current version of import21 has a progress thermometer for this.
— Oct 03 17:08:48 2019 (19118) Skipping and preserving unparseable message: 1570147727.3029191+80f1cc4d18f54868ad6162e8e41e52567185e983 Oct 03 17:08:49 2019 (19118) Uncaught runner exception: [Errno 13] Permission denied: '/var/lib/mailman3/queue/virgin/1570147728.618866+174175d39c301feab4ebb2aac0a51226334fa54c.pck' Oct 03 17:08:49 2019 (19118) Traceback (most recent call last): File "/usr/lib/python3/dist-packages/mailman/core/runner.py", line 156, in _one_iteration msg, msgdata = self.switchboard.dequeue(filebase) File "/usr/lib/python3/dist-packages/mailman/core/switchboard.py", line 150, in dequeue with open(filename, 'rb') as fp: PermissionError: [Errno 13] Permission denied: '/var/lib/mailman3/queue/virgin/1570147728.618866+174175d39c301feab4ebb2aac0a51226334fa54c.pck'
First, I don't know what these messages are. Normally import21 doesn't send any mail. I'm guessing it's trying to send 'you are subscribed' notices to the users, but it shouldn't be doing this.
All these files should wind up in Mailman's var/queue/bad/ directory with a .psv extension. You should be able to examine them with the 'mailman qfile' command.
Whatever they are, the error comes from virgin runner so import21 should have actually succeeded. Did it import the list?
It also dumps some mail footer templates into the template directory that are owned by root, which seems … incorrect?
It makes a weak effort at converting the list's msg_footer and digest_footer into MM 3 templates in templates/list/<list-id>/<list-lang>/. These are owned by root in your case because you ran import21 as root.
Any thoughts on what’s going wrong here?
It looks like this may be an issue with the Debian package, at least the sending of messages part.
As for the rest, the Debian package should install stuff as a Mailman user, perhaps 'mailman' or 'list', and you should be running all commands as that user.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mailman-users mailing list -- mailman-users@mailman3.org To unsubscribe send an email to mailman-users-leave@mailman3.org https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/