config.pck from mailman2 has characters that won't import with import21
Hi. I've set up MM3 in a venv successfully and am importing my legacy lists.
I no longer have a running instance of mailman2, so don't know how I can change the offending list settings (i.e., in config.pck).
The problem seems to be a digest footer from the previous incarnation of the list (list name is edited below, to protect the innocent):
(venv) mailman@mail:~$ mailman import21 abc@lists.abcabcabc.org /var/lib/mailman/lists/abc/config.pck Unable to convert mailing list attribute: msg_footer with value "_______________________________________________ ${display_name} mailing list -- ${listname} To unsubscribe send an email to ${short_listname}-leave@${domain} %(web_page_url)slistinfo/%(_internal_name)s" Unable to convert mailing list attribute: digest_footer with value "_______________________________________________ ${display_name} mailing list -- ${listname} To unsubscribe send an email to ${short_listname}-leave@${domain} %(web_page_url)slistinfo/%(_internal_name)s" Importing members [------------------------------------] 0% Traceback (most recent call last): File "/opt/mailman/venv/bin/mailman", line 8, in <module> sys.exit(main()) ^^^^^^ File "/opt/mailman/venv/lib/python3.12/site-packages/click/core.py", line 1157, in __call__ return self.main(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib/python3.12/site-packages/click/core.py", line 1078, in main rv = self.invoke(ctx) ^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib/python3.12/site-packages/mailman/bin/mailman.py", line 69, in invoke return super().invoke(ctx) ^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib/python3.12/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib/python3.12/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, **ctx.params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib/python3.12/site-packages/click/core.py", line 783, in invoke return __callback(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib/python3.12/site-packages/click/decorators.py", line 33, in new_func return f(get_current_context(), *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib/python3.12/site-packages/mailman/commands/cli_import.py", line 89, in import21 import_config_pck(mlist, config_dict) File "/opt/mailman/venv/lib/python3.12/site-packages/mailman/utilities/importer.py", line 578, in import_config_pck import_roster(mlist, config_dict, members, MemberRole.member) File "/opt/mailman/venv/lib/python3.12/site-packages/mailman/utilities/importer.py", line 625, in import_roster _import_roster(mlist, config_dict, iterator, role, action=action) File "/opt/mailman/venv/lib/python3.12/site-packages/mailman/utilities/importer.py", line 645, in _import_roster if IBanManager(mlist).is_banned(email): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib/python3.12/site-packages/mailman/database/transaction.py", line 106, in wrapper return function(args[0], config.db.store, *args[1:], **kws) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib/python3.12/site-packages/mailman/model/bans.py", line 103, in is_banned re.match(ban.email, email, re.IGNORECASE) is not None): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/re/__init__.py", line 167, in match return _compile(pattern, flags).match(string) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/re/__init__.py", line 307, in _compile p = _compiler.compile(pattern, flags) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/re/_compiler.py", line 745, in compile p = _parser.parse(p, flags) ^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/re/_parser.py", line 979, in parse p = _parse_sub(source, state, flags & SRE_FLAG_VERBOSE, 0) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/re/_parser.py", line 460, in _parse_sub itemsappend(_parse(source, state, verbose, nested + 1, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/re/_parser.py", line 690, in _parse raise source.error("multiple repeat", re.error: multiple repeat at position 11
It seems that I might need to install mailman2 somewhere to try to change the config.pck. Are there any other options? I have dozens of lists with the same problem.
Thanks. Greg Newby
Hi Greg,
I offer the below, but the MM3 support people on this list are more knowledgeable than I, and may have a better idea about fixing corrupted pck files.
It doesn't need to be said, but for safety's sake, I would start by backing up the config.pck files to a safe place. I have un-pickled a few config.pck files for a project I'm working on to understand the structure and to have human-readable output for recovery purposes in the future. There are python one liners and short python scripts to unpickle files to ascii in the python docs and stack overflow, etc. Most mailman2 config.pck files appear to be plain ascii when un-pickled. If the corrupted config.pck files can be un-pickled to ascii format, I'd think you'd be able to open them with a text editor, "stare and compare," locate any corrupted parts and fix them. Line endings (LF, CRLF,) in the un-pickled files may be a factor when re-pickling. Then, re-pickle the edited ascii file and try import21 again. Anyway, it's a place to start.
On Sat, Jul 20, 2024 at 11:19 PM Greg Newby <gbnewby@petascale.org> wrote:
Hi. I've set up MM3 in a venv successfully and am importing my legacy lists.
I no longer have a running instance of mailman2, so don't know how I can change the offending list settings (i.e., in config.pck).
The problem seems to be a digest footer from the previous incarnation of the list (list name is edited below, to protect the innocent):
(venv) mailman@mail:~$ mailman import21 abc@lists.abcabcabc.org /var/lib/mailman/lists/abc/config.pck Unable to convert mailing list attribute: msg_footer with value "_______________________________________________ ${display_name} mailing list -- ${listname} To unsubscribe send an email to ${short_listname}-leave@${domain} %(web_page_url)slistinfo/%(_internal_name)s" Unable to convert mailing list attribute: digest_footer with value "_______________________________________________ ${display_name} mailing list -- ${listname} To unsubscribe send an email to ${short_listname}-leave@${domain} %(web_page_url)slistinfo/%(_internal_name)s" Importing members [------------------------------------] 0% Traceback (most recent call last): File "/opt/mailman/venv/bin/mailman", line 8, in <module> sys.exit(main()) ^^^^^^ File "/opt/mailman/venv/lib/python3.12/site-packages/click/core.py", line 1157, in __call__ return self.main(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib/python3.12/site-packages/click/core.py", line 1078, in main rv = self.invoke(ctx) ^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib/python3.12/site-packages/mailman/bin/mailman.py", line 69, in invoke return super().invoke(ctx) ^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib/python3.12/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib/python3.12/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, **ctx.params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib/python3.12/site-packages/click/core.py", line 783, in invoke return __callback(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib/python3.12/site-packages/click/decorators.py", line 33, in new_func return f(get_current_context(), *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib/python3.12/site-packages/mailman/commands/cli_import.py", line 89, in import21 import_config_pck(mlist, config_dict) File "/opt/mailman/venv/lib/python3.12/site-packages/mailman/utilities/importer.py", line 578, in import_config_pck import_roster(mlist, config_dict, members, MemberRole.member) File "/opt/mailman/venv/lib/python3.12/site-packages/mailman/utilities/importer.py", line 625, in import_roster _import_roster(mlist, config_dict, iterator, role, action=action) File "/opt/mailman/venv/lib/python3.12/site-packages/mailman/utilities/importer.py", line 645, in _import_roster if IBanManager(mlist).is_banned(email): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib/python3.12/site-packages/mailman/database/transaction.py", line 106, in wrapper return function(args[0], config.db.store, *args[1:], **kws) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib/python3.12/site-packages/mailman/model/bans.py", line 103, in is_banned re.match(ban.email, email, re.IGNORECASE) is not None): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/re/__init__.py", line 167, in match return _compile(pattern, flags).match(string) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/re/__init__.py", line 307, in _compile p = _compiler.compile(pattern, flags) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/re/_compiler.py", line 745, in compile p = _parser.parse(p, flags) ^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/re/_parser.py", line 979, in parse p = _parse_sub(source, state, flags & SRE_FLAG_VERBOSE, 0) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/re/_parser.py", line 460, in _parse_sub itemsappend(_parse(source, state, verbose, nested + 1, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/re/_parser.py", line 690, in _parse raise source.error("multiple repeat", re.error: multiple repeat at position 11
It seems that I might need to install mailman2 somewhere to try to change the config.pck. Are there any other options? I have dozens of lists with the same problem.
Thanks. Greg Newby
Mailman-users mailing list -- mailman-users@mailman3.org To unsubscribe send an email to mailman-users-leave@mailman3.org https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/ Archived at: https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/message/...
This message sent to keith1christian@gmail.com
On 7/20/24 10:19 PM, Greg Newby wrote:
Hi. I've set up MM3 in a venv successfully and am importing my legacy lists.
I no longer have a running instance of mailman2, so don't know how I can change the offending list settings (i.e., in config.pck).
The problem seems to be a digest footer from the previous incarnation of the list (list name is edited below, to protect the innocent):
That's not the issue.
(venv) mailman@mail:~$ mailman import21 abc@lists.abcabcabc.org /var/lib/mailman/lists/abc/config.pck Unable to convert mailing list attribute: msg_footer with value "_______________________________________________ ${display_name} mailing list -- ${listname} To unsubscribe send an email to ${short_listname}-leave@${domain} %(web_page_url)slistinfo/%(_internal_name)s" Unable to convert mailing list attribute: digest_footer with value "_______________________________________________ ${display_name} mailing list -- ${listname} To unsubscribe send an email to ${short_listname}-leave@${domain} %(web_page_url)slistinfo/%(_internal_name)s"
The above are non-fatal. The importer just won't make list specific templates for those. If you need the footers to be different from the defaults, you will need to create them manually.
Importing members [------------------------------------] 0% Traceback (most recent call last): File "/opt/mailman/venv/bin/mailman", line 8, in <module> sys.exit(main()) ^^^^^^ File "/opt/mailman/venv/lib/python3.12/site-packages/click/core.py", line 1157, in __call__ return self.main(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib/python3.12/site-packages/click/core.py", line 1078, in main rv = self.invoke(ctx) ^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib/python3.12/site-packages/mailman/bin/mailman.py", line 69, in invoke return super().invoke(ctx) ^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib/python3.12/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib/python3.12/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, **ctx.params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib/python3.12/site-packages/click/core.py", line 783, in invoke return __callback(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib/python3.12/site-packages/click/decorators.py", line 33, in new_func return f(get_current_context(), *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib/python3.12/site-packages/mailman/commands/cli_import.py", line 89, in import21 import_config_pck(mlist, config_dict) File "/opt/mailman/venv/lib/python3.12/site-packages/mailman/utilities/importer.py", line 578, in import_config_pck import_roster(mlist, config_dict, members, MemberRole.member) File "/opt/mailman/venv/lib/python3.12/site-packages/mailman/utilities/importer.py", line 625, in import_roster _import_roster(mlist, config_dict, iterator, role, action=action) File "/opt/mailman/venv/lib/python3.12/site-packages/mailman/utilities/importer.py", line 645, in _import_roster if IBanManager(mlist).is_banned(email): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib/python3.12/site-packages/mailman/database/transaction.py", line 106, in wrapper return function(args[0], config.db.store, *args[1:], **kws) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib/python3.12/site-packages/mailman/model/bans.py", line 103, in is_banned re.match(ban.email, email, re.IGNORECASE) is not None): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/re/__init__.py", line 167, in match return _compile(pattern, flags).match(string) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/re/__init__.py", line 307, in _compile p = _compiler.compile(pattern, flags) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/re/_compiler.py", line 745, in compile p = _parser.parse(p, flags) ^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/re/_parser.py", line 979, in parse p = _parse_sub(source, state, flags & SRE_FLAG_VERBOSE, 0) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/re/_parser.py", line 460, in _parse_sub itemsappend(_parse(source, state, verbose, nested + 1, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/re/_parser.py", line 690, in _parse raise source.error("multiple repeat", re.error: multiple repeat at position 11
The list's ban_list has an invalid regexp.
Do the following in Mailman 3's mailman shell
bin/mailman shell
Welcome to the GNU Mailman shell
Use commit() to commit changes.
Use abort() to discard changes since the last commit.
Exit with ctrl+D does an implicit commit() but exit() does not.
>>> import pickle
>>> from contextlib import ExitStack
>>> from mailman.commands.cli_import import _Mailman, _Bouncer
>>> from mailman.utilities.modules import hacked_sys_modules
>>> with ExitStack() as resources:
... resources.enter_context(hacked_sys_modules('Mailman', _Mailman))
... resources.enter_context(
... hacked_sys_modules('Mailman.Bouncer', _Bouncer))
... with open('/path/to/config.pck', 'rb+') as fp:
... data = pickle.load(fp)
... print(data['ban_list'])
... data['ban_list'] = []
... fp.seek(0)
... pickle.dump(data, fp)
...
>>>
This will print the ban_list so you can see it and then save the config.pck with an empty ban_list. That config.pck will no longer be usable in Mailman 2.1 because the pickle protocol is too high and the bounce_info items if any will be corrupted, but it should work with import21.
Note that all the ExitStack and hacked_sys_modules stuff is only needed if there is bounce_info in the config.pck, but it doesn't hurt in any case.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
The list's ban_list has an invalid regexp
Is this a bug? A brief websearch indicates mailman 2 allowed invalid regexps (but logged them)?
Would it be more appropriate to issue a warning and continue the import?
On 7/25/24 17:14, Karl Semich via Mailman-users wrote:
The list's ban_list has an invalid regexp
Is this a bug? A brief websearch indicates mailman 2 allowed invalid regexps (but logged them)?
Would it be more appropriate to issue a warning and continue the import?
I've created https://gitlab.com/mailman/mailman/-/issues/1166 and will fix it.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
participants (4)
-
Greg Newby -
Karl Semich -
Keith Christian -
Mark Sapiro