How to remove a message with an invalid Message-Id?
I have a fun problem: apparently a message sent to a mailing list has an invalid Message-Id (I think), which is causing the message parse to fail while attempting to archive the message. Unfortunately, the message seems to be trapped somewhere and resulting in error logging every few minutes. Is there any way to find and remove the message?
Literally, the only thing I've been able to find out about it is that the Message-Id contains the text "[83c48dddbea7492e873224a5ae1c04be-JFBVALKQOJXWILKNK4YVA7CBPJ2XEZKEMV3E64DTPRCW2YLJNR6EK6DPKNWXI4A=@microsoft.com]>".
The actual errors in the mailmanweb.log are:
ERROR 2023-06-14 21:49:40,692 3165793 django.request Internal Server Error: /archives/api/mailman/archive Traceback (most recent call last): File "/usr/local/lib/python3.8/email/_header_value_parser.py", line 2069, in get_msg_id token, value = get_dot_atom_text(value) File "/usr/local/lib/python3.8/email/_header_value_parser.py", line 1334, in get_dot_atom_text raise errors.HeaderParseError("expected atom at a start of " email.errors.HeaderParseError: expected atom at a start of dot-atom-text but found '[83c48dddbea7492e873224a5ae1c04be-JFBVALKQOJXWILKNK4YVA7CBPJ2XEZKEMV3E64DTPRCW2YLJNR6EK6DPKNWXI4A=@microsoft.com]>'
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/app/mailman/venv/lib/python3.8/site-packages/django/core/handlers/exception.py", line 34, in inner response = get_response(request) File "/app/mailman/venv/lib/python3.8/site-packages/django/core/handlers/base.py", line 115, in _get_response response = self.process_exception_by_middleware(e, request) File "/app/mailman/venv/lib/python3.8/site-packages/django/core/handlers/base.py", line 113, in _get_response response = wrapped_callback(request, *callback_args, **callback_kwargs) File "/app/mailman/venv/lib/python3.8/site-packages/django/views/decorators/http.py", line 40, in inner return func(request, *args, **kwargs) File "/app/mailman/venv/lib/python3.8/site-packages/hyperkitty/views/mailman.py", line 118, in _decorator return func(request, *args, **kwargs) File "/app/mailman/venv/lib/python3.8/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view return view_func(*args, **kwargs) File "/app/mailman/venv/lib/python3.8/site-packages/hyperkitty/views/mailman.py", line 164, in archive add_to_list(mlist_fqdn, msg) File "/app/mailman/venv/lib/python3.8/site-packages/hyperkitty/lib/incoming.py", line 65, in add_to_list msg_id = get_message_id(message) File "/app/mailman/venv/lib/python3.8/site-packages/hyperkitty/lib/utils.py", line 60, in get_message_id msg_id = email.utils.unquote(re.sub(r'\s', '', message['Message-Id'])) File "/usr/local/lib/python3.8/email/message.py", line 391, in __getitem__ return self.get(name) File "/usr/local/lib/python3.8/email/message.py", line 471, in get return self.policy.header_fetch_parse(k, v) File "/usr/local/lib/python3.8/email/policy.py", line 163, in header_fetch_parse return self.header_factory(name, value) File "/usr/local/lib/python3.8/email/headerregistry.py", line 607, in __call__ return self[name](name, value) File "/usr/local/lib/python3.8/email/headerregistry.py", line 202, in __new__ cls.parse(value, kwds) File "/usr/local/lib/python3.8/email/headerregistry.py", line 535, in parse kwds['parse_tree'] = parse_tree = cls.value_parser(value) File "/usr/local/lib/python3.8/email/_header_value_parser.py", line 2126, in parse_message_id token, value = get_msg_id(value) File "/usr/local/lib/python3.8/email/_header_value_parser.py", line 2073, in get_msg_id token, value = get_obs_local_part(value) File "/usr/local/lib/python3.8/email/_header_value_parser.py", line 1516, in get_obs_local_part if (obs_local_part[0].token_type == 'dot' or IndexError: list index out of range
The errors in the mailman.log are even less useful; they're basically copies of the HTML error page.
Jun 14 17:05:02 2023 (3165816) HyperKitty failure on https://lists.nasa.gov/archives/api/mailman/archive: ... <div class="alert alert-danger">An error occurred while processing your request.</div> ... Jun 14 17:05:02 2023 (3165816) Exception in the HyperKitty archiver: ... Jun 14 17:05:02 2023 (3165816) Traceback (most recent call last): File "/app/mailman/venv/lib/python3.8/site-packages/mailman_hyperkitty/__init__.py", line 151, in _archive_message url = self._send_message(mlist, msg) File "/app/mailman/venv/lib/python3.8/site-packages/mailman_hyperkitty/__init__.py", line 207, in _send_message raise ValueError(result.text) ValueError: ... <div class="alert alert-danger">An error occurred while processing your request.</div> ...
Any help would be appreciated!
Tommy M. McGuire Tommy.M.McGuire@nasa.gov
On 6/14/23 15:12, McGuire, Tommy M. (MSFC-IS61)[EAST2] via Mailman-users wrote:
I have a fun problem: apparently a message sent to a mailing list has an invalid Message-Id (I think), which is causing the message parse to fail while attempting to archive the message. Unfortunately, the message seems to be trapped somewhere and resulting in error logging every few minutes. Is there any way to find and remove the message?
Literally, the only thing I've been able to find out about it is that the Message-Id contains the text "[83c48dddbea7492e873224a5ae1c04be-JFBVALKQOJXWILKNK4YVA7CBPJ2XEZKEMV3E64DTPRCW2YLJNR6EK6DPKNWXI4A=@microsoft.com]>".
This is https://gitlab.com/mailman/mailman/-/issues/1065 which is fixed for the next release by https://gitlab.com/mailman/mailman/-/merge_requests/1099
You will find the message queued at Mailman's
var/archives/hyperkitty/spool/ directory. You can examine the pickled
message object with mailman qfile
and you can simply remove the file.
If you want to "fix" it and archive it, A mailman shell
interaction
like this will do it
>>> import pickle
>>> with open('/path/to/.pck/file', 'rb') as fp:
... msg = pickle.load(fp)
... msgdata = pickle.load(fp)
...
>>> msg.replace_header('Message-ID',
'<83c48dddbea7492e873224a5ae1c04be-JFBVALKQOJXWILKNK4YVA7CBPJ2XEZKEMV3E64DTPRCW2YLJNR6EK6DPKNWXI4A=@microsoft.com>')
>>> with open('/path/to/.pck/file', 'wb') as fp:
... pickle.dump(msg, fp)
... pickle.dump(msgdata, fp)
...
>>>
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
participants (2)
-
Mark Sapiro
-
McGuire, Tommy M. (MSFC-IS61)[EAST2]