On 4/26/21 2:00 AM, Andrew Hodgson wrote:
Yep there is a problem with an imported mbox which seems to be present somewhere in the archive on all the lists which I imported using that method. Here is a recent trace from the download option:
[2021-04-26 08:53:12 +0000] [28181] [ERROR] Error handling request Traceback (most recent call last): File "/opt/mailman/venv/lib/python3.7/site-packages/gunicorn/workers/sync.py", line 180, in handle_request for item in respiter: File "/opt/mailman/venv/lib/python3.7/site-packages/hyperkitty/views/mlist.py", line 335, in stream_mbox msg = email.as_message() File "/opt/mailman/venv/lib/python3.7/site-packages/hyperkitty/models/email.py", line 178, in as_message msg["Message-ID"] = "<%s>" % self.message_id File "/usr/lib/python3.7/email/message.py", line 409, in __setitem__ self._headers.append(self.policy.header_store_parse(name, val)) File "/usr/lib/python3.7/email/policy.py", line 145, in header_store_parse raise ValueError("Header values may not contain linefeed " ValueError: Header values may not contain linefeed or carriage return characters
Thanks for the traceback. The issue appears to be with folded headers that should either be unfolded before storing in HyperKitty's database or upon retrieval for this process. This is probably because of outlook.com folding Message-ID: headers. See https://gitlab.com/mailman/mailman/-/issues/844 for one manifestation of this.
I think if you replace the
msg["Message-ID"] = "<%s>" % self.message_id
line at 178 in /opt/mailman/venv/lib/python3.7/site-packages/hyperkitty/models/email.py with
msg["Message-ID"] = "<%s>" % self.message_id.strip(' <>\r\n')
it will avoid this issue.
Please try it and report.
I do still have the original mbox files for the imported lists which were on Mailman 2.1. The other thing is when downloading these mbox files I noticed they aren't the raw messages as we had in Mailman 2.1 but modified with headers stripped and email addresses modified. Is there any way to get the raw mbox back as in the option we had in Mailman 2.1?
Not from HyperKitty. It doesn't store the raw messages anywhere, but if you enable the prototype archiver for a list, the raw messages will be stored in maildir format in var/archives/prototype/list@example.com/ for each enabled list. Messages imported with hyperkitty_import won't be there, but all posts to the list, at least since enabling the prototype archiver, will be.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan