Hi Mark,
thanks for correcting my assumption.
Based on your information I have been able to identify the suspect message file in /var/lib/mailman3/messages, replace the suspect file with another file, update the file permissions and successfully load the held messages page. This allowed me to discard the suspect message and other held messages are then available for review again.
In case the steps below help someone else who is having this issue here are the steps I followed with my initial search string redacted:
Searched all files in messages folder for matching string:
cd /var/lib/mailman3/messages grep -ir [REDACTED] *
Finds string in one file:
Binary file HE/D7/HED7SJLGJXTJ35AESY32AZXYPXDEVUVV matches
Check whether file can be parsed using mailman command:
mailman qfile ./HE/D7/HED7SJLGJXTJ35AESY32AZXYPXDEVUVV
Throws a matching error:
[----- start pickle -----] <----- start object 1 -----> Traceback (most recent call last): File "/usr/bin/mailman", line 11, in <module> load_entry_point('mailman==3.1.1', 'console_scripts', 'mailman')() File "/usr/lib/python3/dist-packages/mailman/bin/mailman.py", line 97, in main args.func(args) File "/usr/lib/python3/dist-packages/mailman/commands/cli_qfile.py", line 81, in process printer.pprint(obj) File "/usr/lib/python3.6/pprint.py", line 139, in pprint self._format(object, self._stream, 0, 0, {}, 0) File "/usr/lib/python3.6/pprint.py", line 161, in _format rep = self._repr(object, context, level) File "/usr/lib/python3.6/pprint.py", line 393, in _repr self._depth, level) File "/usr/lib/python3.6/pprint.py", line 405, in format return _safe_repr(object, context, maxlevels, level) File "/usr/lib/python3.6/pprint.py", line 555, in _safe_repr rep = repr(object) File "/usr/lib/python3/dist-packages/mailman/email/message.py", line 45, in __repr__ return self.__str__() File "/usr/lib/python3.6/email/message.py", line 135, in __str__ return self.as_string() File "/usr/lib/python3.6/email/message.py", line 158, in as_string g.flatten(self, unixfrom=unixfrom) File "/usr/lib/python3.6/email/generator.py", line 116, in flatten self._write(msg) File "/usr/lib/python3.6/email/generator.py", line 181, in _write self._dispatch(msg) File "/usr/lib/python3.6/email/generator.py", line 214, in _dispatch meth(msg) File "/usr/lib/python3.6/email/generator.py", line 272, in _handle_multipart g.flatten(part, unixfrom=False, linesep=self._NL) File "/usr/lib/python3.6/email/generator.py", line 116, in flatten self._write(msg) File "/usr/lib/python3.6/email/generator.py", line 181, in _write self._dispatch(msg) File "/usr/lib/python3.6/email/generator.py", line 214, in _dispatch meth(msg) File "/usr/lib/python3.6/email/generator.py", line 243, in _handle_text msg.set_payload(payload, charset) File "/usr/lib/python3.6/email/message.py", line 315, in set_payload payload = payload.encode(charset.output_charset) UnicodeEncodeError: 'ascii' codec can't encode characters in position 8-9: ordinal not in range(128)
Move suspicious file out of messages:
mv ./HE/D7/HED7SJLGJXTJ35AESY32AZXYPXDEVUVV /root/20200914/
held_messages page still throws 500 error.
Checking debug log, it is now because the database references a file that is no longer there.
Can I just copy a parseable file to the path of the bad file so I can delete it or do I need to remove the reference from the database?
Copy some other file:
cp ZZ/B2/ZZB24ADTZ2BOKBWTQWI7KG4XACGR6DTJ HE/D7/HED7SJLGJXTJ35AESY32AZXYPXDEVUVV
Still throws an error but this time it's a permissions error. mailman can't access the copied file:
ls -l HE/D7/
-rw-r----- 1 root root 78472 Sep 14 12:48 HED7SJLGJXTJ35AESY32AZXYPXDEVUVV
chown list:list HE/D7/HED7SJLGJXTJ35AESY32AZXYPXDEVUVV chmod g+w HE/D7/HED7SJLGJXTJ35AESY32AZXYPXDEVUVV ls -l HE/D7/
-rw-rw---- 1 list list 78472 Sep 14 12:48 HED7SJLGJXTJ35AESY32AZXYPXDEVUVV
held_messages now loads OK.
Cheers, Simon.
From: Mark Sapiro <mark@msapiro.net> Sent: Saturday, 12 September 2020 06:17 To: mailman-users@mailman3.org <mailman-users@mailman3.org> Subject: [MM3-users] Re: Mailman3 throws 500 error (unicode encode) when loading held messages in Postorius
On 9/6/20 3:39 PM, Simon Handfield via Mailman-users wrote:
Thanks for your advice, Stephen.
As I've removed the file from the shunt queue I don't believe that file is causing the 500 error any more.
I've confirmed the error is not cached in the browser.
If the issue does not reside in the shunt queue (no other files indicate parse errors when checking via command line) are there any other places I should look for data that is addressed by that page?
As Steve said, messages in the shunt queue have nothing to do with the held messages view in postorius. Held messages are in Mailman's var/messages/ directory and referenced by entries in the message table in the database which in turn are referenced by entries in the pendedkeyvalue table.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mailman-users mailing list -- mailman-users@mailman3.org To unsubscribe send an email to mailman-users-leave@mailman3.org https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.mail...
_
[State Library of NSW]<https://www.sl.nsw.gov.au/> Simon Handfield Desktop and Infrastructure Leader simon.handfield@sl.nsw.gov.au<mailto:simon.handfield@sl.nsw.gov.au> +61 2 9273 1535
Macquarie Street, Sydney NSW 2000, Australia www.sl.nsw.gov.au<https://www.sl.nsw.gov.au/>
This message is intended for the addressee named and may contain confidential information. If you are not the intended recipient, please delete it and notify the sender. Views expressed in this message are those of the individual sender, and are not necessarily the views of their organisation.