Thanks Stephen and Mark for your replies.

I have been able to identify a file in the shunt queue that threw a matching error when I performed mailman qfile on it.

I moved this file out of the queue in the hope that I would then be able to load the held_messages page successfully. Unfortunately I am still seeing a 500 error on loading the page.

I temporarily reenabled debug logging to confirm that the same error was being thrown.

I have run a loop over all remaining files in the shunt queue and all the remaining files can be accessed by mailman qfile without error.

Assuming that file was the cause are there other steps I need to perform in addition to removing the file from the shunt queue directory?

Thank you, Simon.

From: Stephen J. Turnbull turnbull.stephen.fw@u.tsukuba.ac.jp Sent: Sunday, 30 August 2020 12:52 To: Simon Handfield simon.handfield@sl.nsw.gov.au Cc: mailman-users@mailman3.org mailman-users@mailman3.org Subject: [MM3-users] Mailman3 throws 500 error (unicode encode) when loading held messages in Postorius

Simon Handfield via Mailman-users writes:

I have been working through other threads that indicate the same symptom (500 error when loading held_messages for a list).

You want threads that deal with Unicode:

Error reported in log with debug enabled:

Traceback (most recent call last):

And that last call is::

File "/usr/lib/python3.6/email/message.py", line 315, in set_payload payload = payload.encode(charset.output_charset) UnicodeEncodeError: 'ascii' codec can't encode characters in position 8-9: ordinal not in range(128)

It appears that the message is improperly constructed, and either a Content-Type field has a omitted or inappropriate charset parameter, or perhaps the Content-Type field is omitted entirely (ASCII is the default charset).

This is frequently a problem created by spam messages.

The message should be in the shunt queue. You can examine it with the command "mailman qfile /path/to/message_file.pck". Unfortunately it's not clear from the backtrace where in the message this is: it's at the beginning of a part ("position 8-9") or maybe line, but the part could be late in the message.

If it's spam, you can just delete the file (and do whatever else you do about spam, of course). If it's not spam, it would help if you can guess what language and charset are being used in the message, and we can go from there.

If this is unclear, feel free to ask more.

_

[State Library of NSW]https://www.sl.nsw.gov.au/ Simon Handfield Desktop and Infrastructure Leader simon.handfield@sl.nsw.gov.aumailto:simon.handfield@sl.nsw.gov.au +61 2 9273 1535

Macquarie Street, Sydney NSW 2000, Australia www.sl.nsw.gov.auhttps://www.sl.nsw.gov.au/

This message is intended for the addressee named and may contain confidential information. If you are not the intended recipient, please delete it and notify the sender. Views expressed in this message are those of the individual sender, and are not necessarily the views of their organisation.

Thanks for your advice, Stephen.

As I've removed the file from the shunt queue I don't believe that file is causing the 500 error any more.

I've confirmed the error is not cached in the browser.

If the issue does not reside in the shunt queue (no other files indicate parse errors when checking via command line) are there any other places I should look for data that is addressed by that page?

Cheers, Simon.

From: Stephen J. Turnbull turnbull.stephen.fw@u.tsukuba.ac.jp Sent: Monday, 31 August 2020 23:51 To: Simon Handfield simon.handfield@sl.nsw.gov.au Cc: mailman-users@mailman3.org mailman-users@mailman3.org Subject: [MM3-users] Re: Mailman3 throws 500 error (unicode encode) when loading held messages in Postorius

Simon Handfield via Mailman-users writes:

I have been able to identify a file in the shunt queue that threw a matching error when I performed mailman qfile on it.

I have run a loop over all remaining files in the shunt queue and all the remaining files can be accessed by mailman qfile without error.

Good. But there are still a bunch of problems with those files since they got shunted.

I moved this file out of the queue in the hope that I would then be able to load the held_messages page successfully. Unfortunately I am still seeing a 500 error on loading the page.

If this is a Mailman problem, I don't understand it. The shunt queue is not the moderation queue. The shunt queue is for messages that Mailman cannot handle without throwing an exception. Once a message is sent to the shunt queue, Mailman ignores it, so it shouldn't cause problems for the moderation page.

Have you tried restarting Mailman since moving the qfile?

If that doesn't work, and if it's the same error message and traceback, the easy suspect is browser caching (although we should be disabling that, I hope). Try a manual refresh (usually Ctrl-R or in the menu). If that doesn't work, try a private browsing window or another browser if you don't want to clear the cache or restart the browser.

I temporarily reenabled debug logging to confirm that the same error was being thrown.

Assuming that file was the cause are there other steps I need to perform in addition to removing the file from the shunt queue directory?

The only one I can think of is the check for browser-side caching.

Unfortunately, Mark's out of town (like, on top of a mountain with no bars on his phone), and I am not going to be able to do much with this until Thursday at the earliest. If you do get a new traceback, post it here (somebody else may have an idea).

If you like you may send me the qfile at the address in From. I'll see if I can reproduce the bug, and I'm pretty sure I can tell you what's in the file so at least you'll know whether it's important to somebody who isn't a scammer/spammer.

Steve

_

[State Library of NSW]https://www.sl.nsw.gov.au/ Simon Handfield Desktop and Infrastructure Leader simon.handfield@sl.nsw.gov.aumailto:simon.handfield@sl.nsw.gov.au +61 2 9273 1535

Macquarie Street, Sydney NSW 2000, Australia www.sl.nsw.gov.auhttps://www.sl.nsw.gov.au/

This message is intended for the addressee named and may contain confidential information. If you are not the intended recipient, please delete it and notify the sender. Views expressed in this message are those of the individual sender, and are not necessarily the views of their organisation.

Hi Mark,

thanks for correcting my assumption.

Based on your information I have been able to identify the suspect message file in /var/lib/mailman3/messages, replace the suspect file with another file, update the file permissions and successfully load the held messages page. This allowed me to discard the suspect message and other held messages are then available for review again.

In case the steps below help someone else who is having this issue here are the steps I followed with my initial search string redacted:

Searched all files in messages folder for matching string:

cd /var/lib/mailman3/messages grep -ir [REDACTED] *

Finds string in one file:

Binary file HE/D7/HED7SJLGJXTJ35AESY32AZXYPXDEVUVV matches

Check whether file can be parsed using mailman command:

mailman qfile ./HE/D7/HED7SJLGJXTJ35AESY32AZXYPXDEVUVV

Throws a matching error:

[----- start pickle -----] <----- start object 1 -----> Traceback (most recent call last): File "/usr/bin/mailman", line 11, in <module> load_entry_point('mailman==3.1.1', 'console_scripts', 'mailman')() File "/usr/lib/python3/dist-packages/mailman/bin/mailman.py", line 97, in main args.func(args) File "/usr/lib/python3/dist-packages/mailman/commands/cli_qfile.py", line 81, in process printer.pprint(obj) File "/usr/lib/python3.6/pprint.py", line 139, in pprint self._format(object, self._stream, 0, 0, {}, 0) File "/usr/lib/python3.6/pprint.py", line 161, in _format rep = self._repr(object, context, level) File "/usr/lib/python3.6/pprint.py", line 393, in _repr self._depth, level) File "/usr/lib/python3.6/pprint.py", line 405, in format return _safe_repr(object, context, maxlevels, level) File "/usr/lib/python3.6/pprint.py", line 555, in _safe_repr rep = repr(object) File "/usr/lib/python3/dist-packages/mailman/email/message.py", line 45, in __repr__ return self.__str__() File "/usr/lib/python3.6/email/message.py", line 135, in __str__ return self.as_string() File "/usr/lib/python3.6/email/message.py", line 158, in as_string g.flatten(self, unixfrom=unixfrom) File "/usr/lib/python3.6/email/generator.py", line 116, in flatten self._write(msg) File "/usr/lib/python3.6/email/generator.py", line 181, in _write self._dispatch(msg) File "/usr/lib/python3.6/email/generator.py", line 214, in _dispatch meth(msg) File "/usr/lib/python3.6/email/generator.py", line 272, in _handle_multipart g.flatten(part, unixfrom=False, linesep=self._NL) File "/usr/lib/python3.6/email/generator.py", line 116, in flatten self._write(msg) File "/usr/lib/python3.6/email/generator.py", line 181, in _write self._dispatch(msg) File "/usr/lib/python3.6/email/generator.py", line 214, in _dispatch meth(msg) File "/usr/lib/python3.6/email/generator.py", line 243, in _handle_text msg.set_payload(payload, charset) File "/usr/lib/python3.6/email/message.py", line 315, in set_payload payload = payload.encode(charset.output_charset) UnicodeEncodeError: 'ascii' codec can't encode characters in position 8-9: ordinal not in range(128)

Move suspicious file out of messages:

mv ./HE/D7/HED7SJLGJXTJ35AESY32AZXYPXDEVUVV /root/20200914/

held_messages page still throws 500 error.

Checking debug log, it is now because the database references a file that is no longer there.

Can I just copy a parseable file to the path of the bad file so I can delete it or do I need to remove the reference from the database?

Copy some other file:

cp ZZ/B2/ZZB24ADTZ2BOKBWTQWI7KG4XACGR6DTJ HE/D7/HED7SJLGJXTJ35AESY32AZXYPXDEVUVV

Still throws an error but this time it's a permissions error. mailman can't access the copied file:

ls -l HE/D7/

-rw-r----- 1 root root 78472 Sep 14 12:48 HED7SJLGJXTJ35AESY32AZXYPXDEVUVV

chown list:list HE/D7/HED7SJLGJXTJ35AESY32AZXYPXDEVUVV chmod g+w HE/D7/HED7SJLGJXTJ35AESY32AZXYPXDEVUVV ls -l HE/D7/

-rw-rw---- 1 list list 78472 Sep 14 12:48 HED7SJLGJXTJ35AESY32AZXYPXDEVUVV

held_messages now loads OK.

Cheers, Simon.

From: Mark Sapiro mark@msapiro.net Sent: Saturday, 12 September 2020 06:17 To: mailman-users@mailman3.org mailman-users@mailman3.org Subject: [MM3-users] Re: Mailman3 throws 500 error (unicode encode) when loading held messages in Postorius

On 9/6/20 3:39 PM, Simon Handfield via Mailman-users wrote:

Thanks for your advice, Stephen.

As I've removed the file from the shunt queue I don't believe that file is causing the 500 error any more.

I've confirmed the error is not cached in the browser.

If the issue does not reside in the shunt queue (no other files indicate parse errors when checking via command line) are there any other places I should look for data that is addressed by that page?

As Steve said, messages in the shunt queue have nothing to do with the held messages view in postorius. Held messages are in Mailman's var/messages/ directory and referenced by entries in the message table in the database which in turn are referenced by entries in the pendedkeyvalue table.

-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

Mailman-users mailing list -- mailman-users@mailman3.org To unsubscribe send an email to mailman-users-leave@mailman3.org https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.mail...

_

[State Library of NSW]https://www.sl.nsw.gov.au/ Simon Handfield Desktop and Infrastructure Leader simon.handfield@sl.nsw.gov.aumailto:simon.handfield@sl.nsw.gov.au +61 2 9273 1535

Macquarie Street, Sydney NSW 2000, Australia www.sl.nsw.gov.auhttps://www.sl.nsw.gov.au/

This message is intended for the addressee named and may contain confidential information. If you are not the intended recipient, please delete it and notify the sender. Views expressed in this message are those of the individual sender, and are not necessarily the views of their organisation.