Held for notification - runner exception: 'ascii' codec can't encode character
Hello all,
Happy new year!
I have a case where a user (set to be moderated), posts a message. She got the appropriate email back saying she was being held for moderation. However, the moderator did not get the email. I found the following block in mailman.log:
Dec 28 19:37:44 2021 (33) Uncaught runner exception: 'ascii' codec can't encode character '\u2019' in position 5: ordinal not in range(128) Dec 28 19:37:44 2021 (33) Traceback (most recent call last): File "/usr/lib/python3.8/site-packages/mailman/core/runner.py", line 173, in _one_iteration self._process_one_file(msg, msgdata) File "/usr/lib/python3.8/site-packages/mailman/core/runner.py", line 266, in _process_one_file keepqueued = self._dispose(mlist, msg, msgdata) File "/usr/lib/python3.8/site-packages/mailman/runners/incoming.py", line 79, in _dispose process(mlist, msg, msgdata, start_chain) File "/usr/lib/python3.8/site-packages/mailman/core/chains.py", line 79, in process link.function(mlist, msg, msgdata) File "/usr/lib/python3.8/site-packages/mailman/chains/hold.py", line 234, in _process text = MIMEText(expand(template, mlist, substitutions), File "/usr/lib/python3.8/email/mime/text.py", line 42, in __init__ self.set_payload(_text, _charset) File "/usr/lib/python3.8/email/message.py", line 315, in set_payload payload = payload.encode(charset.output_charset) UnicodeEncodeError: 'ascii' codec can't encode character '\u2019' in position 5: ordinal not in range(128) Dec 28 19:37:44 2021 (33) SHUNTING: 1640720264.0541506+b139f885caa9444586b43093a09991b7c19581e6
There are two owners listed, one of them is also listed as the moderator. "mailman version" says: GNU Mailman 3.3.4 (Tom Sawyer) We are running from Docker. I can send you the pck file offline if that would help.
Thank you, Seth
On 1/2/22 11:00 AM, Seth Seeger wrote:
I have a case where a user (set to be moderated), posts a message. She got the appropriate email back saying she was being held for moderation. However, the moderator did not get the email. I found the following block in mailman.log:
Dec 28 19:37:44 2021 (33) Uncaught runner exception: 'ascii' codec can't encode character '\u2019' in position 5: ordinal not in range(128) Dec 28 19:37:44 2021 (33) Traceback (most recent call last): File "/usr/lib/python3.8/site-packages/mailman/core/runner.py", line 173, in _one_iteration self._process_one_file(msg, msgdata) File "/usr/lib/python3.8/site-packages/mailman/core/runner.py", line 266, in _process_one_file keepqueued = self._dispose(mlist, msg, msgdata) File "/usr/lib/python3.8/site-packages/mailman/runners/incoming.py", line 79, in _dispose process(mlist, msg, msgdata, start_chain) File "/usr/lib/python3.8/site-packages/mailman/core/chains.py", line 79, in process link.function(mlist, msg, msgdata) File "/usr/lib/python3.8/site-packages/mailman/chains/hold.py", line 234, in _process text = MIMEText(expand(template, mlist, substitutions), File "/usr/lib/python3.8/email/mime/text.py", line 42, in __init__ self.set_payload(_text, _charset) File "/usr/lib/python3.8/email/message.py", line 315, in set_payload payload = payload.encode(charset.output_charset) UnicodeEncodeError: 'ascii' codec can't encode character '\u2019' in position 5: ordinal not in range(128) Dec 28 19:37:44 2021 (33) SHUNTING: 1640720264.0541506+b139f885caa9444586b43093a09991b7c19581e6
There are two owners listed, one of them is also listed as the moderator.
This is redundant. Having the same user as both owner and moderator only results in possible duplicate notifications. There is no reason to have an owner also be a moderator.
"mailman version" says: GNU Mailman 3.3.4 (Tom Sawyer) We are running from Docker. I can send you the pck file offline if that would help.
Please do send me the shunted .pck. what is going on is this is an
English language list which has character set ascii and the notice for
some reason has non-ascii, possbbly the Subject: or the user's display
name has non-ascii. But possibly the message itself is defective.
In any case, I'd like to get the shunted message to see if there's something we should fix.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 1/2/22 11:18 AM, Mark Sapiro wrote:
Please do send me the shunted .pck. what is going on is this is an English language list which has character set
asciiand the notice for some reason has non-ascii, possbbly the Subject: or the user's display name has non-ascii. But possibly the message itself is defective.In any case, I'd like to get the shunted message to see if there's something we should fix.
Thanks for sending me the shunted .pck. It turns out I didn't need it. Had I looked more closely at the Traceback, I would have realized that the issue is that the list's list:admin:action:post template contains non-ascii. In particular, the character in position 5 is ’ which is a unicode right_single_quote, not an ascii apostrophe.
There may be other unicode fancy quotes ad well.
You can fix the template to contain only ascii or you can set the character set for English to utf-8 by adding
[language.en]
charset: utf-8
to mailman.cfg. The downside of the latter is that outgoing message bodies will always be base64 encoded which generally precludes grepping mailboxes to find things.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On Jan 2, 2022, at 3:47 PM, Mark Sapiro <mark@msapiro.net> wrote:
Thanks for sending me the shunted .pck. It turns out I didn't need it. Had I looked more closely at the Traceback, I would have realized that the issue is that the list's list:admin:action:post template contains non-ascii. In particular, the character in position 5 is ’ which is a unicode right_single_quote, not an ascii apostrophe.
There may be other unicode fancy quotes ad well.
You can fix the template to contain only ascii or you can set the character set for English to utf-8 by adding
[language.en] charset: utf-8to mailman.cfg. The downside of the latter is that outgoing message bodies will always be base64 encoded which generally precludes grepping mailboxes to find things.
Ah hah! Right you are. I removed the offending fancy apostrophe. What about that traceback pointed to the template? (Is it something I could have figured out?)
Is it possible to filter out UTF-8 chars from the templates when they are saved in Postorius?
Thanks again, Seth
On 1/2/22 4:24 PM, Seth Seeger wrote:
Ah hah! Right you are. I removed the offending fancy apostrophe. What about that traceback pointed to the template? (Is it something I could have figured out?)
The key is this entry in the traceback.
File "/usr/lib/python3.8/site-packages/mailman/chains/hold.py", line 234, in _process text = MIMEText(expand(template, mlist, substitutions),
This says we are trying to make a MIME text message part based on expanding the substitutions in a template. A look at the source at https://gitlab.com/mailman/mailman/-/blob/master/src/mailman/chains/hold.py#... says the template is list:admin:action:post.
Is it possible to filter out UTF-8 chars from the templates when they are saved in Postorius?
Not really. Postorius would have to know the character set of the list's preferred_language in order to filter anything from the template that wasn't a valid character in that character set, and I don't think it has a way to get that information from Mailman core.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
participants (2)
-
Mark Sapiro -
Seth Seeger