Message is received but is not sent out. Another, similar message, is processed fine.
Hi all,
One of my subscribers is trying to send a message to a list, but it is not being posted. The smtp.log shows that her first message was posted, but a second message (with 2 retries) was not:
mailman@list:~/logs$ grep herName smtp.log Jun 18 11:22:05 2023 (1243316) ('127.0.0.1', 56686) >> b'MAIL FROM:<herName@hotmail.com> SIZE=40109' Jun 18 11:22:05 2023 (1243316) ('127.0.0.1', 56686) sender: herName@hotmail.com Jun 18 11:25:01 2023 (1243318) <CH3PR12MB82593510B90D8F05177C606BBE5EA@CH3PR12MB8259.namprd12.prod.outlook.com> post to theList@domain.tlc from herName@hotmail.com, 40061 bytes Jun 18 11:52:42 2023 (1243316) ('127.0.0.1', 48448) >> b'MAIL FROM:<herName@hotmail.com> SIZE=18343' Jun 18 11:52:42 2023 (1243316) ('127.0.0.1', 48448) sender: herName@hotmail.com Jun 18 19:01:04 2023 (1243316) ('127.0.0.1', 59550) >> b'MAIL FROM:<herName@hotmail.com> SIZE=19123' Jun 18 19:01:04 2023 (1243316) ('127.0.0.1', 59550) sender: herName@hotmail.com Jun 26 17:18:11 2023 (1243316) ('127.0.0.1', 44732) >> b'MAIL FROM:<herName@hotmail.com> SIZE=18930' Jun 26 17:18:11 2023 (1243316) ('127.0.0.1', 44732) sender: herName@hotmail.com
At the second retry she copied me and I saw nothing untowards in her message. She's not getting any error return messages and her subscription is OK. Both messages were multipart/alternative and both contained Hebrew text.
I told her I'd look at this, but I'm at a loss.
Yours,
Allan
On 6/26/23 8:09 PM, Allan Hansen wrote:
Hi all,
One of my subscribers is trying to send a message to a list, but it is not being posted. The smtp.log shows that her first message was posted, but a second message (with 2 retries) was not:
mailman@list:~/logs$ grep herName smtp.log Jun 18 11:22:05 2023 (1243316) ('127.0.0.1', 56686) >> b'MAIL FROM:<herName@hotmail.com> SIZE=40109' Jun 18 11:22:05 2023 (1243316) ('127.0.0.1', 56686) sender: herName@hotmail.com Jun 18 11:25:01 2023 (1243318) <CH3PR12MB82593510B90D8F05177C606BBE5EA@CH3PR12MB8259.namprd12.prod.outlook.com> post to theList@domain.tlc from herName@hotmail.com, 40061 bytes Jun 18 11:52:42 2023 (1243316) ('127.0.0.1', 48448) >> b'MAIL FROM:<herName@hotmail.com> SIZE=18343' Jun 18 11:52:42 2023 (1243316) ('127.0.0.1', 48448) sender: herName@hotmail.com Jun 18 19:01:04 2023 (1243316) ('127.0.0.1', 59550) >> b'MAIL FROM:<herName@hotmail.com> SIZE=19123' Jun 18 19:01:04 2023 (1243316) ('127.0.0.1', 59550) sender: herName@hotmail.com Jun 26 17:18:11 2023 (1243316) ('127.0.0.1', 44732) >> b'MAIL FROM:<herName@hotmail.com> SIZE=18930' Jun 26 17:18:11 2023 (1243316) ('127.0.0.1', 44732) sender: herName@hotmail.com
At the second retry she copied me and I saw nothing untowards in her message. She's not getting any error return messages and her subscription is OK. Both messages were multipart/alternative and both contained Hebrew text.
I told her I'd look at this, but I'm at a loss.
smtp.log is not very useful for diagnosing things like this. It only says a message was received, not why it didn't result in a post from the list as in the first case. What's in mailman.log with those timestamps?
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Thank you, Mark. I see this in mailman.log. I had looked there for her email address, but it does not appear, so I moved on to the smtp.log.
In
Jun 26 17:18:11 2023 (1243315) ACCEPT: ^M <CH3PR12MB82593801A1DF2E329E977046BE27A@CH3PR12MB8259.namprd12.prod.outlook.com> Jun 26 17:18:11 2023 (1243319) Uncaught runner exception: unknown encoding: iso-8859-8-i Jun 26 17:18:11 2023 (1243319) Traceback (most recent call last): File "/opt/mailman/mm/venv/lib/python3.9/site-packages/mailman/core/runner.py", line 179, in _one_iteration self._process_one_file(msg, msgdata) File "/opt/mailman/mm/venv/lib/python3.9/site-packages/mailman/core/runner.py", line 272, in _process_one_file keepqueued = self._dispose(mlist, msg, msgdata) File "/opt/mailman/mm/venv/lib/python3.9/site-packages/mailman/runners/pipeline.py", line 37, in _dispose process(mlist, msg, msgdata, pipeline) File "/opt/mailman/mm/venv/lib/python3.9/site-packages/mailman/core/pipelines.py", line 53, in process handler.process(mlist, msg, msgdata) File "/opt/mailman/mm/venv/lib/python3.9/site-packages/mailman/handlers/mime_delete.py", line 382, in process process(mlist, msg, msgdata) File "/opt/mailman/mm/venv/lib/python3.9/site-packages/mailman/handlers/mime_delete.py", line 151, in process reset_payload(msg, firstalt) File "/opt/mailman/mm/venv/lib/python3.9/site-packages/mailman/handlers/mime_delete.py", line 202, in reset_payload msg.set_payload(subpart.get_payload(decode=True).decode( LookupError: unknown encoding: iso-8859-8-i Jun 26 17:18:11 2023 (1243319) SHUNTING: 1687825091.8457294+478565324778f2eb6f0fecaeccbd29596edaef53
I also see this in queue/shunt:
mailman@list:~/var/queue/shunt$ ls -ltr total 88 -rw-rw---- 1 mailman mailman 7168 Feb 17 23:03 1676703781.903885+fdcba88c680f50a0799c672cd0abe82e4bfe19b3.pck -rw-rw---- 1 mailman mailman 19821 Apr 2 07:08 1680444501.0831451+e110b8c5c20ba3f87c1cea1c5f38e74a6cf8f1c5.pck -rw-rw---- 1 mailman mailman 19489 Jun 18 11:52 1687114364.41209+1684717da96ff76d1ce773a0cf39c5f8b63e9078.pck -rw-rw---- 1 mailman mailman 20274 Jun 18 19:01 1687140067.64057+7f77c1ebca490831f58ef9cfde06109081a4e7d6.pck -rw-rw---- 1 mailman mailman 20085 Jun 26 17:18 1687825091.8457294+478565324778f2eb6f0fecaeccbd29596edaef53.pck
The three last entries coincide with the captured messages.
Finally, in the captured message source (the last which was Cc'd to me) I do see this:
Content-Type: text/plain; charset="iso-8859-8-i"
Which matches the complaint in the mailman log.
In the message that did get accepted, I see this:
Content-Type: text/plain; charset="utf-8"
Another difference: The posted message shows Content-Transfer-Encoding: base64 The shunted message shows Content-Transfer-Encoding: quoted-printable
It appears that the two messages are, indeed, different, despite the similar looks when rendered.
My question thus changes to:
a. Should she be doing something special when sending to the lists?
b. If not, is my setup in need of change?Yours,
AllanOn 6/26/23, 20:44, "Mark Sapiro" <mark@msapiro.net <mailto:mark@msapiro.net>> wrote:
On 6/26/23 8:09 PM, Allan Hansen wrote:
Hi all,
One of my subscribers is trying to send a message to a list, but it is not being posted. The smtp.log shows that her first message was posted, but a second message (with 2 retries) was not:
mailman@list:~/logs$ grep herName smtp.log Jun 18 11:22:05 2023 (1243316) ('127.0.0.1', 56686) >> b'MAIL FROM:<herName@hotmail.com <mailto:herName@hotmail.com>> SIZE=40109' Jun 18 11:22:05 2023 (1243316) ('127.0.0.1', 56686) sender: herName@hotmail.com <mailto:herName@hotmail.com> Jun 18 11:25:01 2023 (1243318) <CH3PR12MB82593510B90D8F05177C606BBE5EA@CH3PR12MB8259.namprd12.prod.outlook.com <mailto:CH3PR12MB82593510B90D8F05177C606BBE5EA@CH3PR12MB8259.namprd12.prod.outlook.com>> post to theList@domain.tlc <mailto:theList@domain.tlc> from herName@hotmail.com <mailto:herName@hotmail.com>, 40061 bytes Jun 18 11:52:42 2023 (1243316) ('127.0.0.1', 48448) >> b'MAIL FROM:<herName@hotmail.com <mailto:herName@hotmail.com>> SIZE=18343' Jun 18 11:52:42 2023 (1243316) ('127.0.0.1', 48448) sender: herName@hotmail.com <mailto:herName@hotmail.com> Jun 18 19:01:04 2023 (1243316) ('127.0.0.1', 59550) >> b'MAIL FROM:<herName@hotmail.com <mailto:herName@hotmail.com>> SIZE=19123' Jun 18 19:01:04 2023 (1243316) ('127.0.0.1', 59550) sender: herName@hotmail.com <mailto:herName@hotmail.com> Jun 26 17:18:11 2023 (1243316) ('127.0.0.1', 44732) >> b'MAIL FROM:<herName@hotmail.com <mailto:herName@hotmail.com>> SIZE=18930' Jun 26 17:18:11 2023 (1243316) ('127.0.0.1', 44732) sender: herName@hotmail.com <mailto:herName@hotmail.com>
At the second retry she copied me and I saw nothing untowards in her message. She's not getting any error return messages and her subscription is OK. Both messages were multipart/alternative and both contained Hebrew text.
I told her I'd look at this, but I'm at a loss.
smtp.log is not very useful for diagnosing things like this. It only says a message was received, not why it didn't result in a post from the list as in the first case. What's in mailman.log with those timestamps?
-- Mark Sapiro <mark@msapiro.net <mailto:mark@msapiro.net>> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mailman-users mailing list -- mailman-users@mailman3.org <mailto:mailman-users@mailman3.org> To unsubscribe send an email to mailman-users-leave@mailman3.org <mailto:mailman-users-leave@mailman3.org> https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/ <https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/> Archived at: https://lists.mailman3.org/archives/list/mailman-users@mailman3.org <mailto:mailman-users@mailman3.org>/message/F4K7HYUYSWEKF6LT35UCT3KBJX6GHY4I/
This message sent to hansen@rc.org <mailto:hansen@rc.org>
Allan Hansen writes:
LookupError: unknown encoding: iso-8859-8-i
Finally, in the captured message source (the last which was Cc'd to me) I do see this:
Content-Type: text/plain; charset="iso-8859-8-i"
Yes, that's the problem. Python doesn't define a codec for that encoding.
It's generally preferable that mail senders use UTF-8, as that is likely to be implemented correctly in almost all modern mail clients, while the ISO-8859/8 series are likely to be implemented only in mail clients that specifically support Hebrew.
That said, if the users are unable or unwilling to use UTF-8, it would be possible to create a codec for "iso-8859-8-i", I don't know how to do that, and I doubt Mark does. You could ask on python-list@python.org, or file an RFE on Python's GitHub tracker. For Mailman's purpose, there's no need to handle bidi, just codepoint for codepoint translation from "iso-8859-8-i" to Python internal encoding. So I guess the encoder and decoder from "iso-8859-8" (which does exist) can be used.
If it's just code-for-code, then we can figure out how to have it used only for Mailman.
Steve
On 6/27/23 12:30 AM, Stephen J. Turnbull wrote:
That said, if the users are unable or unwilling to use UTF-8, it would be possible to create a codec for "iso-8859-8-i", I don't know how to do that, and I doubt Mark does. You could ask on python-list@python.org, or file an RFE on Python's GitHub tracker.
See https://github.com/python/cpython/issues/90079 filed by me in Nov 2021.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Thank you both. Since the first message she sent was UTF-8 and the second iso-8859-8-I, I have asked her what she did diffently between the two messages. No response yet... But now being aware that this can happen when non-ASCII messages are sent, and how to find the issue when it does, is good.
Yours,
AllanOn 6/27/23, 07:57, "Mark Sapiro" <mark@msapiro.net <mailto:mark@msapiro.net>> wrote:
On 6/27/23 12:30 AM, Stephen J. Turnbull wrote:
That said, if the users are unable or unwilling to use UTF-8, it would be possible to create a codec for "iso-8859-8-i", I don't know how to do that, and I doubt Mark does. You could ask on python-list@python.org <mailto:python-list@python.org>, or file an RFE on Python's GitHub tracker.
See https://github.com/python/cpython/issues/90079 <https://github.com/python/cpython/issues/90079> filed by me in Nov 2021.
-- Mark Sapiro <mark@msapiro.net <mailto:mark@msapiro.net>> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mailman-users mailing list -- mailman-users@mailman3.org <mailto:mailman-users@mailman3.org> To unsubscribe send an email to mailman-users-leave@mailman3.org <mailto:mailman-users-leave@mailman3.org> https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/ <https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/> Archived at: https://lists.mailman3.org/archives/list/mailman-users@mailman3.org <mailto:mailman-users@mailman3.org>/message/4WPWU3SKGWM2BTG5AK3LWNG7FDIBVSTP/
This message sent to hansen@rc.org <mailto:hansen@rc.org>
On 6/26/23 11:00 PM, Allan Hansen wrote:
I also see this in queue/shunt:
mailman@list:~/var/queue/shunt$ ls -ltr total 88 -rw-rw---- 1 mailman mailman 7168 Feb 17 23:03 1676703781.903885+fdcba88c680f50a0799c672cd0abe82e4bfe19b3.pck -rw-rw---- 1 mailman mailman 19821 Apr 2 07:08 1680444501.0831451+e110b8c5c20ba3f87c1cea1c5f38e74a6cf8f1c5.pck -rw-rw---- 1 mailman mailman 19489 Jun 18 11:52 1687114364.41209+1684717da96ff76d1ce773a0cf39c5f8b63e9078.pck -rw-rw---- 1 mailman mailman 20274 Jun 18 19:01 1687140067.64057+7f77c1ebca490831f58ef9cfde06109081a4e7d6.pck -rw-rw---- 1 mailman mailman 20085 Jun 26 17:18 1687825091.8457294+478565324778f2eb6f0fecaeccbd29596edaef53.pck
The three last entries coincide with the captured messages.
I suggest you examine those with mailman qfile and remove any
irrelevant or duplicate messages. If you have mailman.log from Feb or
Apr, you could find the error that caused the shunting and maybe fix it.
Finally, in the captured message source (the last which was Cc'd to me) I do see this:
Content-Type: text/plain; charset="iso-8859-8-i"
Which matches the complaint in the mailman log.
If you want to fix and unshunt any of these, you could do, e.g., in
mailman shell (watch out for wrapped lines)
>>> import pickle
>>> file = '1687114364.41209+1684717da96ff76d1ce773a0cf39c5f8b63e9078.pck'
>>> with open(file, 'rb') as fp:
... msg = pickle.load(fp)
... msgdata = pickle.load(fp)
...
>>> msg.replace_header('Content-Type', 'text/plain; charset="iso-8859-8"')
>>> with open('/path/to/.pck/file', 'wb') as fp:
... pickle.dump(msg, fp)
... pickle.dump(msgdata, fp)
...
>>>
Another difference: The posted message shows Content-Transfer-Encoding: base64 The shunted message shows Content-Transfer-Encoding: quoted-printable
That's not important.
My question thus changes to:
a. Should she be doing something special when sending to the lists?
Get her mail client to send utf-8 or iso-8859-8 encoded messages, not iso-8859-8-i.
b. If not, is my setup in need of change?
See Steve's reply.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
participants (3)
-
Allan Hansen -
Mark Sapiro -
Stephen J. Turnbull