Hi,

Am 10. Dezember 2019 um 17:09 Uhr +0900 schrieb Stephen J. Turnbull:

You're the "it's 2019, let's use SMTP UTF8 everywhere" guy, right?

No, I'm not that guy. I've not said anything like that, nor have I knowingly configured Postfix to use that. I've looked up how to do it now that you mention it, and found the smtputf8_enable parameter1. I have not used it, but it turns out that it is enabled by default. I can turn it off and see if that makes any difference in the future. Testing with telnet, Postfix indeed announced "250 SMTPUTF8" on EHLO.

If you're right about the umlaut being the trigger, the first thing I'd look at is a feature-negotiation problem in the MTA delivering to Mailman. I'm pretty sure our LMTP does not offer the SMTP UTF8 feature.

In that case the problem should go away with disabling SMTPUTF8, I suppose?

I would guess this is in the shunt queue. If there's only one file there, you can just delete it.

There was a whole bunch of files in that directory. I've taken a look several of them, and as they were all spam, I've taken the liberty to delete them all (this list is so low volume that I know all the senders personally anyway). However, even after restarting mailman, the 500 error persists when visiting the held messages page in postorius.

I've also looked into all the other queue directories, they were all empty.

Assuming you're right about your name being the trigger, U+FFFD REPLACEMENT CHARACTER is not in your name.

I confess that I haven't visited the held messages page for a very long time (because I didn't have a reason to; I received no notifications of held messages). Only after I received the notification that my errorneous message was being held, I visited it, and I was faced with the 500 error. If Mailman failed to notify me of held messages previously, they might have been in hold status already and one of them might be the cause.

Marvin

PS: Sorry for the missing link 1 from the last mail. Here it is: https://gitlab.com/mailman/hyperkitty/issues/201

-- Blog: https://mg.guelker.eu

Marvin Gülker writes:

Am 10. Dezember 2019 um 17:09 Uhr +0900 schrieb Stephen J. Turnbull:

...

You're the "it's 2019, let's use SMTP UTF8 everywhere" guy, right?

No, I'm not that guy.

I'm sorry about that. I knew I didn't know for sure, but not being able to get at the moderation queue seemed urgent enough to be worth the chance of being wrong.

I have not used it, but it turns out that it is enabled by default. I can turn it off and see if that makes any difference in the future. Testing with telnet, Postfix indeed announced "250 SMTPUTF8" on EHLO.

...

If you're right about the umlaut being the trigger, the first thing I'd look at is a feature-negotiation problem in the MTA delivering to Mailman. I'm pretty sure our LMTP does not offer the SMTP UTF8 feature.

In that case the problem should go away with disabling SMTPUTF8, I suppose?

Well, since it's predicated on a bug in Postfix, I wouldn't bet on it, especially since Postfix probably dominates Mailman 3 installations. But it's cheap to test, and since you don't explicitly want SMTP UTF8 at this point, little harm in turning it off. *If* there is a bug, that will prevent the problem in the future, but it wouldn't fix the problem in the held messages queue.

...

I would guess this is in the shunt queue. If there's only one file there, you can just delete it.

This was a bad guess. I'm not sure why it isn't in the shunt queue (it shouldn't have been able to escape into the main rule chains with that bad breakage in the header), but Mailman doesn't try to moderate shunted messages, that's the whole point of the shunt queue -- they're out of the way.

There was a whole bunch of files in that directory. I've taken a look several of them, and as they were all spam, I've taken the liberty to delete them all (this list is so low volume that I know all the senders personally anyway).

A tiny bit of good came of it, at least.

However, even after restarting mailman, the 500 error persists when visiting the held messages page in postorius.

I've also looked into all the other queue directories, they were all empty.

I guess it's in the MessageStore, then, which is by default in /var/tmp/mailman/messages/. (Defined in schema.cfg.) I guess it's a standard qfile that you can examine with "mailman qfiles /path/to/file". (I've never actually looked at a "raw" held message file in Mailman 3, haven't had a failure of this type.) If not, you can unpickle it with Python (if that makes no sense to you, ask; I'm running out of steam and need sleep).

...

Assuming you're right about your name being the trigger, U+FFFD REPLACEMENT CHARACTER is not in your name.

I'm starting to wonder about this. It's still possible, but if you don't deliberately use SMTP UTF8 then your mail client probably doesn't either, and equally possibly it's a different message causing the problem (something spammy).

Hi,

Am 10. Dezember 2019 um 10:25 Uhr -0800 schrieb Mark Sapiro:

The underlying issue here is probably <https://bugs.python.org/issue32330>, but that knowledge doesn't help us. We need to get rid of the offending message in the message store. The first step is in mailman shell. Note the paths /opt/mailman/mm/bin/ and /opt/mailman/mm/var/ will need to be adjusted.

Thanks for this information. Ubuntu's mailman stores the messages below /var/lib/mailman3/messages. The "mailman" command is in the $PATH.

This should print the file name and Message-ID-Hash (two lines the same) for each message in the messages directory until it prints only one line followed by the UnicodeEncodeError exception. That list line is the file name/Message-ID-Hash of the offending message.

I've execucted that code, but it does not cause an exception at all. It quietly runs through. I've attached the transcript of the shell session to this mail. With a little crude use of grep(1), however I found my held message in WQ/ZS/WQZSFFJLVMWQHMG4VYKGXJEKQVQXVK67.

Does the messages directory contain only held messages? I've looked at some messages and all of these I looked at were spam. The directory structure contains 40 messages in total, accumulating to 480 KiB disk space.

If required, I can zip that directory up and e-mail it to you for inspection.

DELETE FROM message WHERE message_id_hash = xxx;

As the exception you expected did not occur, I didn't do this yet. I can make a database and directory backup and then run it, though. Should I try?

I assume after this I should also delete the respective files from the messages/ directory?

As of Mailman core 3.3.0 there is a 'mailman notify' command designed to be run periodically by cron to notify owners/moderators of prnding requests.

I did receive an e-mail to the list owner address that told me that the e-mail from my incorrect e-mail address "requires approval". That was the first time I received such an e-mail. In parallel, on the incorrect e-mail address, I received the notification for moderator approval.

Marvin

-- Blog: https://mg.guelker.eu

On 12/11/19 12:43 AM, Marvin Gülker wrote:

Thanks for this information. Ubuntu's mailman stores the messages below /var/lib/mailman3/messages. The "mailman" command is in the $PATH.

...

This should print the file name and Message-ID-Hash (two lines the same) for each message in the messages directory until it prints only one line followed by the UnicodeEncodeError exception. That list line is the file name/Message-ID-Hash of the offending message.

I've execucted that code, but it does not cause an exception at all. It quietly runs through. I've attached the transcript of the shell session to this mail. With a little crude use of grep(1), however I found my held message in WQ/ZS/WQZSFFJLVMWQHMG4VYKGXJEKQVQXVK67.

Here's a revised script:

$ mailman shell
Welcome to the GNU Mailman shell

>>> import os
>>> import pickle
>>> for root, dirs, files in os.walk('/var/lib/mailman3/messages'):
...     for fn in files:
...         print(fn)
...         with open(os.path.join(root, fn), 'rb') as fp:
...             msg = pickle.load(fp)
...             print(msg['Message-ID'])
...             x = msg.as_string()
...

I have adjusted the paths for your package, changed the second print to print the message-is and added the x = msg.as_string() line. Now we expect to see the hash of each message-id followed by the message-id followed by the exception for the offending message.

Does the messages directory contain only held messages? I've looked at some messages and all of these I looked at were spam. The directory structure contains 40 messages in total, accumulating to 480 KiB disk space.

Yes, the directory and the corresponding database table contains only messages held for moderator approval.

...

DELETE FROM message WHERE message_id_hash = xxx;

As the exception you expected did not occur, I didn't do this yet. I can make a database and directory backup and then run it, though. Should I try?

I assume after this I should also delete the respective files from the messages/ directory?

Actually, a better way to deal with this is in mailman shell

Welcome to the GNU Mailman shell

>>> ms = getUtility(IMessageStore)
>>> ms.delete_message('xxx')
>>>

where xxx is the message-id value printed from the above script.

...

As of Mailman core 3.3.0 there is a 'mailman notify' command designed to be run periodically by cron to notify owners/moderators of pending requests.

I did receive an e-mail to the list owner address that told me that the e-mail from my incorrect e-mail address "requires approval". That was the first time I received such an e-mail. In parallel, on the incorrect e-mail address, I received the notification for moderator approval.

Yes, those are the only messages unless you are running mailman notify. It seems however that the latest Debian (hence Ubuntu) package doesn't have 3.3 yet.

As to why you didn't receive owner notices about the held spam, perhaps they were spam filtered or somehow otherwise lost.

-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

On 12/11/19 11:40 AM, Marvin Gülker wrote:

Success! This time it crashes with the expected exception. I've attached the shell transcript.

Good.

As expected, it's a spam message. I've uploaded the offending file here for you to look at: <https://ftp.secretchronicles.org/misc/Q5PQZI4QEE4C42XNZMXESSKDCSZBUT4T>

I get a Forbidden status on that URL.

-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

On 12/11/19 12:53 PM, Marvin Gülker wrote:

Am 11. Dezember 2019 um 11:48 Uhr -0800 schrieb Mark Sapiro:

...

I get a Forbidden status on that URL.

Oops. cp copied the file permissions and I didn't check. Should work now.

OK. I have verified that this is due to <https://bugs.python.org/issue32330> and has actually been reported as <https://gitlab.com/mailman/mailman/issues/441> and worked around in Mailman by <https://gitlab.com/mailman/mailman/merge_requests/350> which is included in Mailman 3.2.0 and up.

-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

On 12/12/19 12:42 AM, Marvin Gülker wrote:

But I now have a different problem. I've removed all offending messages (it were more than just one that caused the UnicodeEncodeError), but I still can't access the page for the held messages queue. It gives an error 500, now with a different exception in the log. I've included it below. It looks to me that the messages haven't properly been deleted.

They have, but I apologize - more is required.

What I did was this: for each offending message hash

1. I ran the DELETE query
2. I deleted the file from the file system
3. I ran the delete_message() function

I originally wanted to run only the delete_message() function, but a quick check on the DB and the file system after running it for one hash told me that it neither deletes the message from the DB nor from the file system.

The argument for delete_message() is not the hash, it is the <message-id> including the angle brackets.

So I did everything at once. I do have a backup of both the

DB and the filesystem from before I started this, in case this broke something.

Is there something left that I have to do now?

Yes, my bad for not realizing this.

There are still references to these messages in the _request table. The columns in that table are

id | key | request_type | data_hash | mailing_list_id

key is the <message-id> of the request message request_type is 1 for a held message mailing_list_id is a numeric index into the mailinglist table

The issue is there are still held message requests there for the messages you removed. You need to delete those entries from the _request table.

-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan