On 11/3/20 6:43 PM, Mark Sapiro wrote:

On 11/3/20 3:32 PM, Brian Carpenter wrote:

...

Hello Everyone,

There was a message sent to a list that could not be archived by Hyperkitty for the following reason:

WARNING 2020-11-03 17:20:54,507 800 hyperkitty.views.mailman Could not archive the email with message-id '<67BEBD75-B3DC-45E3-9C2F-55482F811238@ispdomain.com>': A string literal cannot contain NUL (0x00) characters.

It looks like there was a problem with the email message itself. What causes such an error? A null byte in the message body.

I suspect the message is still queued in Mailman's var/archives/hyperkitty/spool/ directory and is being retried with each new post. You can examine the message with Mailman's mailman qfile command.

Thanks Mark. I think the sender embedded a PNG file into the body of the message instead of adding it as an attachment.

-- Brian Carpenter Harmonylists.com Emwd.com

On 11/11/20 2:11 PM, Brian Carpenter wrote:

This happened again. Is there a way to get HK to deal with such messages automatically?

How would you like them dealt with? The issue is in mailman-hyperkitty. When it's attempt to archive the message returns an error, it queues the message for retry. It is not clear if any of these errors are actually retryable - for sure some aren't but maybe others like HyperKitty not responding are - but even if we think retry won't succeed what would you like done with the message?

-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

On 11/11/20 4:18 PM, Brian Carpenter wrote:

Some sort of auto-discard procedure after a number of retries? Is it possible to detect a message with a null characters and automatically discard them? Do such messages make it into the thread table?

The message has been delivered to the list, but it is not archived. The list post has an Archived-At: header with a URL, but the message isn't there at all. The only way to get it archived is to correct the message defect.

Thus, I don't think it's good to just discard the message at that point. I'm fine with moving out of or not queueing it in the first place in var/archives/hyperkitty/spool, but I think it should be preserved somewhere, say in var/archives/hyperkitty/bad, and along with that, some kind of notice sent to the site admin so bad messages don't just accumulate without being seen.

-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

On 11/12/20 6:59 AM, Brian Carpenter wrote:

On 11/11/20 7:40 PM, Mark Sapiro wrote:

...

The message has been delivered to the list, but it is not archived. The list post has an Archived-At: header with a URL, but the message isn't there at all. The only way to get it archived is to correct the message defect.

Does the problematic message (null character) make it to the hyperkitty_thread table?

I thought that saying it is not archived would answer that, but no, nothing about that message is stored in any hyperkitty table.

Can a message with a null character be repaired with an automatic process?

Probably, but I don't think HyperKitty should be in the business of attempting to repair message defects. This one issue of a null byte is arguably not a defective message, but rather an issue of storing a null byte in a database field, so maybe something could be done to cleanse that data before storing it.

If I were going to look at doing that, I would need an example of a problem message, and ideally, an issue in the tracker.

-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan