Shorted Subjects in Mailman
Hello all,
We have been experiencing the following problem lately: When a user sends a mail with an unusually long subject to a mailing list, this subject is shortened. Here are the two examples:
Original subject: "[SAV-LINUX] Notice from Sophos Anti-Virus on lx1bcms-storage.physik.rwth-aachen.de" (82 characters long)
Subject in Mailman: "[edv-support][SAV-LINUX] Notice from Sophos Anti-Virus on" (57 characters long)
Is there a rule that Mailman truncates the subject or why does this happen? Is there any way to work around this? In the Mailman Topic Header, however, the full subject is always readable.
Translated with www.DeepL.com/Translator (free version)
Stephan Krinetzki writes:
We have been experiencing the following problem lately: When a user sends a mail with an unusually long subject to a mailing list, this subject is shortened.
Is this being observed by all subscribers? Especially subscribers with various different mail clients? Do you observe this when you look at the "original message"? (Different clients describe this function in different ways, but most mail clients allow you to look at all of the header and body text exactly as sent, with no formatting).
Here are the two examples:
Original subject: "[SAV-LINUX] Notice from Sophos Anti-Virus on lx1bcms-storage.physik.rwth-aachen.de" (82 characters long)
Subject in Mailman: "[edv-support][SAV-LINUX] Notice from Sophos Anti-Virus on" (57 characters long)
By design, Mailman does not truncate the subject, it folds it to multiple lines to conform to Internet mail standards which state that lines should be no longer than 78 characters. So, including adding the "edv-support" tag, a stock Mailman installation should be changing:
Subject: [SAV-LINUX] Notice from Sophos Anti-Virus on lx1bcms-storage.physik.rwth-aachen.de"
(all one line) to (two lines, split after "on" with the space starting the second line[1]:
Subject: [edv-support] [SAV-LINUX] Notice from Sophos Anti-Virus on lx1bcms-storage.physik.rwth-aachen.de"
A mail client that conforms to Internet mail standards might either display it exactly that way, or it might unfold it to a single line, or it might unfold it, then refold it to match the user's display width.
Of course if you look at the original text and there is no second line, that's a Mailman bug.
Is there any way to work around this?
If Mailman is folding the subject as described above, you'd have to patch the code, and maintain the patch across upgrades. I think the patch might be a little tricky because the code that does this is in the Python distribution, not in Mailman itself, if I remember correctly.
Alteratively, get people to use clients that properly handle Internet mail. Asking people to change their mail clients rarely goes over well though.
Regards, Steve
Footnotes: [1] The mail standards permit splitting header lines only on whitespace under certain conditions, not at hyphens and other punctuation as normal in English text.
Stephen J. Turnbull wrote:
Stephan Krinetzki writes:
We have been experiencing the following problem lately: When a user sends a mail with an unusually long subject to a mailing list, this subject is shortened. Is this being observed by all subscribers? Especially subscribers with various different mail clients? Do you observe this when you look at the "original message"? (Different clients describe this function in different ways, but most mail clients allow you to look at all of the header and body text exactly as sent, with no formatting).
Yes, it doesn't matter which cleint they are using. The cutted subject was reported multiple times.
Here are the two examples: Original subject: "[SAV-LINUX] Notice from Sophos Anti-Virus on lx1bcms-storage.physik.rwth-aachen.de" (82 characters long) Subject in Mailman: "[edv-support][SAV-LINUX] Notice from Sophos Anti-Virus on" (57 characters long) By design, Mailman does not truncate the subject, it folds it to multiple lines to conform to Internet mail standards which state that lines should be no longer than 78 characters. So, including adding the "edv-support" tag, a stock Mailman installation should be changing: Subject: [SAV-LINUX] Notice from Sophos Anti-Virus on lx1bcms-storage.physik.rwth-aachen.de" (all one line) to (two lines, split after "on" with the space starting the second line[1]: Subject: [edv-support] [SAV-LINUX] Notice from Sophos Anti-Virus on lx1bcms-storage.physik.rwth-aachen.de" A mail client that conforms to Internet mail standards might either display it exactly that way, or it might unfold it to a single line, or it might unfold it, then refold it to match the user's display width. Of course if you look at the original text and there is no second line, that's a Mailman bug.
And here we are: There is no second line! The subject header only contains the displayed value, no new line or other things. But the mailman subject line is intact. Only the regular Subject: is truncated.
So, i have to file a bug report, right?
Stephan Krinetzki writes:
Yes, it doesn't matter which cleint they are using. The cutted subject was reported multiple times.
That doesn't tell me that they use different clients, though. Hundreds of millions of people use Gmail, for one. All of them would report the same problem if it occurred, but that could be a Gmail bug.
Of course if you look at the original text and there is no second line, that's a Mailman bug.
And here we are: There is no second line! The subject header only contains the displayed value, no new line or other things. But the mailman subject line is intact. Only the regular Subject: is truncated.
I don't understand what you mean by "mailman subject line" and "regular Subject:". Are you talking about a digest that contains multiple messages? If so, explain more completely what you mean by those terms. It would help a lot if you would forward to us a complete raw message that shows the problem.
Most mail clients have a "forward" function that will do this. A "reply" with the mail quoted does not help very much. A screenshot does not help very much. We need to see some information that is normally hidden by clients and not including in quoted replies.
Steve
Hi Steve,
here are some examples:
https://pastebin.com/xRUyV0n1 https://pastebin.com/0ehnkFBg https://pastebin.com/1JUwJut8
Regards
Stephan
Hi again,
and with the "Mailman header" I mean "Thread-Topic" like in https://pastebin.com/0ehnkFBg
Regards
Stephan
P.S: I collecting more Mails, 2 of 3 are recieved by Mozilla Thunderbird
Stephan Krinetzki writes:
and with the "Mailman header" I mean "Thread-Topic" like in https://pastebin.com/0ehnkFBg
That's not Mailman. It appears to be a Microsoft thing from a quick web search. No RFCs appear to define it. (That's not to say Microsoft can't use it, just that it's not an Internet mail standard.)
Steve
On mar, 2022-07-26 at 09:50 +0000, Stephan Krinetzki wrote:
And again: The Mail from the beginning https://pastebin.com/gsVtvKm1
I'm not sure if Mailman does the transformation of the subject to Quoted Printable, but, I think that whatever is doing that is the most likely culprit, as "it" has added the QP terminator (?=) where the subject line breaks:
Subject: =?utf-8?q?=5Bedv-support=5D=5BSAV- LINUX=5D_Notice_from_Sophos_Anti-Virus_on?=
-- Victoriano Giralt Innovation Director Digital Transformation Vicerectorate University of Malaga +34952131415 SPAIN
Note: signature.asc is the electronic signature of present message A: Yes.
Q: Are you sure ?
A: Because it reverses the logical flow of conversation.
Q: Why is top posting annoying in email ?
Hi,
i tried now with a Postfix 3.x -> Mailman
Original to Mailman: https://pastebin.com/c4BHeEkJ After Recieving: https://pastebin.com/JG0NKbjq
So yes, somwhere it gets QP.
Regards
Stephan
On mar, 2022-07-26 at 10:09 +0000, Stephan Krinetzki wrote:
Hi,
Hi,
i tried now with a Postfix 3.x -> Mailman
Postfix has en extremely useful setting called debug_peer_list where you can list addresses of machines whose messages you are interested in and the level of detail can be set with debug_peer_level. this will result on very detailed logging of the SMTP conversation.
But, before resorting to this very invasive method, I'd put messages on the HOLD queue and inspect them with postcat, before releasing them to be passed to Mailman. this way, you will be able to see if the subjects are cur "before" reaching Mailman.
Original to Mailman: https://pastebin.com/c4BHeEkJ
Subject: =?utf-8?q?=5BJaninsliste=5D_=5BSAV- LINUX=5D_Notice_from_Sophos_Anti-Virus_on_server=2Eexample=2Ecom?=
After Recieving: https://pastebin.com/JG0NKbjq
Subject: =?utf-8?q?=5BJaninsliste=5D_=5BSAV- LINUX=5D_Notice_from_Sophos_Anti-Virus_on_server=2Eexample=2Ecom?= Both of then show the same Subject: header.
So ...
There is a DKIM signer somewhere fiddling with the subject (twice actually):
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=krinetzki.de; s=2019; t=1658829931; h=from:from:reply-to:subject:subject:date:date:message- id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding;
-- Victoriano Giralt Innovation Director Digital Transformation Vicerectorate University of Malaga +34952131415 SPAIN
Note: signature.asc is the electronic signature of present message A: Yes.
Q: Are you sure ?
A: Because it reverses the logical flow of conversation.
Q: Why is top posting annoying in email ?
Victoriano Giralt wrote:
Hi, Hi, i tried now with a Postfix 3.x -> Mailman Postfix has en extremely useful setting called debug_peer_list where you can list addresses of machines whose messages you are interested in and the level of detail can be set with debug_peer_level. this will result on very detailed logging of the SMTP conversation. But, before resorting to this very invasive method, I'd put messages on
On mar, 2022-07-26 at 10:09 +0000, Stephan Krinetzki wrote: the HOLD queue and inspect them with postcat, before releasing them to be passed to Mailman. this way, you will be able to see if the subjects are cur "before" reaching Mailman.
Original to Mailman: https://pastebin.com/c4BHeEkJ Subject: =?utf-8?q?=5BJaninsliste=5D_=5BSAV- LINUX=5D_Notice_from_Sophos_Anti-Virus_on_server=2Eexample=2Ecom?= After Recieving: https://pastebin.com/JG0NKbjq Subject: =?utf-8?q?=5BJaninsliste=5D_=5BSAV- LINUX=5D_Notice_from_Sophos_Anti-Virus_on_server=2Eexample=2Ecom?= Both of then show the same Subject: header.
In https://pastebin.com/c4BHeEkJ, Line 79+80 it says
Subject: [SAV-LINUX] Notice from Sophos Anti-Virus on server.example.com [DEMO1]
So this correct.
in https://pastebin.com/JG0NKbjq, Line 177 its truncated
Subject: =?utf-8?q?=5BJaninsliste=5D_=5BSAV-LINUX=5D_Notice_from_Sophos_Anti-Virus_on_server=2Eexample=2Ecom?=
So my guess is, that the local Postfix on the Mailman Server is doing the QP. In the Chain there are
postfix 3.x -> MS Exchange -> Postfix 2.x -> Mailman
After Releasing the mail from hold Mailman -> Postfix 2.x -> Exchange
Regards
Stephan
On 7/26/22 03:44, Stephan Krinetzki wrote:
Victoriano Giralt wrote:
Hi, Hi, i tried now with a Postfix 3.x -> Mailman Postfix has en extremely useful setting called debug_peer_list where you can list addresses of machines whose messages you are interested in and the level of detail can be set with debug_peer_level. this will result on very detailed logging of the SMTP conversation. But, before resorting to this very invasive method, I'd put messages on
On mar, 2022-07-26 at 10:09 +0000, Stephan Krinetzki wrote: the HOLD queue and inspect them with postcat, before releasing them to be passed to Mailman. this way, you will be able to see if the subjects are cur "before" reaching Mailman.
Original to Mailman: https://pastebin.com/c4BHeEkJ Subject: =?utf-8?q?=5BJaninsliste=5D_=5BSAV- LINUX=5D_Notice_from_Sophos_Anti-Virus_on_server=2Eexample=2Ecom?= After Recieving: https://pastebin.com/JG0NKbjq Subject: =?utf-8?q?=5BJaninsliste=5D_=5BSAV- LINUX=5D_Notice_from_Sophos_Anti-Virus_on_server=2Eexample=2Ecom?= Both of then show the same Subject: header.
In https://pastebin.com/c4BHeEkJ, Line 79+80 it says
Subject: [SAV-LINUX] Notice from Sophos Anti-Virus on server.example.com [DEMO1]
In the original message you posted at https://pastebin.com/c4BHeEkJ, the
continuation line [DEMO1] is not indented and is thus not a
continuation of the Subject: header but is a separate line. It appears
that several other continuation lines are also not indented, so I can't
tell what the original message to Mailman actually looked like.
However, I did a test injecting directly to Mailman a message with
Subject: [SAV-LINUX] Notice from Sophos_Anti-Virus on server.example.com
[DEMO1]
on a German language test list with subject prefix [Janinsliste] and I get the same result, I.e. the RFC 2047 encoded
Subject:
=?utf-8?q?=5BJaninsliste=5D=5BSAV-LINUX=5D_Notice_from_Sophos=5FAnti-Virus_on_server=2Eexample=2Ecom?=
so this is definitely a Mailman bug (or a bug in the underlying Python email package.
Thank you for the report. I have created https://gitlab.com/mailman/mailman/-/issues/1021 for this issue.
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mark Sapiro writes:
However, I did a test injecting directly to Mailman a message with
Subject: [SAV-LINUX] Notice from Sophos_Anti-Virus on server.example.com [DEMO1]on a German language test list with subject prefix [Janinsliste] and I get the same result, I.e. the RFC 2047 encoded
Subject: =?utf-8?q?=5BJaninsliste=5D=5BSAV-LINUX=5D_Notice_from_Sophos=5FAnti-Virus_on_server=2Eexample=2Ecom?=so this is definitely a Mailman bug (or a bug in the underlying Python email package.
There seem to be other bugs here.
First, I can't see any content-based reason for encoding this 100% ASCII content. I guess the list is set to always encode the subject prefix, and we just encode the whole field body if any part is encoded?
Perhaps this is a quick workaround for the OP: set MIME-encoding for the subject prefix to "as needed".
Second (and this is important because of RFC conformance!), the encoded word is over 75 characters. I've filed https://gitlab.com/mailman/mailman/-/issues/1021 for this.
participants (4)
-
Mark Sapiro -
Stephan Krinetzki -
Stephen J. Turnbull -
Victoriano Giralt