Programmatically modify Thread-Topic *before* message is sent out the door?
We use a custom plugin to do numeric thread stamping, which has been working well in the code system.
However, it was discovered by Microsoft that Microsoft Outlook and OWA prioritize the Thread-Topic header over the Subject header, which results in the oddness we've experienced in MS365 clients in testing.
During our tasks though a custom handler goes through during the pipeline and modifies the Subject field accordingly, adding a thread tag to it. While this is 'fragile' it does work; the only problem is that Thread-Topic doesn't get altered.
Programmatically, how would we alter the Thread-Topic during the handler's processing to prepend the corresponding prefix/tag so that when the message actually goes out the door it is properly sending with the modified Thread-Topic?
Thomas
Thomas Ward via Mailman-users writes:
Programmatically, how would we alter the Thread-Topic during the handler's processing to prepend the corresponding prefix/tag so that when the message actually goes out the door it is properly sending with the modified Thread-Topic?
Same way as you munge the Subject:
msg['thread-topic'] = munge(msg['thread-topic'])
Derp, that makes sense. I thought it was something special. I need more sleep. >.<
Thanks Stephen.
Thomas
-----Original Message----- From: Stephen J. Turnbull <turnbull.stephen.fw@u.tsukuba.ac.jp> Sent: Friday, December 22, 2023 7:03 PM To: Thomas Ward <teward@thomas-ward.net> Cc: mailman-users@mailman3.org Subject: [MM3-users] Programmatically modify Thread-Topic *before* message is sent out the door?
Thomas Ward via Mailman-users writes:
Programmatically, how would we alter the Thread-Topic during the > handler's processing to prepend the corresponding prefix/tag so > that when the message actually goes out the door it is properly > sending with the modified Thread-Topic?
Same way as you munge the Subject:
msg['thread-topic'] = munge(msg['thread-topic'])
Thomas Ward via Mailman-users writes:
Derp,
Not at all! Sure, it makes sense (everything Sempai did makes sense! ;-), but it's not obvious until you know.
that makes sense. I thought it was something special. I need more sleep. >.<
Who doesn't? Let us know if it works as expected.
Steve
Stephen,
Any chance Thread-Topic or such is spawned by the msgdata object instead of directly in the message? We're running with the change of del msg['Thread-Topic']; msg['Thread-Topic'] = Modified Thread Subject Here but it fails to change Thread-Topic. Unfortunately there's very little documentation on msgdata in the handlers and what it is/contains so I'm asking again.
Due to the holidays I didn't get around to testing/poking this so...
Thomas
From: Stephen J. Turnbull <turnbull.stephen.fw@u.tsukuba.ac.jp> Sent: Saturday, December 23, 2023 12:41 AM To: Thomas Ward <teward@thomas-ward.net> Cc: mailman-users@mailman3.org <mailman-users@mailman3.org> Subject: [MM3-users] Re: Programmatically modify Thread-Topic *before* message is sent out the door?
Thomas Ward via Mailman-users writes:
Derp,
Not at all! Sure, it makes sense (everything Sempai did makes sense! ;-), but it's not obvious until you know.
that makes sense. I thought it was something special. I need more sleep. >.<
Who doesn't? Let us know if it works as expected.
Steve
Thomas Ward via Mailman-users writes:
Any chance Thread-Topic or such is spawned by the msgdata object instead of directly in the message?
No, the string "thread-topic" does not appear *anywhere* in the Mailman core sources. Mailman itself has no concept of threads. HyperKitty uses References, Reply-To, and Subject to algorithmically identify threads. Mailman does have an independent concept called "topic", which can best be thought of as "sub-list". But it uses the X-Topics header, and as far as I know doesn't pay attention to threading.
AFAIK, Thread-Topic is a Microsoft-specific device. If you use Exchange or Outlook, you have to negotiate with that software about the use of Thread-Topic. My guess is that it is generated by Exchange because somebody at Microsoft decided that threading should be enforced by the server, not inferred by the MUA.
Unfortunately there's very little documentation on msgdata in the handlers and what it is/contains so I'm asking again.
I guess the relevant documentation is RFC 5321. ;-) msgdata's original purpose is to contain envelope and delivery information.
That is, any data about the message that the LMTP runner knows that is not in the message header itself. HELO hostname, sender, recipients, IP address of SMTP client, timestamp of receipt is what I can think of offhand. There may be other stuff in there that is parsed out of the message and copied to msgdata for convenience but I don't know of any offhand. I'd have to read source to confirm.
Due to the holidays I didn't get around to testing/poking this so...
You're welcome to ask even if you haven't. Of course we prefer that you investigate as much as possible, but we're all grownups. Everybody has their limits, and we cover for each other as we can.
Steve
Well, I discovered some chaos, and indeed this is a Microsoft MAPI added component. Thread-Topic and Thread-Index are Microsoft's crap for their handling of threading, and they add it to *any* origin message going out the door for Microsoft's conversation handling. Which is annoying.
I'm going to be exploring some things on my clients' end to see if we can strip those headers *before* they end up at our mailing list system. Painful systems are painful, and Microsoft is one of the worst offenders in this case.
And I've informed Microsoft through our deep diving on their end and mine that they're breaking things. FORTUNATELY, I have administrative access on the MS365 side for the clients involved, so I can do some tweaking on mail ingress to strip certain headers. And hopefully that'll work. Going to be doing more testing on that, but I might need to set up some rules to strip messages of those headers programmatically.
Thomas
-----Original Message----- From: Stephen J. Turnbull <turnbull.stephen.fw@u.tsukuba.ac.jp> Sent: Wednesday, January 3, 2024 3:06 AM To: Thomas Ward <teward@thomas-ward.net> Cc: mailman-users@mailman3.org Subject: [MM3-users] Re: Programmatically modify Thread-Topic *before* message is sent out the door?
Thomas Ward via Mailman-users writes:
Any chance Thread-Topic or such is spawned by the msgdata object > instead of directly in the message?
No, the string "thread-topic" does not appear *anywhere* in the Mailman core sources. Mailman itself has no concept of threads. HyperKitty uses References, Reply-To, and Subject to algorithmically identify threads. Mailman does have an independent concept called "topic", which can best be thought of as "sub-list". But it uses the X-Topics header, and as far as I know doesn't pay attention to threading.
AFAIK, Thread-Topic is a Microsoft-specific device. If you use Exchange or Outlook, you have to negotiate with that software about the use of Thread-Topic. My guess is that it is generated by Exchange because somebody at Microsoft decided that threading should be enforced by the server, not inferred by the MUA.
Unfortunately there's very little documentation on msgdata in the > handlers and what it is/contains so I'm asking again.
I guess the relevant documentation is RFC 5321. ;-) msgdata's original purpose is to contain envelope and delivery information.
That is, any data about the message that the LMTP runner knows that is not in the message header itself. HELO hostname, sender, recipients, IP address of SMTP client, timestamp of receipt is what I can think of offhand. There may be other stuff in there that is parsed out of the message and copied to msgdata for convenience but I don't know of any offhand. I'd have to read source to confirm.
Due to the holidays I didn't get around to testing/poking this so...
You're welcome to ask even if you haven't. Of course we prefer that you investigate as much as possible, but we're all grownups. Everybody has their limits, and we cover for each other as we can.
Steve
participants (2)
-
Stephen J. Turnbull -
Thomas Ward