How to strip DKIM headers on a domain level or a list level
Rob Jenson writes:
I need to strip DKIM headers on the inbound messages to my list or to my domain.
Why do you need to strip DKIM headers? The standard specifies that an invalid DKIM signature must be treated the same as if there were no signature at all. So an invalid DKIM signature may be treated as spammier than a message with a valid signature, but that should not be treated differently from an unsigned message (as would appear after removing those header fields).
I know this feature is in demand from list admins, but I've never seen convincing evidence that it's actually effective for any of the usual suspects (Yahoo!, AOL, Gmail), only for small domains with pigheaded admins who are proud of their non-conforming configurations, and announce that they are discarding messages which have invalid DKIM signatures.
What I do not understand from the documentation for Mailman 3 Core is whether it is possible for my hosting service to set remove_dkim_headers just for our domain.
It currently is not possible. It probably could be made possible on a per-domain or per-list basis, but it's not clear to me why it would be useful to do so. If there really are non-conforming receivers out there, it's probably a good idea to strip host-wide.
If that is not possible, then the question becomes "is there a way, in the list configuration, to strip the DKIM headers?
It is possible, but that requires changing the list's processing pipeline to add that capability. I believe this is not difficult (I haven't done similar changes with Mailman 3 yet) but it does require the assistance of the host, and possibly imposes a future maintenance burden on the host.
We are munging the text of the message in Mailman, so the DKIM headers from the original poster are invalid.
As I explained above, both DKIM and DMARC are specified so that "no signature" == "invalid signature" for the purposes of spam processing. Are you sure that Yahoo! (or any other provider used by your subscribers) is treating invalid signatures differently from absence of signatures? The person responsible for Yahoo! MTA configuration is a well-known mail security expert who participated actively in the specification of all these protocols (and she also gave me a kitten, so I may be biased).
As far as I can see, our service provider is using ARC in our Mailman configuration, but not signing the outbound messages with DKIM.
That surprises me. ARC isn't really a substitute for the MTA's own DKIM signature, at least not yet.
Therefore the DKIM signature from the poster's mail service provider is sent out with their DKIM header, which seems to be problematic.
It shouldn't be.
If I understand ARC correctly, it is validating the DKIM signature from the poster, creating a new signature and metadata indicating that what it received was properly signed.
That is correct. As far as I know, Yahoo! does participate in the ARC protocol and used to have conforming implementations of DKIM and DMARC. That doesn't mean you get a free ride: they may still have your IP on a blacklist from former owners of the IP, for example. Or your posts may "look like" spam for some other reason, or your lists may need more time to build up a clean reputation. But I need evidence more convincing than "list posts are recognized as spam" to believe that removing DKIM headers will help. (For example, my employer's filters regularly recognize messages from department heads as spam, even with DKIM signatures intact. ;-)
On 11/21/20 10:39 AM, Stephen J. Turnbull wrote:
Rob Jenson writes:
I need to strip DKIM headers on the inbound messages to my list or to my domain.
Why do you need to strip DKIM headers? The standard specifies that an invalid DKIM signature must be treated the same as if there were no signature at all. So an invalid DKIM signature may be treated as spammier than a message with a valid signature, but that should not be treated differently from an unsigned message (as would appear after removing those header fields).
I know this feature is in demand from list admins, but I've never seen convincing evidence that it's actually effective for any of the usual suspects (Yahoo!, AOL, Gmail), only for small domains with pigheaded admins who are proud of their non-conforming configurations, and announce that they are discarding messages which have invalid DKIM signatures.
This person is a client of ours for now. There is no evidence at all that I can see from our outgoing SMTP logs for such a requirement. In fact mail delivery to Yahoo and its other domains (verizon, aol) has been great. There are some small deferrals that occur over a small period of time (which is strange) daily but those clear out of the queue within an hour. I was personally shocked when he brought these concerns to this list without asking or informing us.
We are munging the text of the message in Mailman, so the DKIM headers from the original poster are invalid.
As I explained above, both DKIM and DMARC are specified so that "no signature" == "invalid signature" for the purposes of spam processing. Are you sure that Yahoo! (or any other provider used by your subscribers) is treating invalid signatures differently from absence of signatures? The person responsible for Yahoo! MTA configuration is a well-known mail security expert who participated actively in the specification of all these protocols (and she also gave me a kitten, so I may be biased).
I have the same question Steve. I see no proof for such a practice nor came across any documentation regarding such. The number one reason why Yahoo defers mail is mail volume from what I see. I don't even think they are hard on SPF violations as other ISPs such as Google.
As far as I can see, our service provider is using ARC in our Mailman configuration, but not signing the outbound messages with DKIM.
That surprises me. ARC isn't really a substitute for the MTA's own DKIM signature, at least not yet.
It ought to surprise you because we are not using ARC. His comment surprised me as well. Again I have no idea where he gets that from. He certainly did not bring that to my attention nor presented proof.
Therefore the DKIM signature from the poster's mail service provider is sent out with their DKIM header, which seems to be problematic.
It shouldn't be.
I agree. It shouldn't be and it's not. Otherwise this would impact all lists that are DMARC munging their Mailman 3 lists unconditionally. It's not.
If I understand ARC correctly, it is validating the DKIM signature from the poster, creating a new signature and metadata indicating that what it received was properly signed.
That is correct. As far as I know, Yahoo! does participate in the ARC protocol and used to have conforming implementations of DKIM and DMARC. That doesn't mean you get a free ride: they may still have your IP on a blacklist from former owners of the IP, for example. Or your posts may "look like" spam for some other reason, or your lists may need more time to build up a clean reputation. But I need evidence more convincing than "list posts are recognized as spam" to believe that removing DKIM headers will help. (For example, my employer's filters regularly recognize messages from department heads as spam, even with DKIM signatures intact. ;-)
This particular client did bring up some Yahoo issues a week ago which I looked into it. The conclusion was Yahoo is deferring a small percentage of outbound mail for a small period of time due to mail volume. Overall we have great successful deliveries to Yahoo addresses on our Affinity server. So I am simply surprised at his communication sent to this list. Yahoo delivery always improves when mail volume becomes more consistent. That week we have moved over a moderate amount of lists that had a few dozen Yahoo members. That caused our SenderScore.org rep score to drop from 98 to 96 (which is still a very high reputation). It's back up to 98 (pats myself on the back) because the new outgoing mail volume has become more consistent. Eventually we will fill up the server and will bring up a new server to add new lists to. This approach has worked well for all of our servers for years. So conclusion: broken DKIM signatures are not playing a part with his issues at all. At least from the evidence I have seen.
-- Brian Carpenter Harmonylists.com Emwd.com
On Nov 21, 2020, at 8:51 AM, Brian Carpenter <brian_carpenter@emwd.com> wrote:
On 11/21/20 10:39 AM, Stephen J. Turnbull wrote:
This particular client did bring up some Yahoo issues a week ago which I looked into it. The conclusion was Yahoo is deferring a small percentage of outbound mail for a small period of time due to mail volume. Overall we have great successful deliveries to Yahoo addresses on our Affinity server. So I am simply surprised at his communication sent to this list. Yahoo delivery always improves when mail volume becomes more consistent. That week we have moved over a moderate amount of lists that had a few dozen Yahoo members. That caused our SenderScore.org <http://senderscore.org/> rep score to drop from 98 to 96 (which is still a very high reputation). It's back up to 98 (pats myself on the back) because the new outgoing mail volume has become more consistent. Eventually we will fill up the server and will bring up a new server to add new lists to. This approach has worked well for all of our servers for years. So conclusion: broken DKIM signatures are not playing a part with his issues at all. At least from the evidence I have seen.
Also a client of harmony lists and assume to have have same server settings. I hear some Yahoo users have 100% email in spam. But no bounce related to yahoo. Maybe 10% of Yahoo users have problems. After marking them as not spam all users have reported that it is fixed and Yahoo doesn’t mark emails as spam. Yahoo spam detection obviously is a very personalized thing. Gmail overall is much better where < 1% of users see emails in spam. A good resource is here to understand what can be done. https://www.whitelist.guide/yahoo/
Unfortunately it’s not possible to debug such problems without having access to each subscribers email account. Subscribers without technical knowledge are very vague in their reports and do not always understand instructions how to fix it. My suspicion is that some users like to use the “report spam” buttons way too much and then end up with a extremely sensitive individual spam scoring.
-- Brian Carpenter Harmonylists.com <http://harmonylists.com/> Emwd.com <http://emwd.com/>
Mailman-users mailing list -- mailman-users@mailman3.org <mailto:mailman-users@mailman3.org> To unsubscribe send an email to mailman-users-leave@mailman3.org <mailto:mailman-users-leave@mailman3.org> https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/ <https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/>
Apollinaris Schöll writes:
My suspicion is that some users like to use the “report spam” buttons way too much and then end up with a extremely sensitive individual spam scoring.
Interesting hypothesis, though I see no easy way to confirm it except surveying the users, which probably isn't something one wants to do.
I will keep in mind your advice that marking as "not spam" helps (at least on Yahoo!). That's something that's actionable for both subscribers and list admins.
Steve
On Nov 23, 2020, at 7:05 PM, Stephen J. Turnbull <turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:
Apollinaris Schöll writes:
My suspicion is that some users like to use the “report spam” buttons way too much and then end up with a extremely sensitive individual spam scoring.
Interesting hypothesis, though I see no easy way to confirm it except surveying the users, which probably isn't something one wants to do.
Anecdotally, I believe this to be true as well. I also believe that it’s common for unsophisticated users to just start marking list emails as spam stop seeing them instead of figuring out how to unsubscribe. Which really sucks for everyone else.
- Mark
mark@pdc-racing.net | 408-348-2878
On 11/23/20 7:22 PM, Mark Dadgar wrote:
I also believe that it’s common for unsophisticated users to just start marking list emails as spam stop seeing them instead of figuring out how to unsubscribe. Which really sucks for everyone else.
Several servers I'm involved with managing are subscribed to Yahoo's feedback loop (go to https://help.yahoo.com/kb/postmaster and select "Complaint Feedback Loop"). Reporting list mail as spam is one way to get quickly removed from a list. However, this is a bit complicated because the "report spam" button in Yahoo's web mail UI is right next to the "delete" button, so some reports are accidents.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 11/23/2020 10:48 PM, Mark Sapiro wrote:
On 11/23/20 7:22 PM, Mark Dadgar wrote:
I also believe that it’s common for unsophisticated users to just start marking list emails as spam stop seeing them instead of figuring out how to unsubscribe. Which really sucks for everyone else.
Several servers I'm involved with managing are subscribed to Yahoo's feedback loop (go to https://help.yahoo.com/kb/postmaster and select "Complaint Feedback Loop"). Reporting list mail as spam is one way to get quickly removed from a list. However, this is a bit complicated because the "report spam" button in Yahoo's web mail UI is right next to the "delete" button, so some reports are accidents.
I've done the same thing, but Yahoo is pretty good at sanitizing their report to make it difficult to impossible to tell who actually needs to get ejected. I added "Sent to %(user_name)s at %(user_address)s" to the footer and while they'd generally get the user_address wiped out, the user_name would generally survive.
And because Yahoo's UI is flawed as Mark pointed out, with the delete and spam buttons adjacent, I'd usually give people a couple chances before I'd airlock them.
-- Joel Lord
On 11/23/20 8:00 PM, Joel Lord wrote:
I've done the same thing, but Yahoo is pretty good at sanitizing their report to make it difficult to impossible to tell who actually needs to get ejected. I added "Sent to %(user_name)s at %(user_address)s" to the footer and while they'd generally get the user_address wiped out, the user_name would generally survive.
For this reason, Mailman 2.1 has this feature.
# If the following is set to a non-empty string, that string is the name of a # header that will be added to personalized and VERPed deliveries with value # equal to the base64 encoding of the recipient's email address. This is # intended to enable identification of the recipient otherwise redacted from # "spam report" feedback loop messages. For example, if # RCPT_BASE64_HEADER_NAME = 'X-Mailman-R-Data' # a header like # X-Mailman-R-Data: dXNlckBleGFtcGxlLmNvbQo= # will be added to messages sent to user@@example.com. RCPT_BASE64_HEADER_NAME = ''
This hasn't been implemented for Mailman 3 but could be if there's demand. -- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 11/23/20 11:15 PM, Mark Sapiro wrote:
# If the following is set to a non-empty string, that string is the name of a # header that will be added to personalized and VERPed deliveries with value # equal to the base64 encoding of the recipient's email address. This is # intended to enable identification of the recipient otherwise redacted from # "spam report" feedback loop messages. For example, if # RCPT_BASE64_HEADER_NAME = 'X-Mailman-R-Data' # a header like # X-Mailman-R-Data: dXNlckBleGFtcGxlLmNvbQo= # will be added to messages sent to user@@example.com. RCPT_BASE64_HEADER_NAME = ''
This hasn't been implemented for Mailman 3 but could be if there's demand.
I hadn't spotted that in Mailman 2.1, I will go looking for it there. My 2.1 server has not moved to 3 yet for several reasons, but it's getting close. This feature would get me just a bit closer. -- Joel Lord
On Nov 23, 2020, at 8:00 PM, Joel Lord <jpl@ilk.org> wrote:
On 11/23/2020 10:48 PM, Mark Sapiro wrote:
On 11/23/20 7:22 PM, Mark Dadgar wrote:
I also believe that it’s common for unsophisticated users to just start marking list emails as spam stop seeing them instead of figuring out how to unsubscribe. Which really sucks for everyone else.
Several servers I'm involved with managing are subscribed to Yahoo's feedback loop (go to https://help.yahoo.com/kb/postmaster and select "Complaint Feedback Loop"). Reporting list mail as spam is one way to get quickly removed from a list. However, this is a bit complicated because the "report spam" button in Yahoo's web mail UI is right next to the "delete" button, so some reports are accidents.
I've done the same thing, but Yahoo is pretty good at sanitizing their report to make it difficult to impossible to tell who actually needs to get ejected. I added "Sent to %(user_name)s at %(user_address)s" to the footer and while they'd generally get the user_address wiped out, the user_name would generally survive.
And because Yahoo's UI is flawed as Mark pointed out, with the delete and spam buttons adjacent, I'd usually give people a couple chances before I'd airlock them.
Yeah, I’ve been subscribed to the Yahoo feedback loops for ever and ever.
If you’ve got verp turned on, you can decode the original address from that in the headers.
- Mark
mark@pdc-racing.net | 408-348-2878
participants (6)
-
Apollinaris Schöll -
Brian Carpenter -
Joel Lord -
Mark Dadgar -
Mark Sapiro -
Stephen J. Turnbull