On 8/13/19 5:04 AM, Marvin Gülker wrote:

Do any large e-mail providers actually support ARC? Smaller ones like Fastmail appearently do at least semi-manually[1], but what about the relevant ones -- GMail, Microsoft, Yahoo (and in Germany, GMX et al.)?

Gmail does.

The question is not whether they support ARC, but whether they will trust you.

-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

Am 13. August 2019 um 05:42 Uhr -0700 schrieb Mark Sapiro:

The question is not whether they support ARC, but whether they will trust you.

Well. That's nothing new, given that GMail and Microsoft liberally reject e-mail from small providers for no reason anyway (I have experienced it myself). If this situation doesn't change with ARC, I can live with this (by paying for an SMTP relay provider that proxies my e-mail).

However, I thought the entire point in ARC was fixing automatic rejections caused by DMARC invalidation by mailing lists, which are vital for so many open-source projects (including Mailman iself). If ARC falls short of solving that problem, because the large providers do not automatically trust ARC, then supporting ARC on my mailing list server is kind of void. Why should almighty Microsoft trust my small personal mailing list server?

ARC can only achieve its goal with black-listing untrusted relays rather than white-listing known good ones, because it is simply unfeasable to manually maintain a list of all open-source project mailing lists in the entire world. I could set a new one up tomorrow, how should you know?

-- Blog: https://mg.guelker.eu

Marvin Gülker writes:

However, I thought the entire point in ARC was fixing automatic rejections caused by DMARC invalidation by mailing lists, which are vital for so many open-source projects (including Mailman iself).

Yes. s/fixing/making it possible to fix/ and s/mailing lists/indirect mail/ (the REALLY big losers in April 2014 were small businesses which had a large percentage of their invoices become undeliverable).

That doesn't help you one bit with your "rejection with no reason" problem, though. It helps you with one specific reason (DMARC p=reject).

If ARC falls short of solving that problem, because the large providers do not automatically trust ARC, then supporting ARC on my mailing list server is kind of void. Why should almighty Microsoft trust my small personal mailing list server?

The answer to that is "why shouldn't they?"

I've never had a problem with any of the big services. There are reasons why some hosts might have problems due to no misbehavior of their own, and they wouldn't necessarily know about them either. But mostly the mail does go through.

If the mail itself gets through, I see no reason why they wouldn't trust your ARC fields, so you won't get DMARC rejections, as long as nobody downstream of you breaks your ARC signatures.

If you're seeing something different (ARC-authenticated messages getting rejected for DMARC alignment failure), let us know and I'll talk to a guy who knows this gal. My experience is good, the people I talk to are happy. If your experience is different, the ARC developers should know about it, otherwise we can't do anything about it.

ARC can only achieve its goal with black-listing untrusted relays rather than white-listing known good ones,

The goal of ARC is to keep users at p=reject sites happy. A lot of the people involved both at the IETF and at the DMARC Consortium are old-timers who are very sympathetic to mailing lists. But the day-to- day tuning of algorithms is targeted at users with mailboxes, not the lists that fill those mailboxes. Sad to say, users care more about spam (that's their provider's fault) than about mailing lists (which they assess as the mailing list's fault). Last I heard the Google folks and the IETF are very happy with ARC.

because it is simply unfeasable to manually maintain a list of all open-source project mailing lists in the entire world. I could set a new one up tomorrow, how should you know?

That's not how this works. All of these services use huge databases of spam, ham, hosts, and mailing lists, and apply machine-learning algorithms to that data to create classifiers. It may take them some time to tune them for ARC, but the principle is there's no black or white here, it's all grey.

Steve

Am 15. August 2019 um 02:08 Uhr +0900 schrieb Stephen J. Turnbull:

That doesn't help you one bit with your "rejection with no reason" problem, though. It helps you with one specific reason (DMARC p=reject).

I don't have that problem anymore since I pay for my SMTP relay provider. For me, the issue is solved. Before my personal mail (I'm not talking about mailing lists, just simple direct e-mail) was pretty much always sorted into the recipient's spam folder or directly rejected at the SMTP level (this was the case with both Microsoft and AOL; GMail usually filed my e-mail als spam). I looked up the reasons they gave in the SMTP reject; it was always bad reputation of IPs. Tools like mail-tester on the other hand say my e-mail and IP was fine, and delivery to small providers always went nicely. My mail server has never sent a single spam e-mail. I run it since years, and I am very sure that this never has happened. This mail server only hosts my and my family's e-mail.

I have no AOL contacts anymore, so I left that is it was. For Microsoft (i.e. outlook.com), I went through their IP unblocking formula (which at the time contained a CAPTCHA that I'm sure a computer could solve better than me simple human) multiple times, and I received "conditionally mitigated" as a response each time. After a few months, the rejections would start again.

Thus, I say they were rejecting my e-mail for no reason. Might be a hyperbole, but from my point of view, there was no reason. Again, the problem is now past with my workaround, so there is no need to investigate any further. I've given up on it. I don't believe you can have a VPS that directly delivers e-mail to the large providers successfully. The workaround with the SMTP relay provider still allows me to self-host, which fulfills my needs.

The answer to that is "why shouldn't they?"

I hope it goes like that.

I've never had a problem with any of the big services. There are reasons why some hosts might have problems due to no misbehavior of their own, and they wouldn't necessarily know about them either. But mostly the mail does go through.

Lucky you. My experience was the complete contrary, and it's not just me. Take a look: https://lobste.rs/s/lvajly/google_is_eating_our_mail

If the mail itself gets through, I see no reason why they wouldn't trust your ARC fields, so you won't get DMARC rejections, as long as nobody downstream of you breaks your ARC signatures.

So, my situation currently is this. I have no (public) mailing lists on my own mail server, so there is no need for DMARC or ARC. Then, I'm involved in an open-source project that does have a mailing list and I'm considering to add ARC support for the list (one of the reasons why I constantly read this very mailing list). The project is small, so we'd like to avoid any effort that is fruitless anyway. Finally, I moderate the German ruby-de mailing list without any kind of influence on the software stack running it. That mailing list server runs Mailman 2.1.15 and this is not something I have any influence on.

All of these list servers currently do not do ARC, hence I've not seen any rejections of ARC-signed messages (as there are none). I'm only evaluating how likely it'd be to see them. Please don't get me wrong; I appreciate all the effort that went into ARC! It looks like a good solution to the DMARC problems.

If you're seeing something different (ARC-authenticated messages getting rejected for DMARC alignment failure), let us know and I'll talk to a guy who knows this gal.

This is a nice move. Thank you for this offer. When that happens, I'm going to post here!

But the day-to-day tuning of algorithms is targeted at users with mailboxes, not the lists that fill those mailboxes.

I am aware of that. See, when DMARC was spread over the world, ARC was not available. Even the Mailman wiki itself said and still says that DMARC has "negative consequences for mailing lists" and is "breaking long established mailing list norms, standards, and behaviors."[1] The problem still exists. On the ruby-de mailing list I moderate, there was a problem with a person sending e-mail from a p=reject domain to the list[2] (which is, remember, run by Mailman 2.1.15). Two subscribers were force-unsubscribed for the DMARC bounces. I had to manually re-subscribe them and had to explain the p=reject problem to the person sending the e-mails. This was... frustrating. And I think my statements on DMARC show my frustration.

Last I heard the Google folks and the IETF are very happy with ARC.

I appreciate it. I hope it solves these problems. It's just that I'm reluctant to believe there's a quick solution available. It sounds to good to be true.

...

because it is simply unfeasable to manually maintain a list of all open-source project mailing lists in the entire world. I could set a new one up tomorrow, how should you know?

That's not how this works.

I'm afraid it does. Fastmail does it exactly like that: manual white list, see https://www.fastmail.com/help/technical/senderauthentication.html. Quote of the relevant passage:

Until the list of ARC enabled forwarders grows we maintain lists of known good email forwarders and mailing lists, which we use to whitelist known-good mail forwarders, and apply this list when evaluating DMARC policies for inbound mail.

To do them justice, they say a little further down that a failed DMARC check at a p=reject policy domain does not cause an actual reject, but only increases the spam score. What worries me is that other providers might go that whitelisting approach, but don't follow the second part and still obey p=reject.

If everything goes well, nobody is going to do that and ARC is successful. I'm all in for that result, but I lack the knowledge to judge what is likely and what not. Which is why I'm asking here. From your answer, I see that you think such behaviour is unlikely. That's good news, and it answers my question. Thanks!

[1] https://wiki.list.org/DEV/DMARC [2] See July archive: https://lists.ruby-lang.org/pipermail/ruby-de/2019-July/thread.html

-- Blog: https://mg.guelker.eu

Am 14. August 2019 um 20:44 Uhr -0000 schrieb brian@emwd.com:

Did you keep an eye on the reputation on the IP addresses you were sending from? Sometimes an IP address just doesn't send enough volume of mail to build one up.

This mail server ever only sent my personal and my family's e-mail -- no public mailing lists. The volume was very low. Maybe about ten e-mails a week in average, sometimes no e-mail a week at all. If low e-mail volume is indeed a criterion, then it could be the cause. But low volume as a spam criterion strikes me as doubtful, because a spammer specifically wants to issue mass e-mail.

The server was online since I think 2013 and always had the same IP. It wasn't a shared IP, i.e. this mail server was the only one sending from that IP. The spam classification problem was there from the very beginning.

We did get that fixed. The key to Microsoft is to sign up for their Hotmail Feedback Loop. They also provide a nice interface (SNDS) that shows you what they think of your IP addresses so you can mitigate any potential delivery issues ahead of time.

I'm leaving that to my SMTP relay provider now. I find it too much hassle to sign up for such services just for my private e-mail. I'm not getting paid for sinking my time into this; keeping the server running and clean is enough of time investment into my hobby (I use it for other things than e-mail, too). Microsoft is not the only e-mail provider, so if all e-mail providers did it like that, you'd have to create dozens and hundreds of accounts. Microsoft's pure size is what allows them to pose such a restriction successfully. The real problem is the centralisation of services at only a few providers.

Yahoo is the worst.

Luckily, Yahoo e-mail addresses are so rare in Germany that I can ignore them. I don't have anyone with a Yahoo address in my address book currently. outlook.com is much more common, but from my experience most German e-mail addresses are hosted by the United Internet group (GMX et al.), which I never had problems delivering to. They do a decent job. As for GMail, after a few attempts with followup questions my contacts started looking into their spam folder. That's why I was fine without the SMTP relay provider for years. But since recently, more and more people use outlook.com, which outright rejected.

The majority reason why messages ends up in Spam folders has nothing to do with the mail server at all. It is because the content of the message comes across as spammy, i.e. a message that contains several URLs, using exclamation points, etc.

Normal private 1-1 correspondence usually does not fulfill these criteria, and I've still been filed/rejected as spam (and likewise other family members). I felt like Don Quixote fighting the windmills. Today's e-mail infrastructure makes things difficult for private individuals who want to self-host.

Anyway, I think this subthread has grown large enough. My questions were answered. Thanks to everybody who commented!

-- Blog: https://mg.guelker.eu