Hi all,
Is the documentation correct that ARC should be as simple to configure in mailman 3 as to add the private DKIM key and restarting mailman? (https://mailman.readthedocs.io/en/latest/src/mailman/handlers/docs/arc_sign....)
I've tried that without finding any effect in my so far limited tests. I have not found anything in logs although I might have missed looking in one. Are there any known limitations?
I have previously a working DKIM signature on outgoing mail as well as correctly set up SPF in DNS and SPF-checking on received mail. The verification on incoming mail adds a few headers but I do not think that that should be an issue, neither would the DKIM signing change anything should mailman add new headers.
I now run Mailman 3.2.3 (La Villa Strangiato) on Python 3.7.4.
Cheers // David
Am 12. August 2019 um 23:49 Uhr +0200 schrieb David Krantz:
Is the documentation correct that ARC should be as simple to configure in mailman 3 as to add the private DKIM key and restarting mailman?
Do any large e-mail providers actually support ARC? Smaller ones like Fastmail appearently do at least semi-manually1, but what about the relevant ones -- GMail, Microsoft, Yahoo (and in Germany, GMX et al.)?
I infer from the spare results of searching for "ARC suport" that even with a perfectly set-up ARC infrastructure, the large e-mail providers are still going to reject DMARC-protected mailing list e-mail, because they simply don't support ARC.
-- Blog: https://mg.guelker.eu
On 8/13/19 5:04 AM, Marvin Gülker wrote:
Do any large e-mail providers actually support ARC? Smaller ones like Fastmail appearently do at least semi-manually[1], but what about the relevant ones -- GMail, Microsoft, Yahoo (and in Germany, GMX et al.)?
Gmail does.
The question is not whether they support ARC, but whether they will trust you.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Am 13. August 2019 um 05:42 Uhr -0700 schrieb Mark Sapiro:
The question is not whether they support ARC, but whether they will trust you.
Well. That's nothing new, given that GMail and Microsoft liberally reject e-mail from small providers for no reason anyway (I have experienced it myself). If this situation doesn't change with ARC, I can live with this (by paying for an SMTP relay provider that proxies my e-mail).
However, I thought the entire point in ARC was fixing automatic rejections caused by DMARC invalidation by mailing lists, which are vital for so many open-source projects (including Mailman iself). If ARC falls short of solving that problem, because the large providers do not automatically trust ARC, then supporting ARC on my mailing list server is kind of void. Why should almighty Microsoft trust my small personal mailing list server?
ARC can only achieve its goal with black-listing untrusted relays rather than white-listing known good ones, because it is simply unfeasable to manually maintain a list of all open-source project mailing lists in the entire world. I could set a new one up tomorrow, how should you know?
-- Blog: https://mg.guelker.eu
Marvin Gülker writes:
However, I thought the entire point in ARC was fixing automatic rejections caused by DMARC invalidation by mailing lists, which are vital for so many open-source projects (including Mailman iself).
Yes. s/fixing/making it possible to fix/ and s/mailing lists/indirect mail/ (the REALLY big losers in April 2014 were small businesses which had a large percentage of their invoices become undeliverable).
That doesn't help you one bit with your "rejection with no reason" problem, though. It helps you with one specific reason (DMARC p=reject).
If ARC falls short of solving that problem, because the large providers do not automatically trust ARC, then supporting ARC on my mailing list server is kind of void. Why should almighty Microsoft trust my small personal mailing list server?
The answer to that is "why shouldn't they?"
I've never had a problem with any of the big services. There are reasons why some hosts might have problems due to no misbehavior of their own, and they wouldn't necessarily know about them either. But mostly the mail does go through.
If the mail itself gets through, I see no reason why they wouldn't trust your ARC fields, so you won't get DMARC rejections, as long as nobody downstream of you breaks your ARC signatures.
If you're seeing something different (ARC-authenticated messages getting rejected for DMARC alignment failure), let us know and I'll talk to a guy who knows this gal. My experience is good, the people I talk to are happy. If your experience is different, the ARC developers should know about it, otherwise we can't do anything about it.
ARC can only achieve its goal with black-listing untrusted relays rather than white-listing known good ones,
The goal of ARC is to keep users at p=reject sites happy. A lot of the people involved both at the IETF and at the DMARC Consortium are old-timers who are very sympathetic to mailing lists. But the day-to- day tuning of algorithms is targeted at users with mailboxes, not the lists that fill those mailboxes. Sad to say, users care more about spam (that's their provider's fault) than about mailing lists (which they assess as the mailing list's fault). Last I heard the Google folks and the IETF are very happy with ARC.
because it is simply unfeasable to manually maintain a list of all open-source project mailing lists in the entire world. I could set a new one up tomorrow, how should you know?
That's not how this works. All of these services use huge databases of spam, ham, hosts, and mailing lists, and apply machine-learning algorithms to that data to create classifiers. It may take them some time to tune them for ARC, but the principle is there's no black or white here, it's all grey.
Steve
Am 15. August 2019 um 02:08 Uhr +0900 schrieb Stephen J. Turnbull:
That doesn't help you one bit with your "rejection with no reason" problem, though. It helps you with one specific reason (DMARC p=reject).
I don't have that problem anymore since I pay for my SMTP relay provider. For me, the issue is solved. Before my personal mail (I'm not talking about mailing lists, just simple direct e-mail) was pretty much always sorted into the recipient's spam folder or directly rejected at the SMTP level (this was the case with both Microsoft and AOL; GMail usually filed my e-mail als spam). I looked up the reasons they gave in the SMTP reject; it was always bad reputation of IPs. Tools like mail-tester on the other hand say my e-mail and IP was fine, and delivery to small providers always went nicely. My mail server has never sent a single spam e-mail. I run it since years, and I am very sure that this never has happened. This mail server only hosts my and my family's e-mail.
I have no AOL contacts anymore, so I left that is it was. For Microsoft (i.e. outlook.com), I went through their IP unblocking formula (which at the time contained a CAPTCHA that I'm sure a computer could solve better than me simple human) multiple times, and I received "conditionally mitigated" as a response each time. After a few months, the rejections would start again.
Thus, I say they were rejecting my e-mail for no reason. Might be a hyperbole, but from my point of view, there was no reason. Again, the problem is now past with my workaround, so there is no need to investigate any further. I've given up on it. I don't believe you can have a VPS that directly delivers e-mail to the large providers successfully. The workaround with the SMTP relay provider still allows me to self-host, which fulfills my needs.
The answer to that is "why shouldn't they?"
I hope it goes like that.
I've never had a problem with any of the big services. There are reasons why some hosts might have problems due to no misbehavior of their own, and they wouldn't necessarily know about them either. But mostly the mail does go through.
Lucky you. My experience was the complete contrary, and it's not just me. Take a look: <https://lobste.rs/s/lvajly/google_is_eating_our_mail>
If the mail itself gets through, I see no reason why they wouldn't trust your ARC fields, so you won't get DMARC rejections, as long as nobody downstream of you breaks your ARC signatures.
So, my situation currently is this. I have no (public) mailing lists on my own mail server, so there is no need for DMARC or ARC. Then, I'm involved in an open-source project that does have a mailing list and I'm considering to add ARC support for the list (one of the reasons why I constantly read this very mailing list). The project is small, so we'd like to avoid any effort that is fruitless anyway. Finally, I moderate the German ruby-de mailing list without any kind of influence on the software stack running it. That mailing list server runs Mailman 2.1.15 and this is not something I have any influence on.
All of these list servers currently do not do ARC, hence I've not seen any rejections of ARC-signed messages (as there are none). I'm only evaluating how likely it'd be to see them. Please don't get me wrong; I appreciate all the effort that went into ARC! It looks like a good solution to the DMARC problems.
If you're seeing something different (ARC-authenticated messages getting rejected for DMARC alignment failure), let us know and I'll talk to a guy who knows this gal.
This is a nice move. Thank you for this offer. When that happens, I'm going to post here!
But the day-to-day tuning of algorithms is targeted at users with mailboxes, not the lists that fill those mailboxes.
I am aware of that. See, when DMARC was spread over the world, ARC was not available. Even the Mailman wiki itself said and still says that DMARC has "negative consequences for mailing lists" and is "breaking long established mailing list norms, standards, and behaviors."[1] The problem still exists. On the ruby-de mailing list I moderate, there was a problem with a person sending e-mail from a p=reject domain to the list[2] (which is, remember, run by Mailman 2.1.15). Two subscribers were force-unsubscribed for the DMARC bounces. I had to manually re-subscribe them and had to explain the p=reject problem to the person sending the e-mails. This was... frustrating. And I think my statements on DMARC show my frustration.
Last I heard the Google folks and the IETF are very happy with ARC.
I appreciate it. I hope it solves these problems. It's just that I'm reluctant to believe there's a quick solution available. It sounds to good to be true.
because it is simply unfeasable to manually maintain a list of all open-source project mailing lists in the entire world. I could set a new one up tomorrow, how should you know?
That's not how this works.
I'm afraid it does. Fastmail does it exactly like that: manual white list, see <https://www.fastmail.com/help/technical/senderauthentication.html>. Quote of the relevant passage:
Until the list of ARC enabled forwarders grows we maintain lists of known good email forwarders and mailing lists, which we use to whitelist known-good mail forwarders, and apply this list when evaluating DMARC policies for inbound mail.
To do them justice, they say a little further down that a failed DMARC check at a p=reject policy domain does not cause an actual reject, but only increases the spam score. What worries me is that other providers might go that whitelisting approach, but don't follow the second part and still obey p=reject.
If everything goes well, nobody is going to do that and ARC is successful. I'm all in for that result, but I lack the knowledge to judge what is likely and what not. Which is why I'm asking here. From your answer, I see that you think such behaviour is unlikely. That's good news, and it answers my question. Thanks!
[1] https://wiki.list.org/DEV/DMARC [2] See July archive: https://lists.ruby-lang.org/pipermail/ruby-de/2019-July/thread.html
-- Blog: https://mg.guelker.eu
Marvin Gülker wrote:
I don't have that problem anymore since I pay for my SMTP relay provider. For me, the issue is solved. Before my personal mail (I'm not talking about mailing lists, just simple direct e-mail) was pretty much always sorted into the recipient's spam folder or directly rejected at the SMTP level (this was the case with both Microsoft and AOL; GMail usually filed my e-mail als spam). I looked up the reasons they gave in the SMTP reject; it was always bad reputation of IPs. Tools like mail-tester on the other hand say my e-mail and IP was fine, and delivery to small providers always went nicely. My mail server has never sent a single spam e-mail. I run it since years, and I am very sure that this never has happened. This mail server only hosts my and my family's e-mail.
Did you keep an eye on the reputation on the IP addresses you were sending from? Sometimes an IP address just doesn't send enough volume of mail to build one up. I do agree some providers do ignore the IP reputation of an IP address. For instance we had one that actually had a 100% reputation from SenderScore and it ran into a block with Microsoft. However the server had only been online for a few months and we placed a client there that increased the volume to Microsoft users. We did get that fixed. The key to Microsoft is to sign up for their Hotmail Feedback Loop. They also provide a nice interface (SNDS) that shows you what they think of your IP addresses so you can mitigate any potential delivery issues ahead of time.
Thus, I say they were rejecting my e-mail for no reason. Might be a hyperbole, but from my point of view, there was no reason. Again, the problem is now past with my workaround, so there is no need to investigate any further. I've given up on it. I don't believe you can have a VPS that directly delivers e-mail to the large providers successfully. The workaround with the SMTP relay provider still allows me to self-host, which fulfills my needs.
I disagree. We are a list hosting provider (Mailman 2 and 3) that has about a dozen of VPS servers that are categorized as high volume senders and we see great delivery rates. Our clients all use their own domain or sub-domains as for their list as well. The key for us has been to keep our clients' lists clean by enforcing reasonable bounce settings and screening new clients to make sure we don't mistakenly take on a spammer. We do work hard to maintain our IP reputations. Yahoo is the worst. They ignore everything, are hard to get a hold of, says they want you to join their feedback loop but don't actually respond to your attempts at doing so, etc. However once the IP address "warms up" (Yahoo's language, not ours) delivery rates are as good as the others.
The majority reason why messages ends up in Spam folders has nothing to do with the mail server at all. It is because the content of the message comes across as spammy, i.e. a message that contains several URLs, using exclamation points, etc.
Brian
Am 14. August 2019 um 20:44 Uhr -0000 schrieb brian@emwd.com:
Did you keep an eye on the reputation on the IP addresses you were sending from? Sometimes an IP address just doesn't send enough volume of mail to build one up.
This mail server ever only sent my personal and my family's e-mail -- no public mailing lists. The volume was very low. Maybe about ten e-mails a week in average, sometimes no e-mail a week at all. If low e-mail volume is indeed a criterion, then it could be the cause. But low volume as a spam criterion strikes me as doubtful, because a spammer specifically wants to issue mass e-mail.
The server was online since I think 2013 and always had the same IP. It wasn't a shared IP, i.e. this mail server was the only one sending from that IP. The spam classification problem was there from the very beginning.
We did get that fixed. The key to Microsoft is to sign up for their Hotmail Feedback Loop. They also provide a nice interface (SNDS) that shows you what they think of your IP addresses so you can mitigate any potential delivery issues ahead of time.
I'm leaving that to my SMTP relay provider now. I find it too much hassle to sign up for such services just for my private e-mail. I'm not getting paid for sinking my time into this; keeping the server running and clean is enough of time investment into my hobby (I use it for other things than e-mail, too). Microsoft is not the only e-mail provider, so if all e-mail providers did it like that, you'd have to create dozens and hundreds of accounts. Microsoft's pure size is what allows them to pose such a restriction successfully. The real problem is the centralisation of services at only a few providers.
Yahoo is the worst.
Luckily, Yahoo e-mail addresses are so rare in Germany that I can ignore them. I don't have anyone with a Yahoo address in my address book currently. outlook.com is much more common, but from my experience most German e-mail addresses are hosted by the United Internet group (GMX et al.), which I never had problems delivering to. They do a decent job. As for GMail, after a few attempts with followup questions my contacts started looking into their spam folder. That's why I was fine without the SMTP relay provider for years. But since recently, more and more people use outlook.com, which outright rejected.
The majority reason why messages ends up in Spam folders has nothing to do with the mail server at all. It is because the content of the message comes across as spammy, i.e. a message that contains several URLs, using exclamation points, etc.
Normal private 1-1 correspondence usually does not fulfill these criteria, and I've still been filed/rejected as spam (and likewise other family members). I felt like Don Quixote fighting the windmills. Today's e-mail infrastructure makes things difficult for private individuals who want to self-host.
Anyway, I think this subthread has grown large enough. My questions were answered. Thanks to everybody who commented!
-- Blog: https://mg.guelker.eu
David Krantz writes:
Is the documentation correct that ARC should be as simple to configure in mailman 3 as to add the private DKIM key and restarting mailman? (https://mailman.readthedocs.io/en/latest/src/mailman/handlers/docs/arc_sign....)
You need to enable ARC by adding the ARC section to mailman.cfg and setting "enabled = yes" in it.
I've submitted a pull request to update the documents.
I've tried that without finding any effect in my so far limited tests.
What do you mean by "effect"? Is the "effect" that you're not seeing the presence of ARC header fields in outgoing mail? If so, you probably haven't set the enabled flag.
If the effect you're looking for is that Google and several other participants in the ARC protocol accept your mail when they didn't before, you'll have to ask them about it.
I have not found anything in logs although I might have missed looking in one.
I don't recall that anything is logged by the module under normal circumstances, there may be some error logging. What would you expect to be logged?
Are there any known limitations?
Of what kind? It's a simple protocol: if there is authentication information (DKIM or ARC) in the incoming message, validate it. Then package up the validation in the ARC-AR, ARC-Signature, and ARC-Seal fields. Those are the operations the module performs.
Steve
-- Associate Professor Division of Policy and Planning Science http://turnbull.sk.tsukuba.ac.jp/ Faculty of Systems and Information Email: turnbull@sk.tsukuba.ac.jp University of Tsukuba Tel: 029-853-5175 Tennodai 1-1-1, Tsukuba 305-8573 JAPAN
On Tue, Aug 13, 2019 at 5:54 PM Stephen J. Turnbull <turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:
You need to enable ARC by adding the ARC section to mailman.cfg and setting "enabled = yes" in it.
Aha! That was the missing config. Thank you!
What do you mean by "effect"? Is the "effect" that you're not seeing the presence of ARC header fields in outgoing mail? If so, you probably haven't set the enabled flag.
No, I hadn't as it was not present in the documentation. Tested that and received a stacktrace where authres (of which I know nothing) throws an exception complaining on a syntax error when parsing the AuthenticationResultsHeader. It seems as if it requires an authserv-id field but the few examples of the AuthenticationResultsHeader I've looked at just starts with the server name followed by a semicolon, with no preceeding tag for that server name value.
Aug 14 13:43:25 2019 (28958) ACCEPT: <msgid_removed> Aug 14 13:43:26 2019 (28962) Uncaught runner exception: Syntax error: Expected authserv-id at: ; dkim=pass header.d=domain.tld; arc=none... Aug 14 13:43:26 2019 (28962) Traceback (most recent call last): File "...VENV/lib/python3.7/site-packages/mailman-3.2.3-py3.7.egg/mailman/core/runner.py", line 173, in _one_iteration self._process_one_file(msg, msgdata) File "...VENV/lib/python3.7/site-packages/mailman-3.2.3-py3.7.egg/mailman/core/runner.py", line 266, in _process_one_file keepqueued = self._dispose(mlist, msg, msgdata) File "...VENV/lib/python3.7/site-packages/mailman-3.2.3-py3.7.egg/mailman/runners/pipeline.py", line 37, in _dispose process(mlist, msg, msgdata, pipeline) File "...VENV/lib/python3.7/site-packages/mailman-3.2.3-py3.7.egg/mailman/core/pipelines.py", line 50, in process handler.process(mlist, msg, msgdata) File "...VENV/lib/python3.7/site-packages/mailman-3.2.3-py3.7.egg/mailman/handlers/arc_sign.py", line 71, in process sign(msg, msgdata) File "...VENV/lib/python3.7/site-packages/mailman-3.2.3-py3.7.egg/mailman/handlers/arc_sign.py", line 50, in sign standardize=('ARC-Standardize' in msgdata)) File "...VENV/lib/python3.7/site-packages/authheaders-0.11.0-py3.7.egg/authheaders/__init__.py", line 256, in sign_message return ARC(msg, logger=logger).sign(selector, domain, privkey, srv_id, include_headers=sig_headers, timestamp=timestamp, standardize=standardize) File "...VENV/lib/python3.7/site-packages/dkimpy-0.9.3-py3.7.egg/dkim/__init__.py", line 953, in sign for res in ar_headers] File "...VENV/lib/python3.7/site-packages/dkimpy-0.9.3-py3.7.egg/dkim/__init__.py", line 953, in <listcomp> for res in ar_headers] File "...VENV/lib/python3.7/site-packages/authres-1.2.0-py3.7.egg/authres/__init__.py", line 206, in parse return authres.core.AuthenticationResultsHeader.parse(core_features(), string) File "...VENV/lib/python3.7/site-packages/authres-1.2.0-py3.7.egg/authres/core.py", line 442, in parse return self.parse_value(feature_context, string) File "...VENV/lib/python3.7/site-packages/authres-1.2.0-py3.7.egg/authres/core.py", line 454, in parse_value header._parse() File "...VENV/lib/python3.7/site-packages/authres-1.2.0-py3.7.egg/authres/core.py", line 504, in _parse raise SyntaxError('Expected authserv-id', self._parse_text) authres.core.SyntaxError: Syntax error: Expected authserv-id at: ; dkim=pass header.d=domain.tld; arc=none... Aug 14 13:43:26 2019 (28962) SHUNTING: 1565790206.708188+e4196a80c2afc177ddc001ce8bbd8789f8d08912
Of what kind? It's a simple protocol: if there is authentication information (DKIM or ARC) in the incoming message, validate it. Then package up the validation in the ARC-AR, ARC-Signature, and ARC-Seal fields. Those are the operations the module performs.
Limitations in Mailman3. I really should have written "other requirements or known incompatibilities".
Thanks // David
David Krantz writes:
Tested that and received a stacktrace where authres (of which I know nothing) throws an exception complaining on a syntax error when parsing the AuthenticationResultsHeader. It seems as if it requires an authserv-id field but the few examples of the AuthenticationResultsHeader I've looked at just starts with the server name followed by a semicolon, with no preceeding tag for that server name value.
There is no tag for the authserv-id field; it is a value immediately after the field name and the colon, followed by a semicolon and the results for the various methods. It appears from the log messages that there is no value there. I believe that this field may not be empty according to RFC 8601. Without the message header, I can't verify that it was or wasn't, but I don't see why the log message would not contain the whole body of the AR or AAR header field.
line 504, in _parse raise SyntaxError('Expected authserv-id', self._parse_text) authres.core.SyntaxError: Syntax error: Expected authserv-id at: ; dkim=pass header.d=domain.tld; arc=none... Aug 14 13:43:26 2019 (28962) SHUNTING: 1565790206.708188+e4196a80c2afc177ddc001ce8bbd8789f8d08912
Steve
This is a way belated response, but since this is the only result which turns up when you search this error, I'm going to add it anyway. In short, when you get this error:
Aug 14 13:43:26 2019 (28962) Uncaught runner exception: Syntax error: Expected authserv-id at: ; dkim=pass header.d=domain.tld; arc=none... ... you've most likely missed to add the following to your mailman.conf: [ARC] authserv_id: yourdomain.tld
Mailman, unfortunately, doesn't verify that the [ARC] configurations are complete, and the documentation for enabling ARC-authentication is, as you noted, deceptively simple. The [ARC] chapter in the documentation for all configuration options (https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/handlers/do...) is much better, albeit a bit terse if you just want a minimal example.
The Arch wiki (https://wiki.archlinux.org/title/mailman#ARC) has a near-minimal example, but no explanations.
participants (6)
-
:3jörn Mårtensson -
brian＠emwd.com -
David Krantz -
Mark Sapiro -
Marvin Gülker -
Stephen J. Turnbull