Office365 posts shunted (authres.core.SyntaxError)
Hi again --
Besides my compress issue described earlier today here:
https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/message/...
I've been dealing with another issue (and why I upgraded to the latest code) when posting messages to my test list from our schools Office365 infrastructure (OWA). The post gets shunted; mailman.log has the following error:
Sep 11 11:29:44 2020 (31396) ACCEPT: <BL0PR14MB3681E636E905E9D34588C85088240@BL0PR14MB3681.namprd14.prod.outlook.com> Sep 11 11:29:45 2020 (31401) Uncaught runner exception: Syntax error: Expected "=" at: ; dmarc=none action=none header.from=o... Sep 11 11:29:45 2020 (31401) Traceback (most recent call last): File "/opt/mailman/mm/venv/lib/python3.6/site-packages/mailman/core/runner.py", line 173, in _one_iteration self._process_one_file(msg, msgdata) File "/opt/mailman/mm/venv/lib/python3.6/site-packages/mailman/core/runner.py", line 266, in _process_one_file keepqueued = self._dispose(mlist, msg, msgdata) File "/opt/mailman/mm/venv/lib/python3.6/site-packages/mailman/runners/pipeline.py", line 37, in _dispose process(mlist, msg, msgdata, pipeline) File "/opt/mailman/mm/venv/lib/python3.6/site-packages/mailman/core/pipelines.py", line 50, in process handler.process(mlist, msg, msgdata) File "/opt/mailman/mm/venv/lib/python3.6/site-packages/mailman/handlers/validate_authenticity.py", line 118, in process authenticate(msg, msgdata) File "/opt/mailman/mm/venv/lib/python3.6/site-packages/mailman/utilities/retry.py", line 44, in f_retry return f(*args, **kwargs) File "/opt/mailman/mm/venv/lib/python3.6/site-packages/mailman/handlers/validate_authenticity.py", line 85, in authenticate prev = trusted_auth_res(msg) File "/opt/mailman/mm/venv/lib/python3.6/site-packages/mailman/handlers/validate_authenticity.py", line 73, in trusted_auth_res authserv_id = AuthenticationResultsHeader.parse(header).authserv_id File "/opt/mailman/mm/venv/lib/python3.6/site-packages/authres/__init__.py", line 206, in parse return authres.core.AuthenticationResultsHeader.parse(core_features(), string) File "/opt/mailman/mm/venv/lib/python3.6/site-packages/authres/core.py", line 442, in parse return self.parse_value(feature_context, string) File "/opt/mailman/mm/venv/lib/python3.6/site-packages/authres/core.py", line 454, in parse_value header._parse() File "/opt/mailman/mm/venv/lib/python3.6/site-packages/authres/core.py", line 517, in _parse result = self._parse_resinfo() File "/opt/mailman/mm/venv/lib/python3.6/site-packages/authres/core.py", line 550, in _parse_resinfo method, version, result = self._parse_methodspec() File "/opt/mailman/mm/venv/lib/python3.6/site-packages/authres/core.py", line 573, in _parse_methodspec raise SyntaxError('Expected "="', self._parse_text) authres.core.SyntaxError: Syntax error: Expected "=" at: ; dmarc=none action=none header.from=o... Sep 11 11:29:45 2020 (31401) SHUNTING: 1599838185.4911082+9dde922f77a1d66e3511f2709ec4b436ab6a3ecc
Debugging the issue further shows this header to be the problem:
authentication-results: lists-test.rutgers.edu; dkim=none (message not signed) header.d=none;lists-test.rutgers.edu; dmarc=none action=none header.from=oit.rutgers.edu;
What is syntactically wrong with it and/or is it more than that?
Note1: If I remove that header from a copy of the message and inject directly into Mailman it will deliver as expected. Note2: Posting messages from my personal Gmail account or cmdline mail program to the same test list will also deliver as expected.
Bryan Loniewski Rutgers University OIT:EAS:Messaging and Collaboration Services System Administrator
Bryan Loniewski writes:
I've been dealing with another issue (and why I upgraded to the latest code) when posting messages to my test list from our schools Office365 infrastructure (OWA). The post gets shunted; mailman.log has the following error:
Bottom line: the reported Authentication-Results (A-R) field has nonconforming syntax. The problem is that the RFC defines a syntax where the authentication service ID is the first thing in the field, and only one is allowed. In the quoted field it occurs twice. There are more syntax errors, but that's the one Mailman flagged.
As far as I know the only part of Mailman that uses A-R is the ARC module. Just disable it and be happy, I think. Like DKIM, DMARC, and friends, this is really an MTA function. If possible you should get your MTA to do the work.
BTW, I'm not sure this is an Office365 problem. It appears that the A-R field was constructed on lists-test.rutgers.edu, which doesn't look like Office365. Is it possible that the Office365 system didn't provide one, and this A-R field was added by Mailman itself? (Don't overthink this: if so, it's our problem and I'll be looking at it. But if you know offhand it will point me in the right direction.)
Excruciating detail follows, in part as notes to me. ;-)
In the following, the in-reply-to quoting is omitted and the field is folded for clarity.
authentication-results: # field name lists-test.rutgers.edu; # authenticator ID dkim=none (message not signed) # DKIM result info header.d=none; # DKIM property (no DKIM signature field) lists-test.rutgers.edu; # #### nonconforming (not "; key=value") dmarc=none # DMARC result info action=none # #### syntax OK, undefined property header.from=oit.rutgers.edu # DMARC property (field to check) ; # #### nonconforming (nothing to separate)
So it looks like whoever wrote the code doesn't understand RFC 7601 at all. A-R starts with a string naming the authentication authority (here lists-test.rutgers.edu) and an optional number naming the version of A-R being used. These are followed either by a semicolon and the word "none", or by a sequence of results blocks. Each block consists of a semicolon followed by a sequence of key=value pairs.
This funny phrase "a semicolon followed by a sequence of key=value pairs" is probably easier to understand if I say "the semicolons separate results blocks from the authenticator ID and from each other" (they are not block terminators).
RFC 7601 is quite pessimistic about nonconforming A-R, and repeatedly advises they be ignored or even deleted, although completely broken syntax is not mentioned.
Steve
Hi Steve --
Thanks for the response. You are correct in the fact that I've enabled/configured the ARC module, however, I don't think Mailman is adding that (malformed) header considering I'm now HOLDing the test email(s) sent to our postfix inbound MX servers *before* they are LMTP'd to the Mailman server and its already there when I postcat it.
At the same time I also can't fathom that Office365 has this wrong (still?) and no one has noticed/complained?
brylon@oit.rutgers.edu writes:
Thanks for the response. You are correct in the fact that I've enabled/configured the ARC module, however, I don't think Mailman is adding that (malformed) header considering I'm now HOLDing the test email(s) sent to our postfix inbound MX servers *before* they are LMTP'd to the Mailman server and its already there when I postcat it.
OK, thanks for that clarification. I feel better (but will still check it! just a bit less urgently ;-)
What do you still need from us? I guess the easiest thing for us would be to catch the Exception in our ARC module, and ignore nonconforming Authentication-Results, so that there will be no ARC signature from Mailman on those posts. Will that do? (Note that "ignore and/or delete" is the RFC recommendation. I'd need more than "it would be nice" as a reason to try to parse non-conforming A-R. Mark's experience with parsing *conforming* (non)delivery notifications is quite a deterrent to that.)
At the same time I also can't fathom that Office365 has this wrong (still?) and no one has noticed/complained?
I guess I would say, how would we know if there were complaints? ARC is a very recent addition to Mailman, so you wouldn't necessarily see it here yet, and Office365 doesn't have a public bug database AFAIK. ;-) Microsoft (and other > 1G$ corporations) have a long history of "embrace, extend, and drive conformant implementations out of business".
Yes, I think catch and ignore in the ARC module would be acceptable. Is there any additional information I can provide to help and/or any further actions to assist with?
brylon@oit.rutgers.edu writes:
Yes, I think catch and ignore in the ARC module would be acceptable. Is there any additional information I can provide to help and/or any further actions to assist with?
I think the work needed is quite clear.
How pressing is this for you? We won't have a new release for some weeks to a few months as far as I know, but we could have a patch (or a single file to replace) that you could apply more quickly. Since it would be in the ARC module, which is independent of the rest of the code, it would be a safe patch to apply, and easy to reverse.
As far as my schedule, I'm very busy for the next two weeks (my heavy teaching term starts on October 1). I'm looking at working on this intermittently starting maybe this weekend but probably not finishing a mergeable patch until late October. A patch you could apply (ie, just the code, not the documentation and tests) would be done sooner.
I haven't talked to others about their schedules, somebody else might pick it up and I would have time to answer emails and review changes from them, but I'm the obvious person to work on ARC.
Steve
Okay. Well the current plan/goal is to have a production setup/rollout in a few months time (i.e., 12/31/2020), so a knowledgeable[1] patch I could apply while still testing would be fantastic. We currently host ~4000 lists on MM2 (although that number will be closer to ~2500 migrating to our in the works MM3 deployment after we audit MM2 in a month or so). Thanks for devoting any time you can find to this! [1] I currently applied this hack to get around the syntax error... $ diff -u venv/lib/python3.6/site-packages/mailman/handlers/validate_authenticity.py.ORIG venv/lib/python3.6/site-packages/mailman/handlers/validate_authenticity.py --- venv/lib/python3.6/site-packages/mailman/handlers/validate_authenticity.py.ORIG 2020-09-16 14:23:03.565482574 -0400 +++ venv/lib/python3.6/site-packages/mailman/handlers/validate_authenticity.py 2020-09-17 13:17:09.296409626 -0400 @@ -70,10 +70,12 @@ if config.arc.trusted_authserv_ids and (AUTH_RESULT_HEADER in msg): header = '{}: {}'.format(AUTH_RESULT_HEADER, msg[AUTH_RESULT_HEADER]) - authserv_id = AuthenticationResultsHeader.parse(header).authserv_id - if authserv_id in config.arc.trusted_authserv_ids: - return header - + try: + authserv_id = AuthenticationResultsHeader.parse(header).authserv_id + if authserv_id in config.arc.trusted_authserv_ids: + return header + except: + log.exception('AR header failed to parse for message: %s', msg['Message-ID']) @retry(Timeout, NUM_TIMEOUT_RETRIES) def authenticate(msg, msgdata):
On 9/17/20 2:41 PM, brylon@oit.rutgers.edu wrote:
[1] I currently applied this hack to get around the syntax error...
$ diff -u venv/lib/python3.6/site-packages/mailman/handlers/validate_authenticity.py.ORIG venv/lib/python3.6/site-packages/mailman/handlers/validate_authenticity.py --- venv/lib/python3.6/site-packages/mailman/handlers/validate_authenticity.py.ORIG 2020-09-16 14:23:03.565482574 -0400 +++ venv/lib/python3.6/site-packages/mailman/handlers/validate_authenticity.py 2020-09-17 13:17:09.296409626 -0400 @@ -70,10 +70,12 @@
if config.arc.trusted_authserv_ids and (AUTH_RESULT_HEADER in msg): header = '{}: {}'.format(AUTH_RESULT_HEADER, msg[AUTH_RESULT_HEADER]) - authserv_id = AuthenticationResultsHeader.parse(header).authserv_id - if authserv_id in config.arc.trusted_authserv_ids: - return header - + try: + authserv_id = AuthenticationResultsHeader.parse(header).authserv_id + if authserv_id in config.arc.trusted_authserv_ids: + return header + except: + log.exception('AR header failed to parse for message: %s', msg['Message-ID'])
@retry(Timeout, NUM_TIMEOUT_RETRIES) def authenticate(msg, msgdata):
I think that's fine. Also, you should look at <https://gitlab.com/mailman/mailman/-/issues/765>. -- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mark Sapiro writes:
I think that's fine. Also, you should look at <https://gitlab.com/mailman/mailman/-/issues/765>.
I agree with Mark. I was pretty sure something like that would work, but I haven't looked at the code in a while, so didn't know exactly where to patch. I'll take a look at it tomorrow, and see if I can spot any edge cases, but that's probably as simple as it looks.
Steve
participants (4)
-
Bryan Loniewski -
brylon＠oit.rutgers.edu
-
Mark Sapiro -
Stephen J. Turnbull