Issues with a DMARC record leading to message being shunted
Hi!
Recently a mail was not being distributed via our MM3 instance; I searched the logs and found:
Dec 22 04:54:27 2023 (24) ACCEPT: <1d431542-b386-467b-8bf2-305f19ee7eb4@somenet.org> Dec 22 04:54:28 2023 (28) Uncaught runner exception: list index out of range Dec 22 04:54:28 2023 (28) Traceback (most recent call last): File "/usr/lib/python3.10/site-packages/mailman/core/runner.py", line 179, in _one_iteration self._process_one_file(msg, msgdata) File "/usr/lib/python3.10/site-packages/mailman/core/runner.py", line 272, in _process_one_file keepqueued = self._dispose(mlist, msg, msgdata) File "/usr/lib/python3.10/site-packages/mailman/runners/pipeline.py", line 37, in _dispose process(mlist, msg, msgdata, pipeline) File "/usr/lib/python3.10/site-packages/mailman/core/pipelines.py", line 53, in process handler.process(mlist, msg, msgdata) File "/usr/lib/python3.10/site-packages/mailman/handlers/validate_authenticity.py", line 125, in process authenticate(msg, msgdata) File "/usr/lib/python3.10/site-packages/mailman/utilities/retry.py", line 44, in f_retry return f(*args, **kwargs) File "/usr/lib/python3.10/site-packages/mailman/handlers/validate_authenticity.py", line 93, in authenticate auth_result = authenticate_message( File "/usr/lib/python3.10/site-packages/authheaders/__init__.py", line 395, in authenticate_message dmarc_result = check_dmarc(msg, spf_result, dkim_result, dnsfunc=dnsfunc, psddmarc=psddmarc) File "/usr/lib/python3.10/site-packages/authheaders/__init__.py", line 343, in check_dmarc result, result_comment, from_domain, policy = dmarc_per_from(from_domain, spf_result, dkim_result, dnsfunc, psddmarc) File "/usr/lib/python3.10/site-packages/authheaders/__init__.py", line 90, in dmarc_per_from record, orgdomain = receiver_record(from_domain) File "/usr/lib/python3.10/site-packages/authheaders/dmarc_lookup.py", line 117, in receiver_record retval = lookup_receiver_record(hostSansDmarc, dnsfunc) File "/usr/lib/python3.10/site-packages/authheaders/dmarc_lookup.py", line 92, in lookup_receiver_record tags = answer_to_dict(str(result)) File "/usr/lib/python3.10/site-packages/authheaders/dmarc_lookup.py", line 42, in answer_to_dict retval = {t[0].strip().lower(): t[1].strip().lower() for t in rawTags} File "/usr/lib/python3.10/site-packages/authheaders/dmarc_lookup.py", line 42, in <dictcomp> retval = {t[0].strip().lower(): t[1].strip().lower() for t in rawTags} IndexError: list index out of range
Dec 22 04:54:28 2023 (28) SHUNTING: 1703220868.1418982+b27f5db154375fcd2c418d467172e4aad0d3d57a
So the message has been shunted, due to this error. Something with DMARC and tags. So I checked the DMARC record for the sending domain somenet.org. It is "v=DMARC1;p=none;pct=100;rua=mailto:postmaster@somenet.org;mailto:postmaster@somenet.org;ri=3600;fo=1;" which is syntactically incorrect. The extra ";mailto:postmaster@somenet.org" is wrong, I guess the tag "ruf=" is missing here!
It should probably read "v=DMARC1;p=none;pct=100;rua=mailto:postmaster@somenet.org;ruf=mailto:postmaster@somenet.org;ri=3600;fo=1;"
So, should mm3 somehow "catch" this error somehow?
According to the DMARC specs, an syntactically incorrect DMARC record should be ignored / handled as if none were present.
-- Ralf Hildebrandt Charité - Universitätsmedizin Berlin Geschäftsbereich IT | Abteilung Netz | Netzwerk-Administration Invalidenstraße 120/121 | D-10115 Berlin
Tel. +49 30 450 570 155 ralf.hildebrandt@charite.de https://www.charite.de
On 12/23/23 1:57 AM, Ralf Hildebrandt via Mailman-users wrote:
Hi!
Recently a mail was not being distributed via our MM3 instance; I searched the logs and found:
Dec 22 04:54:27 2023 (24) ACCEPT: <1d431542-b386-467b-8bf2-305f19ee7eb4@somenet.org> Dec 22 04:54:28 2023 (28) Uncaught runner exception: list index out of range Dec 22 04:54:28 2023 (28) Traceback (most recent call last): File "/usr/lib/python3.10/site-packages/mailman/core/runner.py", line 179, in _one_iteration self._process_one_file(msg, msgdata) File "/usr/lib/python3.10/site-packages/mailman/core/runner.py", line 272, in _process_one_file keepqueued = self._dispose(mlist, msg, msgdata) File "/usr/lib/python3.10/site-packages/mailman/runners/pipeline.py", line 37, in _dispose process(mlist, msg, msgdata, pipeline) File "/usr/lib/python3.10/site-packages/mailman/core/pipelines.py", line 53, in process handler.process(mlist, msg, msgdata) File "/usr/lib/python3.10/site-packages/mailman/handlers/validate_authenticity.py", line 125, in process authenticate(msg, msgdata) File "/usr/lib/python3.10/site-packages/mailman/utilities/retry.py", line 44, in f_retry return f(*args, **kwargs) File "/usr/lib/python3.10/site-packages/mailman/handlers/validate_authenticity.py", line 93, in authenticate auth_result = authenticate_message( File "/usr/lib/python3.10/site-packages/authheaders/__init__.py", line 395, in authenticate_message dmarc_result = check_dmarc(msg, spf_result, dkim_result, dnsfunc=dnsfunc, psddmarc=psddmarc) File "/usr/lib/python3.10/site-packages/authheaders/__init__.py", line 343, in check_dmarc result, result_comment, from_domain, policy = dmarc_per_from(from_domain, spf_result, dkim_result, dnsfunc, psddmarc) File "/usr/lib/python3.10/site-packages/authheaders/__init__.py", line 90, in dmarc_per_from record, orgdomain = receiver_record(from_domain) File "/usr/lib/python3.10/site-packages/authheaders/dmarc_lookup.py", line 117, in receiver_record retval = lookup_receiver_record(hostSansDmarc, dnsfunc) File "/usr/lib/python3.10/site-packages/authheaders/dmarc_lookup.py", line 92, in lookup_receiver_record tags = answer_to_dict(str(result)) File "/usr/lib/python3.10/site-packages/authheaders/dmarc_lookup.py", line 42, in answer_to_dict retval = {t[0].strip().lower(): t[1].strip().lower() for t in rawTags} File "/usr/lib/python3.10/site-packages/authheaders/dmarc_lookup.py", line 42, in <dictcomp> retval = {t[0].strip().lower(): t[1].strip().lower() for t in rawTags} IndexError: list index out of range
Dec 22 04:54:28 2023 (28) SHUNTING: 1703220868.1418982+b27f5db154375fcd2c418d467172e4aad0d3d57a
So the message has been shunted, due to this error. Something with DMARC and tags. So I checked the DMARC record for the sending domain somenet.org. It is "v=DMARC1;p=none;pct=100;rua=mailto:postmaster@somenet.org;mailto:postmaster@somenet.org;ri=3600;fo=1;" which is syntactically incorrect. The extra ";mailto:postmaster@somenet.org" is wrong, I guess the tag "ruf=" is missing here!
It should probably read "v=DMARC1;p=none;pct=100;rua=mailto:postmaster@somenet.org;ruf=mailto:postmaster@somenet.org;ri=3600;fo=1;"
So, should mm3 somehow "catch" this error somehow?
Perhaps we should catch it.
We already have other issues involving authheaders.authenticate_message. There's an invalid From: issue at https://gitlab.com/mailman/mailman/-/issues/1100 reported at https://github.com/ValiMail/authentication-headers/issues/25 and a different invalid DMARC issue at https://gitlab.com/mailman/mailman/-/issues/1109 reported at https://github.com/ValiMail/authentication-headers/issues/26.
For now, I reported this at https://github.com/ValiMail/authentication-headers/issues/27
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Attempted upgrade on Debian 10 to Mailman3.3 went badly. Prior to attempted upgrade system was working but with incremental unsubscribes due to erroneous bounces upping subscribers scores. Restored from backup. System now not working. Postorius webpage says "Mailman REST API not available. Please start Mailman core." Mailman will not start. Command line thread is as follows:
root@lists:~# systemctl start mailman Job for mailman.service failed because the control process exited with error code. See "systemctl status mailman.service" and "journalctl -xe" for details. root@lists:~# systemctl status mailman.service ● mailman.service - GNU Mailing List Manager Loaded: loaded (/lib/systemd/system/mailman.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Sat 2023-12-23 09:01:41 PST; 5min ago Process: 3773 ExecStart=/opt/mailman/mm/bin/mailman start (code=exited, status=2)
Dec 23 09:01:41 lists.ccalternatives.org systemd[1]: Starting GNU Mailing List Manager... Dec 23 09:01:41 lists.ccalternatives.org mailman[3773]: Usage: mailman start [OPTIONS] Dec 23 09:01:41 lists.ccalternatives.org mailman[3773]: Try 'mailman start -h' for help. Dec 23 09:01:41 lists.ccalternatives.org mailman[3773]: Error: A previous run of GNU Mailman did not exit cleanly (host_mismatch). Try using --force Dec 23 09:01:41 lists.ccalternatives.org systemd[1]: mailman.service: Control process exited, code=exited, status=2/INVALIDARGUMENT Dec 23 09:01:41 lists.ccalternatives.org systemd[1]: mailman.service: Failed with result 'exit-code'. Dec 23 09:01:41 lists.ccalternatives.org systemd[1]: Failed to start GNU Mailing List Manager. root@lists:~#
Please help. Thank you!
On 12/23/23 9:13 AM, Christian via Mailman-users wrote:
Dec 23 09:01:41 lists.ccalternatives.org mailman[3773]: Usage: mailman start [OPTIONS] Dec 23 09:01:41 lists.ccalternatives.org mailman[3773]: Try 'mailman start -h' for help. Dec 23 09:01:41 lists.ccalternatives.org mailman[3773]: Error: A previous run of GNU Mailman did not exit cleanly (host_mismatch). Try using --force
Remove all the files from Mailman's var/locks/ directory.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
- Mark Sapiro <mark@msapiro.net>:
For now, I reported this at https://github.com/ValiMail/authentication-headers/issues/27
Thanks!
-- Ralf Hildebrandt Charité - Universitätsmedizin Berlin Geschäftsbereich IT | Abteilung Netz | Netzwerk-Administration Invalidenstraße 120/121 | D-10115 Berlin
Tel. +49 30 450 570 155 ralf.hildebrandt@charite.de https://www.charite.de
Ralf Hildebrandt wrote:
Mark Sapiro mark@msapiro.net:
For now, I reported this at https://github.com/ValiMail/authentication-headers/issues/27
And that issue and others have been fixed in authheaders 0.16.0 which is now a requirement. See https://gitlab.com/mailman/mailman/-/merge_requests/1165
Ralf Hildebrandt wrote:
Mark Sapiro mark@msapiro.net:
For now, I reported this at https://github.com/ValiMail/authentication-headers/issues/27
And that issue and others have been fixed in authheaders 0.16.0 which is now a requirement. See https://gitlab.com/mailman/mailman/-/merge_requests/1165
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
participants (3)
-
Christian -
Mark Sapiro -
Ralf Hildebrandt