April 2024 - Mailman-users - lists.mailman3.org

Issues with a DMARC record leading to message being shunted
by Ralf Hildebrandt 23 Jul '24

23 Jul '24

Hi! Recently a mail was not being distributed via our MM3 instance; I searched the logs and found: Dec 22 04:54:27 2023 (24) ACCEPT: <1d431542-b386-467b-8bf2-305f19ee7eb4(a)somenet.org> Dec 22 04:54:28 2023 (28) Uncaught runner exception: list index out of range Dec 22 04:54:28 2023 (28) Traceback (most recent call last): File "/usr/lib/python3.10/site-packages/mailman/core/runner.py", line 179, in _one_iteration self._process_one_file(msg, msgdata) File "/usr/lib/python3.10/site-packages/mailman/core/runner.py", line 272, in _process_one_file keepqueued = self._dispose(mlist, msg, msgdata) File "/usr/lib/python3.10/site-packages/mailman/runners/pipeline.py", line 37, in _dispose process(mlist, msg, msgdata, pipeline) File "/usr/lib/python3.10/site-packages/mailman/core/pipelines.py", line 53, in process handler.process(mlist, msg, msgdata) File "/usr/lib/python3.10/site-packages/mailman/handlers/validate_authenticity.py", line 125, in process authenticate(msg, msgdata) File "/usr/lib/python3.10/site-packages/mailman/utilities/retry.py", line 44, in f_retry return f(*args, **kwargs) File "/usr/lib/python3.10/site-packages/mailman/handlers/validate_authenticity.py", line 93, in authenticate auth_result = authenticate_message( File "/usr/lib/python3.10/site-packages/authheaders/__init__.py", line 395, in authenticate_message dmarc_result = check_dmarc(msg, spf_result, dkim_result, dnsfunc=dnsfunc, psddmarc=psddmarc) File "/usr/lib/python3.10/site-packages/authheaders/__init__.py", line 343, in check_dmarc result, result_comment, from_domain, policy = dmarc_per_from(from_domain, spf_result, dkim_result, dnsfunc, psddmarc) File "/usr/lib/python3.10/site-packages/authheaders/__init__.py", line 90, in dmarc_per_from record, orgdomain = receiver_record(from_domain) File "/usr/lib/python3.10/site-packages/authheaders/dmarc_lookup.py", line 117, in receiver_record retval = lookup_receiver_record(hostSansDmarc, dnsfunc) File "/usr/lib/python3.10/site-packages/authheaders/dmarc_lookup.py", line 92, in lookup_receiver_record tags = answer_to_dict(str(result)) File "/usr/lib/python3.10/site-packages/authheaders/dmarc_lookup.py", line 42, in answer_to_dict retval = {t[0].strip().lower(): t[1].strip().lower() for t in rawTags} File "/usr/lib/python3.10/site-packages/authheaders/dmarc_lookup.py", line 42, in <dictcomp> retval = {t[0].strip().lower(): t[1].strip().lower() for t in rawTags} IndexError: list index out of range Dec 22 04:54:28 2023 (28) SHUNTING: 1703220868.1418982+b27f5db154375fcd2c418d467172e4aad0d3d57a So the message has been shunted, due to this error. Something with DMARC and tags. So I checked the DMARC record for the sending domain somenet.org. It is "v=DMARC1;p=none;pct=100;rua=mailto:postmaster@somenet.org;mailto:postmaster@somenet.org;ri=3600;fo=1;" which is syntactically incorrect. The extra ";mailto:postmaster@somenet.org" is wrong, I guess the tag "ruf=" is missing here! It should probably read "v=DMARC1;p=none;pct=100;rua=mailto:postmaster@somenet.org;ruf=mailto:postmaster@somenet.org;ri=3600;fo=1;" So, should mm3 somehow "catch" this error somehow? According to the DMARC specs, an syntactically incorrect DMARC record should be ignored / handled as if none were present. -- Ralf Hildebrandt Charité - Universitätsmedizin Berlin Geschäftsbereich IT | Abteilung Netz | Netzwerk-Administration Invalidenstraße 120/121 | D-10115 Berlin Tel. +49 30 450 570 155 ralf.hildebrandt(a)charite.de https://www.charite.de

4 10

Digests not working correctly
by Joel Lord 08 Jun '24

08 Jun '24

So I seem to have problems with both periodic AND size-triggered digests. Size first, this is simpler. Just got one sent out, clearly triggered by a new email getting added to the queued up messages, but the setting for the list is the default 30KB and the digest came out at 23.2 KB. Is it measuring the accumulated queue as size on disk including metadata and filesystem block sizes or something, rather than the size of what the emailed digest will be? Periodic is much more complicated, even to explain. I have the cron job running nightly at 11:00 PM and it certainly seems to be working, except for one really odd thing. The list also definitely has periodic digests enabled in the web UI. This is a very bursty list, so it will go mad with traffic for a few weeks and then will go mostly idle for a month or so. The first digest of a burst is definitely size-triggered, but pretty reliably it has the last few messages from the previous burst in there. So the one I got today had three messages from today and one message from May 4th. But that May 4th message should have gone out at 11:00 PM on May 4th with the daily digest and then gotten flushed from the pending digest queue, right? -- Joel Lord

3 14

Change Dmark settings via command line
by bob B 08 May '24

08 May '24

I need to set all the DMARC settings on are lists to "replace from with list address". Is there an easy want to do this via cli? I could do it by viewing the web interface but we have a lot of lists.

4 9

Welcome message template errors
by Mark 01 May '24

01 May '24

Hi All I'm getting errors when a custom "Welcome message" template is in place [list:user:notice:welcome]. Case: Subscribing a new user via the admin "Mass subscribe". Although the new user is successfully subscribed, a welcome message is not sent and the following error message displays on the webpage. === HTTP Error 400: HTTPConnectionPool(host='localhost', port=8000): Read timed out. (read timeout=5) === And the site-owner receives an email with ... ==== Internal Server Error: /mailman3/lists/ DoesNotExist at /mailman3/lists/ Site matching query does not exist. Request Method: GET Request URL: http://<REDACTED-PUBLIC-IPv4-ADDRESS>/mailman3/lists/ Django Version: 4.2.11 Python Executable: /opt/mailman/venv/bin/python3 Python Version: 3.11.2 ... etc ... === Regarding the second line in the email - my thinking is that it should be looking in "/opt/mailman/mm/var/lists" and not "/mailman3/lists/". If that's right, where would I look to change that directive? If not, any other clues would be greatly appreciated. The error doesn't occur if the is no custom "Welcome message" template - ie. the user is sent the default welcome message. $ mailman info GNU Mailman 3.3.9 (Tom Sawyer) Python 3.11.2 (main, Mar 13 2023, 12:18:29) [GCC 12.2.0] config file: /etc/mailman3/mailman.cfg REST root url: http://localhost:8001/3.1/

2 29

Re: [MM3-users off list] Re: Welcome message template errors
by Mark 29 Apr '24

29 Apr '24

>> So, I'm thinking (in broad brush strokes) that if I'm stuck with this >> "http glitch" I could write a script to: >> >> 1. extract the text of the custom message from the database, >> 2. write that message text to a file >> 3. update the templates.uri entries to replace the >> http://localhost... with file:///path/to/file >> 4. cron the script to run every few minutes. >> >> It screams of utter clunk I know... > but it would work. > > Note that this may not be as big an issue as it seems. It doesn't > affect all templates, for example list:member:digest:footer, > list:member:digest:header, list:member:digest:masthead, > list:member:regular:footer, list:member:regular:header and > list:member:generic:footer are only used by the `out` runner when > sending messages, so these aren't affected. Right. Affected are all the "list:user:notice:***" (eg. goodbye, hold, reject,...) > > I'm still trying to determine what the issue is. To this end can you > tell me what Python version your Mailman uses and what is the output > from `pip freeze` with your venv active? $ mailman info GNU Mailman 3.3.9 (Tom Sawyer) Python 3.11.2 (main, Mar 13 2023, 12:18:29) [GCC 12.2.0] $ pip freeze aiosmtpd==1.4.4.post2 alembic==1.13.1 arrow==1.3.0 asgiref==3.7.2 atpublic==4.0 attrs==23.2.0 authheaders==0.16.2 authres==1.2.0 blessed==1.20.0 certifi==2023.11.17 cffi==1.16.0 charset-normalizer==3.3.2 click==8.1.7 cmarkgfm==2024.1.14 cryptography==42.0.1 defusedxml==0.7.1 Django==4.2.11 django-allauth==0.60.1 django-appconf==1.0.6 django-compressor==4.4 django-extensions==3.2.3 django-gravatar2==1.4.4 django-haystack==3.2.1 django-mailman3==1.3.12 django-picklefield==3.1 django-q==1.3.9 djangorestframework==3.14.0 dkimpy==1.1.5 dnspython==2.5.0 docutils==0.20.1 falcon==3.1.3 filelock==3.13.1 flufl.bounce==4.0 flufl.i18n==5.0.2 flufl.lock==8.0.2 greenlet==3.0.3 gunicorn==21.2.0 HyperKitty==1.3.9 idna==3.6 lazr.config==3.0 lazr.delegates==2.1.0 mailman==3.3.9 mailman-hyperkitty==1.2.1 mailman-web==0.0.9 mailmanclient==3.3.5 Mako==1.3.0 MarkupSafe==2.1.4 mistune==3.0.2 networkx==3.2.1 nh3==0.2.15 oauthlib==3.2.2 packaging==23.2 passlib==1.7.4 postorius==1.3.10 psutil==5.9.8 psycopg2-binary==2.9.9 publicsuffix2==2.20191221 pycparser==2.21 Pygments==2.17.2 PyJWT==2.8.0 python-dateutil==2.8.2 python3-openid==3.2.0 pytz==2023.3.post1 rcssmin==1.1.1 readme-renderer==42.0 redis==3.5.3 requests==2.31.0 requests-oauthlib==1.3.1 rjsmin==1.2.1 robot-detection==0.4 six==1.16.0 SQLAlchemy==2.0.25 sqlparse==0.4.4 types-python-dateutil==2.8.19.20240106 typing_extensions==4.9.0 urllib3==2.1.0 wcwidth==0.2.13 Whoosh==2.7.4 xapian @ file:///opt/mailman/xapian-1.4.24-cp311-cp311-linux_aarch64.whl xapian-haystack==3.1.0 zope.component==6.0 zope.configuration==5.0 zope.event==5.0 zope.hookable==6.0 zope.i18nmessageid==6.1.0 zope.interface==6.1 zope.schema==7.0.1

1 0

Mailman with Python 3.12
by Andrew Hodgson 26 Apr '24

26 Apr '24

Hi. What's the latest on Mailman with Python 3.12? Looking at Ubuntu 24.04 upgrade plans and realised it comes with Python 3.12. Ubuntu are shipping Mailman packages from Debian so not sure if they have made any changes to make Python 3.12 work. I don't use those packages myself. Andrew.

3 3

mailman3-web and the django middleware problem
by Thomas Stein 26 Apr '24

26 Apr '24

Howdy! I'm trying to migrate an aged mailman3 installation from ubuntu 20.04 to ubuntu 22.04. Fresh install of mailman3-web gives me: Creating config file /etc/dbconfig-common/mailman3-web.conf with new version creating database mailman3web.db: success. verifying database mailman3web.db exists: success. Creating config file /etc/mailman3/mailman-web.py with new version /usr/lib/python3/dist-packages/django_q/conf.py:174: UserWarning: Retry and timeout are misconfigured. Set retry larger than t imeout,failure to do so will cause the tasks to be retriggered before completion.See https://django-q2.readthedocs.io/en/maste r/configure.html#retry for details. warn( Traceback (most recent call last): File "/usr/bin/django-admin", line 33, in <module> sys.exit(load_entry_point('Django==4.2.11', 'console_scripts', 'django-admin')()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 442, in execute_from_command_line utility.execute() File "/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 416, in execute django.setup() File "/usr/lib/python3/dist-packages/django/__init__.py", line 24, in setup apps.populate(settings.INSTALLED_APPS) File "/usr/lib/python3/dist-packages/django/apps/registry.py", line 124, in populate app_config.ready() File "/usr/lib/python3/dist-packages/allauth/account/apps.py", line 15, in ready raise ImproperlyConfigured( django.core.exceptions.ImproperlyConfigured: allauth.account.middleware.AccountMiddleware must be added to settings.MIDDLEWARE dpkg: error processing package mailman3-web (--configure): installed mailman3-web package post-installation script subprocess returned error exit status 1 Errors were encountered while processing: mailman3-web needrestart is being skipped since dpkg has failed E: Sub-process /usr/bin/dpkg returned an error code (1) I've already seen this thread: https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/message… ...but that does not get me anywhere. Could someone please push me in the right direction? cheers! t.

4 12

Slow performance when listing members on Postorius
by Alan So 24 Apr '24

24 Apr '24

We are about to upgrade to Mailman core 3.3.9 and Postorius 1.3.10. We notice that for setup with large number of members, the performance is much slower when listing members on web (Postorius). Further troubleshooting show that there are quite a number of API calls which do not appear in Mailman core 3.3.5 and Postorius 1.3.6 when we list members. [17/Apr/2024:14:12:22 +0800] "GET /3.1/members/33a6266d0a04472c8d5d6c78869a964c/preferences HTTP/1.1" 200 156 "-" "GNU Mailman REST client v3.3.5" [17/Apr/2024:14:12:22 +0800] "GET /3.1/members/7a18b7cb90a34ac98881a3f01b5488d1/preferences HTTP/1.1" 200 156 "-" "GNU Mailman REST client v3.3.5" [17/Apr/2024:14:12:22 +0800] "GET /3.1/members/61c6c3e2aaf443babd3bd191519b0424/preferences HTTP/1.1" 200 156 "-" "GNU Mailman REST client v3.3.5" [17/Apr/2024:14:12:22 +0800] "GET /3.1/members/8dd681165b8c436fac6793cae1ed2c27/preferences HTTP/1.1" 200 156 "-" "GNU Mailman REST client v3.3.5" Further tests showing that each database query using "_member_id" in table member can take as much as 0.3 to 0.5 seconds for some large installations. The default 25 members per page mean that it will take at least 7.5 to 12.5 seconds. Changing to 200 members per page will take even longer time. We wonder whether "_member_id" should be indexed just like most other id in the database mailman_core in order to solve the issue. Or are there any other workarounds? +----------------------+--------------+------+-----+---------+----------------+ | Field | Type | Null | Key | Default | Extra | +----------------------+--------------+------+-----+---------+----------------+ | id | int(11) | NO | PRI | NULL | auto_increment | | _member_id | char(32) | YES | | NULL | | | role | int(11) | YES | MUL | NULL | | | list_id | varchar(255) | YES | MUL | NULL | | | moderation_action | int(11) | YES | | NULL | | | address_id | int(11) | YES | MUL | NULL | | | preferences_id | int(11) | YES | MUL | NULL | | | user_id | int(11) | YES | MUL | NULL | | | bounce_score | int(11) | YES | | NULL | | | last_bounce_received | datetime | YES | | NULL | | | last_warning_sent | datetime | YES | | NULL | | | total_warnings_sent | int(11) | YES | | NULL | | +----------------------+--------------+------+-----+---------+----------------+

1 1

Change config to create a new list with all archives off
by Tibor Molnar 21 Apr '24

21 Apr '24

Can archiving be changed anywhere in mailman3 configs, so then all new lists created do not archive by default?

5 8

Mailman 3 hosting query
by Bill Robertson 18 Apr '24

18 Apr '24

Hi I'm investigating the possibility of transferring a fairly small mailing list I own from Google Groups to MM3 via a hosting service. I'd appreciate any pointers regarding what I should be looking out for with companies that offer this service It also seems very hard to find out which hosting companies are worth considering (In particular, reliability and support are important) Any help much is much appreciated Thanks

4 5