February 2024 - Mailman-users - lists.mailman3.org

Issues with a DMARC record leading to message being shunted
by Ralf Hildebrandt 27 Jul '24

27 Jul '24

Hi! Recently a mail was not being distributed via our MM3 instance; I searched the logs and found: Dec 22 04:54:27 2023 (24) ACCEPT: <1d431542-b386-467b-8bf2-305f19ee7eb4(a)somenet.org> Dec 22 04:54:28 2023 (28) Uncaught runner exception: list index out of range Dec 22 04:54:28 2023 (28) Traceback (most recent call last): File "/usr/lib/python3.10/site-packages/mailman/core/runner.py", line 179, in _one_iteration self._process_one_file(msg, msgdata) File "/usr/lib/python3.10/site-packages/mailman/core/runner.py", line 272, in _process_one_file keepqueued = self._dispose(mlist, msg, msgdata) File "/usr/lib/python3.10/site-packages/mailman/runners/pipeline.py", line 37, in _dispose process(mlist, msg, msgdata, pipeline) File "/usr/lib/python3.10/site-packages/mailman/core/pipelines.py", line 53, in process handler.process(mlist, msg, msgdata) File "/usr/lib/python3.10/site-packages/mailman/handlers/validate_authenticity.py", line 125, in process authenticate(msg, msgdata) File "/usr/lib/python3.10/site-packages/mailman/utilities/retry.py", line 44, in f_retry return f(*args, **kwargs) File "/usr/lib/python3.10/site-packages/mailman/handlers/validate_authenticity.py", line 93, in authenticate auth_result = authenticate_message( File "/usr/lib/python3.10/site-packages/authheaders/__init__.py", line 395, in authenticate_message dmarc_result = check_dmarc(msg, spf_result, dkim_result, dnsfunc=dnsfunc, psddmarc=psddmarc) File "/usr/lib/python3.10/site-packages/authheaders/__init__.py", line 343, in check_dmarc result, result_comment, from_domain, policy = dmarc_per_from(from_domain, spf_result, dkim_result, dnsfunc, psddmarc) File "/usr/lib/python3.10/site-packages/authheaders/__init__.py", line 90, in dmarc_per_from record, orgdomain = receiver_record(from_domain) File "/usr/lib/python3.10/site-packages/authheaders/dmarc_lookup.py", line 117, in receiver_record retval = lookup_receiver_record(hostSansDmarc, dnsfunc) File "/usr/lib/python3.10/site-packages/authheaders/dmarc_lookup.py", line 92, in lookup_receiver_record tags = answer_to_dict(str(result)) File "/usr/lib/python3.10/site-packages/authheaders/dmarc_lookup.py", line 42, in answer_to_dict retval = {t[0].strip().lower(): t[1].strip().lower() for t in rawTags} File "/usr/lib/python3.10/site-packages/authheaders/dmarc_lookup.py", line 42, in <dictcomp> retval = {t[0].strip().lower(): t[1].strip().lower() for t in rawTags} IndexError: list index out of range Dec 22 04:54:28 2023 (28) SHUNTING: 1703220868.1418982+b27f5db154375fcd2c418d467172e4aad0d3d57a So the message has been shunted, due to this error. Something with DMARC and tags. So I checked the DMARC record for the sending domain somenet.org. It is "v=DMARC1;p=none;pct=100;rua=mailto:postmaster@somenet.org;mailto:postmaster@somenet.org;ri=3600;fo=1;" which is syntactically incorrect. The extra ";mailto:postmaster@somenet.org" is wrong, I guess the tag "ruf=" is missing here! It should probably read "v=DMARC1;p=none;pct=100;rua=mailto:postmaster@somenet.org;ruf=mailto:postmaster@somenet.org;ri=3600;fo=1;" So, should mm3 somehow "catch" this error somehow? According to the DMARC specs, an syntactically incorrect DMARC record should be ignored / handled as if none were present. -- Ralf Hildebrandt Charité - Universitätsmedizin Berlin Geschäftsbereich IT | Abteilung Netz | Netzwerk-Administration Invalidenstraße 120/121 | D-10115 Berlin Tel. +49 30 450 570 155 ralf.hildebrandt(a)charite.de https://www.charite.de

4 11

Digests not working correctly
by Joel Lord 08 Jun '24

08 Jun '24

So I seem to have problems with both periodic AND size-triggered digests. Size first, this is simpler. Just got one sent out, clearly triggered by a new email getting added to the queued up messages, but the setting for the list is the default 30KB and the digest came out at 23.2 KB. Is it measuring the accumulated queue as size on disk including metadata and filesystem block sizes or something, rather than the size of what the emailed digest will be? Periodic is much more complicated, even to explain. I have the cron job running nightly at 11:00 PM and it certainly seems to be working, except for one really odd thing. The list also definitely has periodic digests enabled in the web UI. This is a very bursty list, so it will go mad with traffic for a few weeks and then will go mostly idle for a month or so. The first digest of a burst is definitely size-triggered, but pretty reliably it has the last few messages from the previous burst in there. So the one I got today had three messages from today and one message from May 4th. But that May 4th message should have gone out at 11:00 PM on May 4th with the daily digest and then gotten flushed from the pending digest queue, right? -- Joel Lord

3 14

Change config to create a new list with all archives off
by Tibor Molnar 21 Apr '24

21 Apr '24

Can archiving be changed anywhere in mailman3 configs, so then all new lists created do not archive by default?

5 8

Mailman 3 hosting query
by Bill Robertson 18 Apr '24

18 Apr '24

Hi I'm investigating the possibility of transferring a fairly small mailing list I own from Google Groups to MM3 via a hosting service. I'd appreciate any pointers regarding what I should be looking out for with companies that offer this service It also seems very hard to find out which hosting companies are worth considering (In particular, reliability and support are important) Any help much is much appreciated Thanks

4 5

mailman3 digest sending questions and also adding docker to the mix
by bob B 17 Apr '24

17 Apr '24

So I have been down the digest rabbit hole. So to confirm my assumptions/findings: Digest size threshold---- means when the total size of held messages hits this setting, a digest is sent Digest Volume Frequency. ---- specifies when the digest volume number gets incremented for digest messages, IT DOES NOT send out digests based on the setting, daily, weekly ... So what does the GUI setting "Send Digest Periodically yes no" do? how periodically does it send digests, if this is set to yes? If I want to send a digest out for a list every day, or every week, how do I set that, especially in the docker environment? Is this something that can be configured in the GUI and configured by each of my list admins? I found that ”'mailman digests --periodic' via cron” should be set in googling, but does that mean I have to do a cron job for any list that wants a scheduled digest sent? My use case is, that I have a list admin who set a size threshold, but regardless of that, they also want their list digest sent out daily, they thought it was the "Digest Volume Frequency" setting but that is not correct. However in their testing they are not getting daily digest, they are getting digests only when the threshold is reached, BUT they have "Send Digest Periodically" set to "yes" and "Enable Digests" is set to "yes"

7 17

hyperkitty.lib.utils Failed to update the fulltext index
by Ben Harris 13 Apr '24

13 Apr '24

hello there, i migrated to a new host last night and i've been getting this error from a cronjob mail every minute. anyone have any ideas what would cause this? ERROR 2019-04-17 14:26:33,056 27205 hyperkitty.lib.utils Failed to update the fulltext index: ord() expected a character, but string of length 0 found Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/hyperkitty/lib/utils.py", line 186, in run_with_lock fn(*args, **kwargs) File "/usr/lib/python2.7/dist-packages/hyperkitty/search_indexes.py", line 87, in update_index update_cmd.update_backend("hyperkitty", "default") File "/usr/lib/python2.7/dist-packages/haystack/management/commands/update_index.py", line 276, in update_backend last_max_pk=max_pk) File "/usr/lib/python2.7/dist-packages/haystack/management/commands/update_index.py", line 97, in do_update backend.update(index, current_qs, commit=commit) File "/usr/lib/python2.7/dist-packages/haystack/backends/whoosh_backend.py", line 178, in update self.setup() File "/usr/lib/python2.7/dist-packages/haystack/backends/whoosh_backend.py", line 126, in setup self.index = self.storage.open_index(schema=self.schema) File "/usr/lib/python2.7/dist-packages/whoosh/filedb/filestore.py", line 176, in open_index return indexclass(self, schema=schema, indexname=indexname) File "/usr/lib/python2.7/dist-packages/whoosh/index.py", line 421, in __init__ TOC.read(self.storage, self.indexname, schema=self._schema) File "/usr/lib/python2.7/dist-packages/whoosh/index.py", line 632, in read check_size("int", _INT_SIZE) File "/usr/lib/python2.7/dist-packages/whoosh/index.py", line 626, in check_size sz = stream.read_varint() File "/usr/lib/python2.7/dist-packages/whoosh/filedb/structfile.py", line 191, in read_varint return read_varint(self.read) File "/usr/lib/python2.7/dist-packages/whoosh/util/varints.py", line 102, in read_varint b = ord(readfn(1)) thanks, ben

5 8

Error -- postorius Un-handled exception
by Mark 12 Apr '24

12 Apr '24

Hi All, I've just installed MM3 on a fresh Debian 12 using the Virtualenv method. (venv)...$ mailman info GNU Mailman 3.3.9 (Tom Sawyer) Python 3.11.2 (main, Mar 13 2023, 12:18:29) [GCC 12.2.0] Postorius Version 1.3.10 After creating a couple of lists, the lists are shown in the table on the "Mailing lists" webpage as they should, but clicking on any of the listnames returns an error page: Something went wrong HTTP Error 500: {"title": "500 Internal Server Error"} And the Mailman log is a mystery to me, and no error is reported in the Nginx log ... # tail -f /opt/mailman/web/logs/mailmanweb.log ERROR 2024-01-24 05:41:46,327 51799 postorius Un-handled exception: HTTP Error 500: {"title": "500 Internal Server Error"} Traceback (most recent call last): File "/opt/mailman/venv/lib/python3.11/site-packages/django/core/handlers/base.py", line 197, in _get_response response = wrapped_callback(request, *callback_args, **callback_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib/python3.11/site-packages/django/views/generic/base.py", line 103, in view return self.dispatch(request, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib/python3.11/site-packages/postorius/views/generic.py", line 76, in dispatch return super(MailingListView, self).dispatch(request, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib/python3.11/site-packages/django/views/generic/base.py", line 142, in dispatch return handler(request, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib/python3.11/site-packages/postorius/views/list.py", line 359, in get and 'hyperkitty' in archivers # noqa: W504 ^^^^^^^^^^^^^^^^^^^^^^^^^ File "<frozen _collections_abc>", line 780, in __contains__ File "/opt/mailman/venv/lib/python3.11/site-packages/mailmanclient/restbase/base.py", line 146, in __getitem__ return self._get(key) ^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib/python3.11/site-packages/mailmanclient/restbase/base.py", line 88, in _get return self.rest_data[key] ^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib/python3.11/site-packages/mailmanclient/restbase/base.py", line 74, in rest_data response, content = self._connection.call(self._url) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib/python3.11/site-packages/mailmanclient/restbase/connection.py", line 160, in call raise HTTPError(params.get('url'), response.status_code, urllib.error.HTTPError: HTTP Error 500: {"title": "500 Internal Server Error"} ERROR 2024-01-24 05:41:46,333 51799 django.request Internal Server Error: /postorius/lists/testing.example.com/ ERROR 2024-01-24 05:41:46,333 51799 django.request Internal Server Error: /postorius/lists/testing.example.com/ Could anyone steer me in the right direction for fixing this? Thanks, Mark

6 20

export/import list?
by Dave McGuire 01 Mar '24

01 Mar '24

I'm migrating to a new Mailman3 instance, finally escaping a fairly old (3.1.1) and problematic distribution repo install on a Linux VM and moving to a venv install in a native SmartOS Zone. I've gotten everything running well and integrated with our (fairly complex) mail server. I'd like to begin migration of the lists on the old mm3 instance. Is there a way to export a list from one Mailman 3 instance and import it into another, on a different machine? Ideally some way to capture its settings, members, members' settings, etc? Thanks, -Dave -- Dave McGuire, AK4HZ New Kensington, PA

6 9

Migration from old server to new server failed
by Helio Loureiro 01 Mar '24

01 Mar '24

Hi, So we tried to the migration following the steps: - stop mailman in the old machine (v3.1.1 from ubuntu packages) - dump DB mailman3 and mailman3web - backup /var/lib/mailman3 content On the new machine: - checkout from git the version 3.1.1 to match production and install the services - stop mailman3 and mailman3-web on the new machine - load DBs into new machine - copy content from /var/lib/mailman3 into /local/mailman/var (which means: archives/ cache/ data/ ext/ lists/ locks/ messages/ public_suffix_list.dat queue/ templates/ web/) - start mailman3 and mailman-3web - run mailman3-web migrate And I stopped here. Because it crashed. (venv) mailman@new-mailman3-server ~ (v3.1.1) [0|SIGINT]> mailman-web migrate System check identified some issues: WARNINGS: account.EmailAddress: (models.W036) MariaDB does not support unique constraints with conditions. HINT: A constraint won't be created. Silence this warning if you don't care about it. account.EmailAddress: (models.W043) MariaDB does not support indexes on expressions. HINT: An index won't be created. Silence this warning if you don't care about it. Operations to perform: Apply all migrations: account, admin, auth, contenttypes, django_mailman3, django_q, hyperkitty, postorius, sessions, sites, socialaccount Running migrations: Applying account.0003_alter_emailaddress_create_unique_verified_email... OK Applying account.0004_alter_emailaddress_drop_unique_email... OK Applying account.0005_emailaddress_idx_upper_email... OK Applying admin.0003_logentry_add_action_flag_choices... OK Applying auth.0009_alter_user_last_name_max_length... OK Applying auth.0010_alter_group_name_max_length... OK Applying auth.0011_update_proxy_permissions... OK Applying auth.0012_alter_user_first_name_max_length... OK Applying django_mailman3.0003_sessions... OK Applying django_q.0010_auto_20200610_0856... OK Applying django_q.0011_auto_20200628_1055... OK Applying django_q.0012_auto_20200702_1608... OK Applying django_q.0013_task_attempt_count... OK Applying django_q.0014_schedule_cluster... OK Applying hyperkitty.0016_auto_20180309_0056... OK Applying hyperkitty.0017_file_attachments... OK Applying hyperkitty.0018_threadcategory_color... OK Applying hyperkitty.0019_auto_20190127_null_description... OK Applying hyperkitty.0020_auto_20190907_1927... OK Applying hyperkitty.0021_add_owners_mods...Traceback (most recent call last): File "/local/mailman/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 87, in _execute return self.cursor.execute(sql) File "/local/mailman/venv/lib/python3.10/site-packages/django/db/backends/mysql/base.py", line 75, in execute return self.cursor.execute(query, args) File "/local/mailman/venv/lib/python3.10/site-packages/MySQLdb/cursors.py", line 179, in execute res = self._query(mogrified_query) File "/local/mailman/venv/lib/python3.10/site-packages/MySQLdb/cursors.py", line 330, in _query db.query(q) File "/local/mailman/venv/lib/python3.10/site-packages/MySQLdb/connections.py", line 261, in query _mysql.connection.query(self, query) MySQLdb.OperationalError: (1050, "Table 'hyperkitty_mailinglist_moderators' already exists") The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/local/mailman/venv/bin/mailman-web", line 8, in <module> sys.exit(main()) File "/local/mailman/venv/lib/python3.10/site-packages/mailman_web/manage.py", line 90, in main execute_from_command_line(sys.argv) File "/local/mailman/venv/lib/python3.10/site-packages/django/core/management/__init__.py", line 446, in execute_from_command_line utility.execute() File "/local/mailman/venv/lib/python3.10/site-packages/django/core/management/__init__.py", line 440, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/local/mailman/venv/lib/python3.10/site-packages/django/core/management/base.py", line 402, in run_from_argv self.execute(*args, **cmd_options) File "/local/mailman/venv/lib/python3.10/site-packages/django/core/management/base.py", line 448, in execute output = self.handle(*args, **options) File "/local/mailman/venv/lib/python3.10/site-packages/django/core/management/base.py", line 96, in wrapped res = handle_func(*args, **kwargs) File "/local/mailman/venv/lib/python3.10/site-packages/django/core/management/commands/migrate.py", line 349, in handle post_migrate_state = executor.migrate( File "/local/mailman/venv/lib/python3.10/site-packages/django/db/migrations/executor.py", line 135, in migrate state = self._migrate_all_forwards( File "/local/mailman/venv/lib/python3.10/site-packages/django/db/migrations/executor.py", line 167, in _migrate_all_forwards state = self.apply_migration( File "/local/mailman/venv/lib/python3.10/site-packages/django/db/migrations/executor.py", line 252, in apply_migration state = migration.apply(state, schema_editor) File "/local/mailman/venv/lib/python3.10/site-packages/django/db/migrations/migration.py", line 130, in apply operation.database_forwards( File "/local/mailman/venv/lib/python3.10/site-packages/django/db/migrations/operations/fields.py", line 108, in database_forwards schema_editor.add_field( File "/local/mailman/venv/lib/python3.10/site-packages/django/db/backends/mysql/schema.py", line 104, in add_field super().add_field(model, field) File "/local/mailman/venv/lib/python3.10/site-packages/django/db/backends/base/schema.py", line 636, in add_field return self.create_model(field.remote_field.through) File "/local/mailman/venv/lib/python3.10/site-packages/django/db/backends/base/schema.py", line 447, in create_model self.execute(sql, params or None) File "/local/mailman/venv/lib/python3.10/site-packages/django/db/backends/base/schema.py", line 199, in execute cursor.execute(sql, params) File "/local/mailman/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers( File "/local/mailman/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers return executor(sql, params, many, context) File "/local/mailman/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 84, in _execute with self.db.wrap_database_errors: File "/local/mailman/venv/lib/python3.10/site-packages/django/db/utils.py", line 91, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/local/mailman/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 87, in _execute return self.cursor.execute(sql) File "/local/mailman/venv/lib/python3.10/site-packages/django/db/backends/mysql/base.py", line 75, in execute return self.cursor.execute(query, args) File "/local/mailman/venv/lib/python3.10/site-packages/MySQLdb/cursors.py", line 179, in execute res = self._query(mogrified_query) File "/local/mailman/venv/lib/python3.10/site-packages/MySQLdb/cursors.py", line 330, in _query db.query(q) File "/local/mailman/venv/lib/python3.10/site-packages/MySQLdb/connections.py", line 261, in query _mysql.connection.query(self, query) django.db.utils.OperationalError: (1050, "Table 'hyperkitty_mailinglist_moderators' already exists") I tried to drop the table, but then it complained about another one. I dropped all the complained tables and as result hypperkitty never started. So I went back to see why the first error happened. Any clue what to do next? Best Regards, Helio Loureiro https://helio.loureiro.eng.br https://github.com/helioloureiro https://mastodon.social/@helioloureiro

3 8

Google Summer of Code 2024
by Stephen J. Turnbull 26 Feb '24

26 Feb '24

Hi, everybody It's that time of year again! PSF is in GSoC 2024, I'm 99% certain PSF will accept Mailman as a suborg (don't ask about the 1%, it's embarrassing ;-). Student applications open on March 18. So ... If you have ideas for improving Mailman 3, whether misfeatures that need fixing or cool new features, please consult our ideas page https://wiki.list.org/DEV/Google%20Summer%20of%20Code%202024 to get an idea of level of detail and style of description needed (and to see if maybe it's already there. You can send them to me if they're already formed in your mind, or send them here for input from users on requirements, or to mailman-developers(a)python.org for input on implementation feasibility and appropriateness for GSoC. If you know anybody who would be interested in participating as a contributor, please encourage them to look at us. Rules are somewhat relaxed from years ago. They still don't accept below college age, but they no longer restrict to students. They're also offering multiple period lengths from 8 to 22 weeks (stipends vary according to period and by price level in country of residence, see https://developers.google.com/open-source/gsoc/help/student-stipends). They're not as generous as a few years ago, at least for the "rich" countries (US, UK, Japan). You don't need to be an experienced coder or project manager to be a secondary mentor, just willing to communicate with the code contributor. If you're interested in being a primary or secondary mentor, the burden is likely to be "as much as you want". We rarely get more than one good application, and we have enough "default" primary mentors to cover the likely maximum number of slots we could be allocated. It would be very helpful to us and to the PSF organization administrators if you express interest and get signed up by the end of next week, but as far as I know we can add or substitute mentors at any time. Potential mentors should 1. Have a Google account (needed for step 2, and required by GSoC) 2. Sign up with Python-GSoC at https://forms.gle/coJYxhor6FAZcZfU8 3. If you have been a GSoC mentor or org admin in the past, go to https://summerofcode.withgoogle.com and accept the contributor agreement and program rules for 2024. (Definitely if you did last year, probably for 2022 the system still remembers you, and unlikely it has more memory than that.) Steve

1 0