July 2022 - Mailman-users - lists.mailman3.org

Problem with database change sqlite -> mysql
by moritz both July 7, 2022

July 7, 2022

Greetings, I am totally new to this list and relatively new to mailman3 and came here because I have a problem I would like to solve soon. I hope somebody has a clue: I am running a really small list site with 10 or so lists, all of them have low traffic and no more than ~70 members. The lists are running on a shared server at uberspace. I followed their guide at https://lab.uberspace.de/guide_mailman-3/ Mailman Core Version GNU Mailman 3.3.5 (Tom Sawyer) Mailman Core API Version 3.1 Mailman Core Python Version 3.8.13 (default, May 19 2022, 15:06:19) [GCC 9.3.1 20200408 (Red Hat 9.3.1-2)] Since my migration from mm2 to mm3 I see what I consider high loads and excessive i/o and memory usage. My guess is that the main problem is the database backend for mailman, posterious and hyperkitty, which is sqlite. The database is >200MB at the moment (I have not found out why it is so huge, I tried to delete the hyperkitty archives from there and no longer archive messages, but maybe I have not managed to delete all). I take any recommandation for this situation. My current idea is changing the database backend to something else, i.e. mysql. I tried to follow this guide https://www.rockandnull.com/django-dumpdata/ but the "loaddata" command fails. The stack trace is below. Unfortunatelly, I have close to no knowledge of django and don't know why the Email matching query not exists. Will postgres make things easier? Thanks in advance for any help. Moritz Here is the stacktrace: Traceback (most recent call last): File "/home/daneben/.local/lib/python3.8/site-packages/django/db/models/fields/related_descriptors.py", line 173, in __get__ rel_obj = self.field.get_cached_value(instance) File "/home/daneben/.local/lib/python3.8/site-packages/django/db/models/fields/mixins.py", line 15, in get_cached_value return instance._state.fields_cache[cache_name] KeyError: 'starting_email' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "manage.py", line 10, in <module> execute_from_command_line(sys.argv) File "/home/daneben/.local/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line utility.execute() File "/home/daneben/.local/lib/python3.8/site-packages/django/core/management/__init__.py", line 413, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/home/daneben/.local/lib/python3.8/site-packages/django/core/management/base.py", line 354, in run_from_argv self.execute(*args, **cmd_options) File "/home/daneben/.local/lib/python3.8/site-packages/django/core/management/base.py", line 398, in execute output = self.handle(*args, **options) File "/home/daneben/.local/lib/python3.8/site-packages/django/core/management/commands/loaddata.py", line 78, in handle self.loaddata(fixture_labels) File "/home/daneben/.local/lib/python3.8/site-packages/django/core/management/commands/loaddata.py", line 123, in loaddata self.load_label(fixture_label) File "/home/daneben/.local/lib/python3.8/site-packages/django/core/management/commands/loaddata.py", line 190, in load_label obj.save(using=self.using) File "/home/daneben/.local/lib/python3.8/site-packages/django/core/serializers/base.py", line 223, in save models.Model.save_base(self.object, using=using, raw=True, **kwargs) File "/home/daneben/.local/lib/python3.8/site-packages/django/db/models/base.py", line 763, in save_base pre_save.send( File "/home/daneben/.local/lib/python3.8/site-packages/django/dispatch/dispatcher.py", line 180, in send return [ File "/home/daneben/.local/lib/python3.8/site-packages/django/dispatch/dispatcher.py", line 181, in <listcomp> (receiver, receiver(signal=self, sender=sender, **named)) File "/home/daneben/.local/lib/python3.8/site-packages/hyperkitty/signals.py", line 96, in Thread_on_pre_save kwargs["instance"].on_pre_save() File "/home/daneben/.local/lib/python3.8/site-packages/hyperkitty/models/thread.py", line 157, in on_pre_save self.find_starting_email() File "/home/daneben/.local/lib/python3.8/site-packages/hyperkitty/models/thread.py", line 149, in find_starting_email if self.starting_email is not None: File "/home/daneben/.local/lib/python3.8/site-packages/django/db/models/fields/related_descriptors.py", line 187, in __get__ rel_obj = self.get_object(instance) File "/home/daneben/.local/lib/python3.8/site-packages/django/db/models/fields/related_descriptors.py", line 307, in get_object return super().get_object(instance) File "/home/daneben/.local/lib/python3.8/site-packages/django/db/models/fields/related_descriptors.py", line 154, in get_object return qs.get(self.field.get_reverse_related_filter(instance)) File "/home/daneben/.local/lib/python3.8/site-packages/django/db/models/query.py", line 435, in get raise self.model.DoesNotExist( hyperkitty.models.email.DoesNotExist: Problem installing fixture '/home/daneben/mailman-suite/datadump.json': Email matching query does not exist.

3 5

Django errors after deleting a thread
by peter＠chubb.wattle.id.au July 6, 2022

July 6, 2022

After deleting a thread via the web interface (because it's spam), I get lots of messages from hyperkitty E.g. Internal Server Error: /hyperkitty/list/list@domain/top-threads DoesNotExist at /hyperkitty/list/list@domain/top-threads Thread matching query does not exist. Request Method: GET Request URL: https://example.com/hyperkitty/list/list@domain/top-threads Django Version: 2.2.17 Python Executable: /usr/bin/uwsgi-core Python Version: 3.8.6 Python Path: ['.', '', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] Server time: Wed, 25 Nov 2020 07:12:00 +1100 Installed Applications: ('hyperkitty', 'postorius', 'django_mailman3', 'django.contrib.admin', 'django.contrib.auth', 'django.contrib.contenttypes', 'django.contrib.sessions', 'django.contrib.sites', 'django.contrib.messages', 'django.contrib.staticfiles', 'rest_framework', 'django_gravatar', 'compressor', 'haystack', 'django_extensions', 'django_q', 'allauth', 'allauth.account', 'allauth.socialaccount') Installed Middleware: ('django.contrib.sessions.middleware.SessionMiddleware', 'django.middleware.common.CommonMiddleware', 'django.middleware.csrf.CsrfViewMiddleware', 'django.middleware.locale.LocaleMiddleware', 'django.contrib.auth.middleware.AuthenticationMiddleware', 'django.contrib.messages.middleware.MessageMiddleware', 'django.middleware.clickjacking.XFrameOptionsMiddleware', 'django.middleware.security.SecurityMiddleware', 'django_mailman3.middleware.TimezoneMiddleware', 'postorius.middleware.PostoriusMiddleware') Traceback: File "/usr/lib/python3/dist-packages/django/core/handlers/exception.py" in inner 34. response = get_response(request) File "/usr/lib/python3/dist-packages/django/core/handlers/base.py" in _get_response 115. response = self.process_exception_by_middleware(e, request) File "/usr/lib/python3/dist-packages/django/core/handlers/base.py" in _get_response 113. response = wrapped_callback(request, *callback_args, **callback_kwargs) File "/usr/lib/python3/dist-packages/hyperkitty/lib/view_helpers.py" in inner 135. return func(request, *args, **kwargs) File "/usr/lib/python3/dist-packages/hyperkitty/views/mlist.py" in overview_top_threads 220. 'threads': mlist.top_threads, File "/usr/lib/python3/dist-packages/hyperkitty/models/mailinglist.py" in top_threads 168. return self.cached_values["top_threads"]() File "/usr/lib/python3/dist-packages/hyperkitty/models/common.py" in __call__ 59. return self.get_or_set(*args, **kwargs) File "/usr/lib/python3/dist-packages/hyperkitty/models/mailinglist.py" in get_or_set 376. return [Thread.objects.get(pk=pk) for pk in thread_ids] File "/usr/lib/python3/dist-packages/hyperkitty/models/mailinglist.py" in <listcomp> 376. return [Thread.objects.get(pk=pk) for pk in thread_ids] File "/usr/lib/python3/dist-packages/django/db/models/manager.py" in manager_method 82. return getattr(self.get_queryset(), name)(*args, **kwargs) File "/usr/lib/python3/dist-packages/django/db/models/query.py" in get 406. raise self.model.DoesNotExist( Exception Type: DoesNotExist at /hyperkitty/list/devel(a)sel4.systems/top-threads Exception Value: Thread matching query does not exist. Request information: USER: AnonymousUser GET: No GET data POST: No POST data FILES: No FILES data COOKIES: No cookie data

3 4

mailman3 suddenly stopped working after routine package upgrade with "apt update" and "apt upgrade".
by gordon＠dickens.com July 5, 2022

July 5, 2022

My mailman3 installation is installed in Debian using the Mailman Suite's Virtualenv (venv) instructions (not the Debian mailman3 packages) and has been working great for several months until I recently executed a routine package upgrade with "apt update" and "apt upgrade" after which the mailman3 service quit working with the following error dialog in response to "systemctl start mailman3" # systemctl start mailman3 Job for mailman3.service failed because the control process exited with error code. See "systemctl status mailman3.service" and "journalctl -xe" for details. root@slc:/opt/mailman/venv/bin# systemctl status mailman3 ● mailman3.service - GNU Mailing List Manager Loaded: loaded (/etc/mailman3/mailman3.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Tue 2022-07-05 10:31:55 EDT; 4s ago Process: 3011826 ExecStart=/opt/mailman/venv/bin/mailman start (code=exited, status=2) CPU: 822ms Jul 05 10:31:54 slc.mailhub4u.com systemd[1]: Starting GNU Mailing List Manager... Jul 05 10:31:55 slc.mailhub4u.com mailman[3011826]: Usage: mailman start [OPTIONS] Jul 05 10:31:55 slc.mailhub4u.com mailman[3011826]: Try 'mailman start -h' for help. Jul 05 10:31:55 slc.mailhub4u.com mailman[3011826]: Error: A previous run of GNU Mailman did not exit cleanly (stale_lock). Try using --force Jul 05 10:31:55 slc.mailhub4u.com systemd[1]: mailman3.service: Control process exited, code=exited, status=2/INVALIDARGUMENT Jul 05 10:31:55 slc.mailhub4u.com systemd[1]: mailman3.service: Failed with result 'exit-code'. Jul 05 10:31:55 slc.mailhub4u.com systemd[1]: Failed to start GNU Mailing List Manager. Then, the command, "systemctl status mailman3" results in: # systemctl status mailman3 ● mailman3.service - GNU Mailing List Manager Loaded: loaded (/etc/mailman3/mailman3.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Tue 2022-07-05 10:31:55 EDT; 19min ago Process: 3011826 ExecStart=/opt/mailman/venv/bin/mailman start (code=exited, status=2) CPU: 822ms Jul 05 10:31:54 slc.mailhub4u.com systemd[1]: Starting GNU Mailing List Manager... Jul 05 10:31:55 slc.mailhub4u.com mailman[3011826]: Usage: mailman start [OPTIONS] Jul 05 10:31:55 slc.mailhub4u.com mailman[3011826]: Try 'mailman start -h' for help. Jul 05 10:31:55 slc.mailhub4u.com mailman[3011826]: Error: A previous run of GNU Mailman did not exit cleanly (stale_lock). Try using --force Jul 05 10:31:55 slc.mailhub4u.com systemd[1]: mailman3.service: Control process exited, code=exited, status=2/INVALIDARGUMENT Jul 05 10:31:55 slc.mailhub4u.com systemd[1]: mailman3.service: Failed with result 'exit-code'. Jul 05 10:31:55 slc.mailhub4u.com systemd[1]: Failed to start GNU Mailing List Manager. The only reason that I believe that the apt package upgrade caused the problem is because everything stopped working immediately after the package upgrade commands, "apt update" and "apt upgrade" were executed. Looking at the apt logs, the following packages were apparently upgraded: Start-Date: 2022-07-01 13:18:17 Commandline: apt upgrade Requested-By: gordon (1000) Install: linux-image-5.10.0-15-amd64:amd64 (5.10.120-1, automatic) Upgrade: tzdata:amd64 (2021a-1+deb11u3, 2021a-1+deb11u4), linux-image-amd64:amd64 (5.10.113-1, 5.10.120-1), libssl-dev:amd64 (1.1.1n-0+deb11u2, 1.1.1n-0+deb11u3), libssl-doc:amd64 (1.1.1n-0+deb11u2, 1.1.1n-0+deb11u3), libssl1.1:amd64 (1.1.1n-0+deb11u2, 1.1.1n-0+deb11u3), openssl:amd64 (1.1.1n-0+deb11u2, 1.1.1n-0+deb11u3), linux-libc-dev:amd64 (5.10.113-1, 5.10.120-1) End-Date: 2022-07-01 13:19:04 Start-Date: 2022-07-01 13:21:18 Commandline: apt autoremove Requested-By: gordon (1000) Remove: linux-image-5.10.0-13-amd64:amd64 (5.10.106-1) End-Date: 2022-07-01 13:21:22 Otherwise, other than this package upgrade command, nothing changed on my system between the time that mailman3 was working properly and when it stopped working. Does anyone see any potential problems with any of the packages that were upgraded? The logfile, /var/log/syslog, has the following lines immediately after the "systemctl start mailman3" command was executed: Jul 5 11:07:41 slc mailman[3030034]: Usage: mailman start [OPTIONS] Jul 5 11:07:41 slc mailman[3030034]: Try 'mailman start -h' for help. Jul 5 11:07:41 slc mailman[3030034]: Error: A previous run of GNU Mailman did not exit cleanly (stale_lock). Try using --force Jul 5 11:07:41 slc mariadbd[859]: 2022-07-05 11:07:41 1744540 [Warning] Aborted connection 1744540 to db: 'mailman3' user: 'mailman' host: 'localhost' (Got an error reading communication packets) Jul 5 11:07:41 slc systemd[1]: mailman3.service: Control process exited, code=exited, status=2/INVALIDARGUMENT Jul 5 11:07:41 slc systemd[1]: mailman3.service: Failed with result 'exit-code'. As an FYI, I have attempted to start the service using --force, however, it made no difference. BTW, I don't know if this is significant or not but the following error message (or similar message) is displayed often in the logs: slc mariadbd[859]: 2022-07-05 11:07:41 1744540 [Warning] Aborted connection 1744540 to db: 'mailman3' user: 'mailman' host: 'localhost' (Got an error reading communication packets) In any event, does anyone see any other potential causes for why the mailman3 service will not run? Thanks very much! Gordon

2 2

Monthly mailman-web job using a lot of memory
by Andy Smith July 4, 2022

July 4, 2022

Hi, Is it normal for the monthly job: @monthly www-data [ -f /usr/bin/django-admin ] && flock -n /var/run/mailman3-web/cron.monthly /usr/share/mailman3-web/manage.py runjobs monthly to use nearly 3½GiB of memory? PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 770203 www-data 20 0 4516756 3.4g 1.6g R 99.7 43.5 12:08.00 python3 /usr/share/mailman3-web/manage.py runjo+ I assume it's updating the full text indexes or something. There are around 250k messages in our archives. In the job list I see: appname - jobname - when - help -------------------------------------------------------------------------------- hyperkitty - update_and_clean_index - monthly - Update the full-text index and clean old entries so I'm guessing it's that, but there is also: hyperkitty - update_index - hourly - Update the full-text index which hasn't been noitceably a problem so I wonder why the "and clean old entries" bit is so much more heavyweight, what exactly it dopes and if there's any way to make it lighter on resources? Thanks, Andy

3 7

mailman3 + postfix + dovecot + postfixadmin
by Geoff Nordli July 3, 2022

July 3, 2022

Hi. Did anyone figure out the issue or have a working configuration with this: https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/message… I am getting the same error, with the same configuration. Issue: Postfix LMTP does not pass messages onto MM3:LMTP. Instead the behavior summary is as follows: Jul 3 11:56:30 mail1 postfix/lmtp[824956]: 5168B20017A: to=<lmtp:[127.0.0.1]:8024@domain.com>, orig_to=<testing(a)domain.com>, relay=mail1.domain.com[private/dovecot-lmtp], delay=0.08, delays=0.03/0.01/0.02/0.01, dsn=5.1.1, status=bounced (host mail1.domain.com[private/dovecot-lmtp] said: 550 5.1.1 <"lmtp:[127.0.0.1]:8024"@domain.com> User doesn't exist: lmtp:[127.0.0.1]:8024@domain.com (in reply to RCPT TO command)) thanks, Geoff

2 2

Emails about moderation are sent to list owners, want them sent to moderators instead
by Elias Rudberg July 2, 2022

July 2, 2022

Hello, This is about GNU Mailman 3.3.3 (Tom Sawyer). For each list there are both "owners" and "moderators". As I understand it, the idea is that owners have the power to change configuration for the list, while moderators have more limited power, they can only moderate but not change configuration. When an email is sent from a non-member to a list that requires moderation in such a case, an email with subject like "[listname] post from [sender] requires approval" is sent to the list owners. I would like those emails to be sent to moderators instead. The reason why I want it to work that way is that list owners sometimes want to delegate the work of moderating to the moderators. List owners then want to keep the power to change list configuration, but they do not want to be bothered by individual moderation emails. Is there a way to configure mailman3 to make it send moderation emails to moderators instead of owners? Or, have I misunderstood the purpose of "owners" and "moderators"? If so, I would be grateful if someone could explain. / Elias

2 1