First: Thank you all for your input! I hope that we will get down to the root of the problem.
I've checked all cronjobs and there is no cronjob that starts at midnight. All jobs are running later. So i looked into /var/log/messages. The earliest entry there is
Jul 31 23:56:16 iris-s01 bash[324554]: Jul 31 23:56:16 2025 (324554) Starting new HTTPS connection (1): lists.example.com:443 Jul 31 23:56:16 iris-s01 bash[324553]: Jul 31 23:56:16 2025 (324553) Starting new HTTPS connection (1): lists.example.com:443 Jul 31 23:56:16 iris-s01 bash[324556]: Jul 31 23:56:16 2025 (324556) Starting new HTTPS connection (1): lists.example.com:443 Jul 31 23:56:16 iris-s01 bash[324554]: Jul 31 23:56:16 2025 (324554) Starting new HTTP connection (1): localhost:8000 Jul 31 23:56:16 iris-s01 bash[324553]: Jul 31 23:56:16 2025 (324553) Starting new HTTP connection (1): localhost:8000 Jul 31 23:56:16 iris-s01 bash[324556]: Jul 31 23:56:16 2025 (324556) Starting new HTTP connection (1): localhost:8000 Jul 31 23:56:16 iris-s01 bash[324554]: Jul 31 23:56:16 2025 (324554) http://localhost:8000 "GET /archives/api/mailman/urls?mlist=rzcluster%40lists.example.com&msgid=c6140b6e649f46ee8a97aec5c5b80584%40sub.example.com HTTP/11" 200 126 Jul 31 23:56:16 iris-s01 bash[324553]: Jul 31 23:56:16 2025 (324553) http://localhost:8000 "GET /archives/api/mailman/urls?mlist=ww10%40lists.example.com&msgid=1f86b244cce9479490b4bffe184f6c03%40sub.example.com HTTP/11" 200 121 Jul 31 23:56:16 iris-s01 bash[324554]: Jul 31 23:56:16 2025 (324554) Starting new HTTPS connection (1): lists.example.com:443 Jul 31 23:56:16 iris-s01 bash[324556]: Jul 31 23:56:16 2025 (324556) http://localhost:8000 "GET /archives/api/mailman/urls?mlist=fg-informatik-ma-wiss%40lists.example.com&msgid=914dbace411247d7983b5dff8fa34bda%40i3.informatik.rwth-aachen.de HTTP/11" 200 138 Jul 31 23:56:16 iris-s01 bash[324553]: Jul 31 23:56:16 2025 (324553) Starting new HTTPS connection (1): lists.example.com:443 Jul 31 23:56:16 iris-s01 bash[324556]: Jul 31 23:56:16 2025 (324556) Starting new HTTPS connection (1): lists.example.com:443 Jul 31 23:56:16 iris-s01 bash[324553]: Jul 31 23:56:16 2025 (324553) Starting new HTTP connection (1): localhost:8000 Jul 31 23:56:16 iris-s01 bash[324553]: Jul 31 23:56:16 2025 (324553) http://localhost:8000 "GET /archives/api/mailman/urls?mlist=ww10%40lists.example.com&msgid=bd99715d619744699ddf806fd3bd67b1%40sub.example.com HTTP/11" 200 121 Aug 1 00:02:48 iris-s01 bash[567713]: Aug 01 00:02:48 2025 (567713) Starting new HTTPS connection (1): lists.example.com:443 Aug 1 00:02:48 iris-s01 bash[567713]: Aug 01 00:02:48 2025 (567713) Starting new HTTP connection (1): localhost:8000 Aug 1 00:02:48 iris-s01 bash[567713]: Aug 01 00:02:48 2025 (567713) http://localhost:8000 "GET /archives/api/mailman/urls?mlist=ww10%40lists.example.com&msgid=1f86b244cce9479490b4bffe184f6c03%40sub.example.com HTTP/11" 200 121 Aug 1 00:02:48 iris-s01 bash[567713]: Aug 01 00:02:48 2025 (567713) Starting new HTTPS connection (1): lists.example.com:443 Aug 1 00:02:48 iris-s01 bash[567715]: Aug 01 00:02:48 2025 (567715) Starting new HTTP connection (1): localhost:8000 Aug 1 00:02:48 iris-s01 bash[567715]: Aug 01 00:02:48 2025 (567715) http://localhost:8000 "GET /archives/api/mailman/urls?mlist=ww10%40lists.example.com&msgid=1f86b244cce9479490b4bffe184f6c03%40sub.example.come HTTP/11" 200 121 Aug 1 00:02:48 iris-s01 bash[567715]: Aug 01 00:02:48 2025 (567715) Starting new HTTPS connection (1): lists.example.com:443 Aug 1 00:02:48 iris-s01 bash[567712]: Aug 01 00:02:48 2025 (567712) Starting new HTTPS connection (1): lists.example.com:443 Aug 1 00:02:48 iris-s01 bash[567712]: Aug 01 00:02:48 2025 (567712) Starting new HTTP connection (1): localhost:8000 Aug 1 00:02:48 iris-s01 bash[567712]: Aug 01 00:02:48 2025 (567712) http://localhost:8000 "GET /archives/api/mailman/urls?mlist=ww10%40lists.example.com&msgid=1f86b244cce9479490b4bffe184f6c03%40sub.example.com HTTP/11" 200 121 Aug 1 00:02:48 iris-s01 bash[567712]: Aug 01 00:02:48 2025 (567712) Starting new HTTPS connection (1): lists.example.com:443 Aug 1 00:02:49 iris-s01 bash[567712]: Aug 01 00:02:49 2025 (567712) Starting new HTTP connection (1): localhost:8000 Aug 1 00:02:49 iris-s01 bash[567712]: Aug 01 00:02:49 2025 (567712) http://localhost:8000 "GET /archives/api/mailman/urls?mlist=ww10%40lists.example.com&msgid=1f86b244cce9479490b4bffe184f6c03%40sub.example.com HTTP/11" 200 121
And then there is a logrotate:
/var/log/mailman/mailman-logs/* { missingok daily compress delaycompress nomail notifempty rotate 14 dateext su mailman mailman olddir /var/log/mailman/mailman-logs/oldlogs create 664 mailman mailman postrotate if [ -f /opt/mailman/var/master.pid ]; then /bin/systemctl restart mailman.service > /dev/null; fi endscript } /var/log/mailman/* { missingok daily compress delaycompress nomail notifempty rotate 14 dateext su mailman mailman olddir /var/log/mailman/oldlogs create 664 mailman mailman }
And yes, this restarts mailman at midnight. Maybe i should optimize the logrotate.
Now to the files:
mailman qfile /opt/mailman/var/queue/virgin/1736550035.6163204+472f81ece5e45a2651a4499bef418f611b43c619.pck.tmp
in object 1 there is a complete mail from 11 January 2025, it's a digest.
In object 2:
<----- start object 2 -----> { '_parsemsg': False, 'isdigest': True, 'listid': '<Name of the List>>', 'recipients': { '<Rec1>', '<Rec2>', '<Rec3>', '<Rec4>'}, 'version': 3}
So i would say there is no error in the mail. But i will delete this file now, because it's older then 6 months.
Now let's move on to the shunt queue: Here is the fole form the 31 July 2025:
mailman qfile /opt/mailman/var/queue/shunt/1753912822.2651796+28eceef7e18eb70393377b88dc7117af8f9362a0.pck
object 1 is empty object 2:
{ '_parsemsg': False, 'digest_number': 1, 'digest_path': '/opt/mailman/var/lists/it-besteller.lists.example.com/digest.11.1.mmdf', 'lang': 'de', 'listid': 'it-besteller.lists.example.com', 'version': 3, 'volume': 11, 'whichq': 'digest'}
Well - this is a digest too. Maybe the restart through the logrotate kills the digest generation?
After looking at all midnights mails: It's the smae: No Mail body and a digest.
Then we have a special mail from 31 July 2025 at 11:44:
mailman qfile /opt/mailman/var/queue/shunt/1753955094.900898+67bc76525412da66a7c76363f65f583989716305.pck <Database line omitted> [----- start pickle -----] <----- start object 1 ----->
<----- start object 2 -----> { '_parsemsg': False, 'digest_number': 6, 'digest_path': '/opt/mailman/var/lists/slcm_fak.lists.example.com/digest.47.6.mmdf', 'lang': 'en', 'listid': 'slcm_fak.lists.example.com', 'version': 3, 'volume': 47, 'whichq': 'digest'} [----- end pickle -----]
Again - an digest. Seems to be a problem with digests. But that does not explain my problem. It's 'just another problem to be fixed' :D
Questions about my setup:
- my local Postfix is connected to a smarthost. In the maillog i can see successfully delivered mails to a lot of people. If there is a problem, i would see it
Maybe there is something problematic with my mailman.cfg?
# For example, uncomment the following lines to run Mailman in developer mode. # # [devmode] # enabled: yes # recipient: your.address@your.domain
[mailman] site_owner: mailman@lists.example.com layout: here pending_request_life: 3d default_language: de
[paths.here] var_dir: /opt/mailman/var log_dir: /var/log/mailman/mailman-logs
[database] class: mailman.database.postgresql.PostgreSQLDatabase url: postgresql://mailman:<Pass>@db01.example.com/mailman debug: no
[runner.out] instances: 4
[archiver.hyperkitty] class: mailman_hyperkitty.Archiver enable: yes configuration: /opt/mailman/mailman-hyperkitty.cfg
[archiver.prototype] enable: yes
[mta] verp_probes: yes verp_confirmations: yes verp_personalized_deliveries: yes verp_delivery_interval: 0 remove_dkim_headers: yes
[webservice] hostname: localhost port: 8001 use_https: no admin_user: mailmanapiuser admin_pass: <PASS> api_version: 3.1 workers: 4 configuration: /opt/mailman/gunicorn.conf
#log_dir: $var_dir/logs [logging.root] level: debug path: mailman.log [logging.archiver] level: debug path: archiver.log [logging.bounce] path: bounce.log level: warn [logging.config] level: debug path: mailman.log [logging.database] level: warn path: database.log [logging.debug] path: debug.log level: warn [logging.error] level: debug path: error.log [logging.fromusenet] level: warn path: mailman.log [logging.http] level: warn path: httpd.log [logging.locks] level: warn path: locks.log [logging.mischief] level: warn path: mailman.log [logging.plugins] path: mailman.log level: warn [logging.runner] level: warn path: runner.log [logging.smtp] path: smtp.log level: debug [logging.subscribe] path: subscribe.log level: debug [logging.vette] path: vette.log level: debug
# Some list posts and mail to the -owner address may contain DomainKey or # DomainKeys Identified Mail (DKIM) signature headers <http://www.dkim.org/>. # Various list transformations to the message such as adding a list header or # footer or scrubbing attachments or even reply-to munging can break these # signatures. It is generally felt that these signatures have value, even if # broken and even if the outgoing message is resigned. However, some sites # may wish to remove these headers by setting this to 'yes'.
[bounces] # How often should the bounce runner process queued detected bounces? register_bounces_every: 15m
[antispam] header_checks: X-Spam-Flag: YES jump_chain: discard
There are no memory peaks on the os side, as far as my monitoring tells me.
So - any other ideas? Hopefully i didn't missed an important detail in your questions.