Digests not working correctly
So I seem to have problems with both periodic AND size-triggered digests. Size first, this is simpler. Just got one sent out, clearly triggered by a new email getting added to the queued up messages, but the setting for the list is the default 30KB and the digest came out at 23.2 KB. Is it measuring the accumulated queue as size on disk including metadata and filesystem block sizes or something, rather than the size of what the emailed digest will be?
Periodic is much more complicated, even to explain. I have the cron job running nightly at 11:00 PM and it certainly seems to be working, except for one really odd thing. The list also definitely has periodic digests enabled in the web UI. This is a very bursty list, so it will go mad with traffic for a few weeks and then will go mostly idle for a month or so. The first digest of a burst is definitely size-triggered, but pretty reliably it has the last few messages from the previous burst in there. So the one I got today had three messages from today and one message from May 4th. But that May 4th message should have gone out at 11:00 PM on May 4th with the daily digest and then gotten flushed from the pending digest queue, right?
-- Joel Lord
On 6/4/23 11:45, Joel Lord wrote:
So I seem to have problems with both periodic AND size-triggered digests. Size first, this is simpler. Just got one sent out, clearly triggered by a new email getting added to the queued up messages, but the setting for the list is the default 30KB and the digest came out at 23.2 KB. Is it measuring the accumulated queue as size on disk including metadata and filesystem block sizes or something, rather than the size of what the emailed digest will be?
The threshold is determined by the size in bytes of the MMDF format mailbox file in which the the messages for the digest are saved. This does not include metadata and is not affected by block sizes. It contains the raw messages with all headers and 8 bytes of message separators per message and a 'From ' line per message.
Once the digest is triggered, the messages are configured into a digest message with boilerplate, a TOC and possibly a header and/or footer added. The messages themselves may be added as separate MIME message/rfc822 parts to a mime format digest or added as text to a plain format digest. Many headers are removed as the messages are added. See <https://gitlab.com/mailman/mailman/-/blob/master/src/mailman/config/schema.c...> for what is kept. In addition any non text/plain parts in the message are not included in the plain format digest, but are replaced by notes that they have been removed. All this says that a MIME format may be larger or smaller than the threshold depending on the size of the removed headers vs. the added parts, and a plain text digest will probably be smaller if the messages have non text/plain parts, e.g. if content filtering doesn't collapse alternatives and leaves the text/html alternative part, this part will not be in a plain format digest.
Periodic is much more complicated, even to explain. I have the cron job running nightly at 11:00 PM and it certainly seems to be working, except for one really odd thing. The list also definitely has periodic digests enabled in the web UI. This is a very bursty list, so it will go mad with traffic for a few weeks and then will go mostly idle for a month or so. The first digest of a burst is definitely size-triggered, but pretty reliably it has the last few messages from the previous burst in there. So the one I got today had three messages from today and one message from May 4th. But that May 4th message should have gone out at 11:00 PM on May 4th with the daily digest and then gotten flushed from the pending digest queue, right?
That's correct. Do you ever get periodic digests sent at 11:00 PM? If not, there is some issue either with configuration or something. If you get them sometimes, I don't know what could be wrong.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 6/4/2023 9:25 PM, Mark Sapiro wrote:
On 6/4/23 11:45, Joel Lord wrote:
Thanks for the explanation on size-triggered digests.
Periodic is much more complicated, even to explain. I have the cron job running nightly at 11:00 PM and it certainly seems to be working, except for one really odd thing. The list also definitely has periodic digests enabled in the web UI. This is a very bursty list, so it will go mad with traffic for a few weeks and then will go mostly idle for a month or so. The first digest of a burst is definitely size-triggered, but pretty reliably it has the last few messages from the previous burst in there. So the one I got today had three messages from today and one message from May 4th. But that May 4th message should have gone out at 11:00 PM on May 4th with the daily digest and then gotten flushed from the pending digest queue, right?
That's correct. Do you ever get periodic digests sent at 11:00 PM? If not, there is some issue either with configuration or something. If you get them sometimes, I don't know what could be wrong.
The periodic digests do seem to be coming out. I also now have confirmation that the one message in this morning's digest that was from May 4th was also included in the last digest back on May 4th, so it seems that the one message was left behind in the digest queue when the periodic digest was sent. There was one earlier message to the list back on May 4th, before the one that got duplicated, but I can't tell if that triggered a size-based digest to be sent: the logs aren't clear enough on that detail for me to tell.
Just to inform things:
(venv) lists@host2:~/mailman/venv/bin$ pip freeze | grep mailman django-mailman3==1.3.9 mailman==3.3.8 mailman-hyperkitty==1.2.1 mailman-web==0.0.6 mailmanclient==3.3.5 (venv) lists@host2:~/mailman/venv/bin$ pip freeze | grep hyper mailman-hyperkitty==1.2.1 (venv) lists@host2:~/mailman/venv/bin$ pip freeze | grep post postorius==1.3.8
-- Joel Lord
On 6/4/23 18:35, Joel Lord wrote:
The periodic digests do seem to be coming out. I also now have confirmation that the one message in this morning's digest that was from May 4th was also included in the last digest back on May 4th, so it seems that the one message was left behind in the digest queue when the periodic digest was sent.
I don't see how that can happen. The process that sends a digest renames
the var/lists/<list-id>/digest.mmdf mailbox file in which the messages
are accumulated to var/lists/<list-id>/digest.<volume>.<issue>.mmdf,
where <volume> and <issue> are the volume and issue numbers of that
digest, and then queues a message in the digest queue to tell the
digest runner to create the digest from the messages in that mbox and
send it. Thus, it leaves no var/lists/<list-id>/digest.mmdf mailbox file
behind and that is created anew when the next post arrives. Further, if
there is a non-empty digest.mmdf file, its messages should be sent no
later than the next 11 PM cron digests.
There was one earlier message to the list back on May 4th, before the one that got duplicated, but I can't tell if that triggered a size-based digest to be sent: the logs aren't clear enough on that detail for me to tell.
OK
Just to inform things:
(venv) lists@host2:~/mailman/venv/bin$ pip freeze | grep mailman django-mailman3==1.3.9 mailman==3.3.8 mailman-hyperkitty==1.2.1 mailman-web==0.0.6 mailmanclient==3.3.5 (venv) lists@host2:~/mailman/venv/bin$ pip freeze | grep hyper mailman-hyperkitty==1.2.1
Actually, it's HyperKitty, not hyperkitty, but I assume HyperKitty is up to date as are the others.
(venv) lists@host2:~/mailman/venv/bin$ pip freeze | grep post postorius==1.3.8
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
The May 4th digest that went out was _also_ size-triggered, so this may have nothing to do with periodic digests at all, and possibly my periodic digests aren't working. I'm not on any of my own lists in digest mode, I'm slowly extracting diagnostic information out of people who are. Also, since this is a ~2 month cycle, it's really difficult to get data points to work with. I'll need to remember to go in and look when this settles down again (new cycle of activity started last night) to see if there's anything left pending.
(venv) root@host2:/home/lists/mailman/venv/bin# pip freeze | grep -i hyper HyperKitty==1.3.7
On 6/4/2023 10:05 PM, Mark Sapiro wrote:
On 6/4/23 18:35, Joel Lord wrote:
The periodic digests do seem to be coming out. I also now have confirmation that the one message in this morning's digest that was from May 4th was also included in the last digest back on May 4th, so it seems that the one message was left behind in the digest queue when the periodic digest was sent.
I don't see how that can happen. The process that sends a digest renames the var/lists/<list-id>/digest.mmdf mailbox file in which the messages are accumulated to var/lists/<list-id>/digest.<volume>.<issue>.mmdf, where <volume> and <issue> are the volume and issue numbers of that digest, and then queues a message in the
digestqueue to tell the digest runner to create the digest from the messages in that mbox and send it. Thus, it leaves no var/lists/<list-id>/digest.mmdf mailbox file behind and that is created anew when the next post arrives. Further, if there is a non-empty digest.mmdf file, its messages should be sent no later than the next 11 PMcron digests.There was one earlier message to the list back on May 4th, before the one that got duplicated, but I can't tell if that triggered a size-based digest to be sent: the logs aren't clear enough on that detail for me to tell >
OK
Just to inform things:
(venv) lists@host2:~/mailman/venv/bin$ pip freeze | grep mailman django-mailman3==1.3.9 mailman==3.3.8 mailman-hyperkitty==1.2.1 mailman-web==0.0.6 mailmanclient==3.3.5 (venv) lists@host2:~/mailman/venv/bin$ pip freeze | grep hyper mailman-hyperkitty==1.2.1
Actually, it's HyperKitty, not hyperkitty, but I assume HyperKitty is up to date as are the others.
(venv) lists@host2:~/mailman/venv/bin$ pip freeze | grep post postorius==1.3.8
-- Joel Lord
Now I'm on one of the lists in digest mode and I can see that it's a mess. Periodic digests are definitely NOT working, so I'll lay that out here.
root@host2:/# cat /etc/cron.d/mailman # This goes in /etc/cron.d/mailman # Replace "apache" by your webserver user ("www-data" on Debian systems) and # set the path to the Django project directory 0 23 * * * lists /usr/local/bin/mailman digests --periodic 0 23 * * * lists /usr/local/bin/mailman notify
root@host2:/# grep digests /var/log/cron.log Jun 11 23:00:01 host2 CRON[1632765]: (lists) CMD (/usr/local/bin/mailman digests --periodic) Jun 12 23:00:01 host2 CRON[2177286]: (lists) CMD (/usr/local/bin/mailman digests --periodic)
root@host2:/home/members/directory# su - lists lists@host2:~$ /usr/local/bin/mailman digests --periodic lists@host2:~$ ls var/lists/<list>/ digest.mmdf
In this case I've got /usr/local/bin/mailman as a symlink to the mailman binary inside the venv's bin directory, just for simplicity. That digest.mmdf file is dated June 9th and clearly ought to have been cleared out on any of the nightly runs between then and today but has not. There are no errors anywhere I can find.
How can I try and track this down?
-Joel
On 6/4/2023 10:15 PM, Joel Lord wrote:
The May 4th digest that went out was _also_ size-triggered, so this may have nothing to do with periodic digests at all, and possibly my periodic digests aren't working. I'm not on any of my own lists in digest mode, I'm slowly extracting diagnostic information out of people who are. Also, since this is a ~2 month cycle, it's really difficult to get data points to work with. I'll need to remember to go in and look when this settles down again (new cycle of activity started last night) to see if there's anything left pending.
(venv) root@host2:/home/lists/mailman/venv/bin# pip freeze | grep -i hyper HyperKitty==1.3.7
On 6/4/2023 10:05 PM, Mark Sapiro wrote:
On 6/4/23 18:35, Joel Lord wrote:
The periodic digests do seem to be coming out. I also now have confirmation that the one message in this morning's digest that was from May 4th was also included in the last digest back on May 4th, so it seems that the one message was left behind in the digest queue when the periodic digest was sent.
I don't see how that can happen. The process that sends a digest renames the var/lists/<list-id>/digest.mmdf mailbox file in which the messages are accumulated to var/lists/<list-id>/digest.<volume>.<issue>.mmdf, where <volume> and <issue> are the volume and issue numbers of that digest, and then queues a message in the
digestqueue to tell the digest runner to create the digest from the messages in that mbox and send it. Thus, it leaves no var/lists/<list-id>/digest.mmdf mailbox file behind and that is created anew when the next post arrives. Further, if there is a non-empty digest.mmdf file, its messages should be sent no later than the next 11 PMcron digests.There was one earlier message to the list back on May 4th, before the one that got duplicated, but I can't tell if that triggered a size-based digest to be sent: the logs aren't clear enough on that detail for me to tell >
OK
Just to inform things:
(venv) lists@host2:~/mailman/venv/bin$ pip freeze | grep mailman django-mailman3==1.3.9 mailman==3.3.8 mailman-hyperkitty==1.2.1 mailman-web==0.0.6 mailmanclient==3.3.5 (venv) lists@host2:~/mailman/venv/bin$ pip freeze | grep hyper mailman-hyperkitty==1.2.1
Actually, it's HyperKitty, not hyperkitty, but I assume HyperKitty is up to date as are the others.
(venv) lists@host2:~/mailman/venv/bin$ pip freeze | grep post postorius==1.3.8
-- Joel Lord
On 6/13/23 09:42, Joel Lord wrote:
Now I'm on one of the lists in digest mode and I can see that it's a mess. Periodic digests are definitely NOT working, so I'll lay that out here. ... root@host2:/# grep digests /var/log/cron.log Jun 11 23:00:01 host2 CRON[1632765]: (lists) CMD (/usr/local/bin/mailman digests --periodic) Jun 12 23:00:01 host2 CRON[2177286]: (lists) CMD (/usr/local/bin/mailman digests --periodic)
OK.
root@host2:/home/members/directory# su - lists lists@host2:~$ /usr/local/bin/mailman digests --periodic lists@host2:~$ ls var/lists/<list>/ digest.mmdf
If there is nothing in Mailman's mailman.log, I don't know what the issue is unless the list's digest_send_periodic flag is False.
How can I try and track this down?
Try manually running
/usr/local/bin/mailman digests --periodic --verbose
to get more info. Also try
/usr/local/bin/mailman digests --send --verbose
to see what it reports and whether it sends the digest.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 6/13/23 10:46, Mark Sapiro wrote:
If there is nothing in Mailman's mailman.log, I don't know what the issue is unless the list's digest_send_periodic flag is False.
One more thought. Output from these cron jobs written to stdout and/or
stderr will be emailed. You can control to where with a MAILTO=...
line in the crontab. Otherwise, they will be mailed to the owner of the
crontab, probably list or maybe root. Check your MTA logs from
23:00+ to see if any emails are sent and to where.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Sadly, no cron output emails.
On 6/13/2023 1:53 PM, Mark Sapiro wrote:
On 6/13/23 10:46, Mark Sapiro wrote:
If there is nothing in Mailman's mailman.log, I don't know what the issue is unless the list's digest_send_periodic flag is False.
One more thought. Output from these cron jobs written to stdout and/or stderr will be emailed. You can control to where with a
MAILTO=...line in the crontab. Otherwise, they will be mailed to the owner of the crontab, probablylistor mayberoot. Check your MTA logs from 23:00+ to see if any emails are sent and to where.
-- Joel Lord
I ran the "/usr/local/bin/mailman digests --periodic" again, sadly without the --verbose, and it kicked out the digests as it ought to have the first time. Now I have to wait for some list traffic to come in to "recharge" the digests so I have something to test with again.
-Joel
On 6/13/2023 1:46 PM, Mark Sapiro wrote:
On 6/13/23 09:42, Joel Lord wrote:
Now I'm on one of the lists in digest mode and I can see that it's a mess. Periodic digests are definitely NOT working, so I'll lay that out here. ... root@host2:/# grep digests /var/log/cron.log Jun 11 23:00:01 host2 CRON[1632765]: (lists) CMD (/usr/local/bin/mailman digests --periodic) Jun 12 23:00:01 host2 CRON[2177286]: (lists) CMD (/usr/local/bin/mailman digests --periodic)
OK.
root@host2:/home/members/directory# su - lists lists@host2:~$ /usr/local/bin/mailman digests --periodic lists@host2:~$ ls var/lists/<list>/ digest.mmdf
If there is nothing in Mailman's mailman.log, I don't know what the issue is unless the list's digest_send_periodic flag is False.
How can I try and track this down?
Try manually running
/usr/local/bin/mailman digests --periodic --verboseto get more info. Also try
/usr/local/bin/mailman digests --send --verboseto see what it reports and whether it sends the digest.
-- Joel Lord
On 6/13/23 10:58, Joel Lord wrote:
I ran the "/usr/local/bin/mailman digests --periodic" again, sadly without the --verbose, and it kicked out the digests as it ought to have the first time. Now I have to wait for some list traffic to come in to "recharge" the digests so I have something to test with again.
Even with the --verbose option, it wouldn't have helped much because it worked. What we would really like to see is the output with the --verbose option when it fails.
It is a mystery to me why it would sometimes fail and sometimes not with the same digest.mmdf file.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Finally got a scenario where there was contents queued up for digest delivery and not enough traffic to kick off size based digest volumes. Running "/usr/local/bin/mailman digests --periodic --verbose" ran through all the lists and kicked out digests successfully.
So something is making this work when I run it as the lists user by hand, but fail when I run it from cron as the lists user. Clearly this is on my end, I'll report back if I find anything useful or interesting.
On 6/13/2023 2:14 PM, Mark Sapiro wrote:
On 6/13/23 10:58, Joel Lord wrote:
I ran the "/usr/local/bin/mailman digests --periodic" again, sadly without the --verbose, and it kicked out the digests as it ought to have the first time. Now I have to wait for some list traffic to come in to "recharge" the digests so I have something to test with again.
Even with the --verbose option, it wouldn't have helped much because it worked. What we would really like to see is the output with the --verbose option when it fails.
It is a mystery to me why it would sometimes fail and sometimes not with the same digest.mmdf file.
-- Joel Lord
On 6/23/23 10:52 AM, Joel Lord wrote:
Finally got a scenario where there was contents queued up for digest delivery and not enough traffic to kick off size based digest volumes. Running "/usr/local/bin/mailman digests --periodic --verbose" ran through all the lists and kicked out digests successfully.
So something is making this work when I run it as the lists user by hand, but fail when I run it from cron as the lists user. Clearly this is on my end, I'll report back if I find anything useful or interesting.
If you haven't done so, add the --verbose option to the command in the
crontab, and maybe add a MAILTO= as well so the output goes where you
can find it.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
I'm a list administrator for a list hosted by mailmanlists.net on a server in Germany. Only one member uses the digest option, but she doesn't receive a digest until the Digest size threshold has been exceeded. (I also am subscribed as a member with the Mime Digest option and see the same behavior.) The option for Send Digest Periodically is checked but doesn't trigger a daily digest on the days that there is some activity if the aggregate size doesn't exceed the threshold. I think "periodically" means daily, and the administrator doesn't seem to have any way of checking or changing the periodicity. The hosting service has attempted several fixes, including fiddling with the cron settings, but hasn't found something that works. They even upgraded to ver. 3.3.9 recently, and that didn't fix the problem.
Any suggestions for what I might do or suggest to the hosting service?
On 6/8/24 12:48 PM, Steve Brown wrote:
I'm a list administrator for a list hosted by mailmanlists.net on a server in Germany. Only one member uses the digest option, but she doesn't receive a digest until the Digest size threshold has been exceeded. (I also am subscribed as a member with the Mime Digest option and see the same behavior.) The option for Send Digest Periodically is checked but doesn't trigger a daily digest on the days that there is some activity if the aggregate size doesn't exceed the threshold. I think "periodically" means daily, and the administrator doesn't seem to have any way of checking or changing the periodicity. The hosting service has attempted several fixes, including fiddling with the cron settings, but hasn't found something that works. They even upgraded to ver. 3.3.9 recently, and that didn't fix the problem.
Periodic digests are sent by the mailman digests --periodic command
which should be run daily by cron. That and the mailman notify command
should be run from mailman's crontab. See
https://docs.mailman3.org/en/latest/install/virtualenv.html#setup-cron-jobs
I don't know what "fiddling with the cron settings" means, but if these commands are in mailman's crontab, including
MAILTO=some_human@example.com
at the beginning of the crontab will ensure any diagnostic output is sent to some_human@example.com where it will be seen. Also, putting
LANG=en_US.UTF-8
near the beginning of the crontab may be required in some cases. See the Attention notice at https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/docs/instal...
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
participants (3)
-
Joel Lord -
Mark Sapiro -
Steve Brown