Problem with digests not being sent daily
Hi,
I have a problem with digests not being sent daily (periodically).
Mailman 3.3.9 installed in virtualenv
Enable Digests = YES Send Digest Periodically = YES Digest size threshold = 0
Cron is set and is running. An error report (below) is generated and emailed.
The last line in the error report says "FileNotFoundError: [Errno 2] No such file or directory ...".
Setting the list's folder to 777 didn't help.
Any clues on where to start looking for a solution would be greatly appreciated.
Thanks, Mark
/////////////////////////////////////// CRON IS SET: /////////////////////////////////////// su mailman (venv) mailman:/$ crontab -e ... 5 6 * * * /opt/mailman/venv/bin/mailman digests --periodic ...
/////////////////////////////////////// CRON RUNS: /////////////////////////////////////// # tail -100 /var/log/cron.log ... 2024-06-21T06:05:01.815948+00:00 deu CRON[3105464]: (mailman) CMD (/opt/mailman/venv/bin/mailman digests --periodic) ...
/////////////////////////////////////// ERROR REPORT EMAILED BY CRON: ///////////////////////////////////////
Traceback (most recent call last): File "/opt/mailman/venv/bin/mailman", line 8, in <module> sys.exit(main()) ^^^^^^ File "/opt/mailman/venv/lib/python3.11/site-packages/click/core.py", line 1157, in __call__ return self.main(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib/python3.11/site-packages/click/core.py", line 1078, in main rv = self.invoke(ctx) ^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib/python3.11/site-packages/mailman/bin/mailman.py", line 69, in invoke return super().invoke(ctx) ^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib/python3.11/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib/python3.11/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, **ctx.params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib/python3.11/site-packages/click/core.py", line 783, in invoke return __callback(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func return f(get_current_context(), *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib/python3.11/site-packages/mailman/commands/cli_digests.py", line 126, in digests maybe_send_digest_now(mlist, force=True) File "/opt/mailman/venv/lib/python3.11/site-packages/mailman/app/digests.py", line 106, in maybe_send_digest_now os.rename(mailbox_path, mailbox_dest) FileNotFoundError: [Errno 2] No such file or directory: '/opt/mailman/mm/var/lists/LISTNAME.LISTDOMAIN/digest.mmdf' -> '/opt/mailman/mm/var/lists/LISTNAME.LISTDOMAIN/digest.252.14.mmdf'
////////////////////////////////////////////////////////////////////////////////
On Fri, Jun 21, 2024 at 9:47 AM Mark <imark@posteo.net> wrote:
Hi,
I have a problem with digests not being sent daily (periodically).
Mailman 3.3.9 installed in virtualenv
Enable Digests = YES Send Digest Periodically = YES Digest size threshold = 0
Cron is set and is running. An error report (below) is generated and emailed.
The last line in the error report says "FileNotFoundError: [Errno 2] No such file or directory ...".
Setting the list's folder to 777 didn't help.
/opt/mailman is owned by the right user:group (or NO??) and so doesn't require to be accessed by others, IMHO, so that move of 777 is just dangerous, if not useless.
Any clues on where to start looking for a solution would be greatly
appreciated.
Thanks, Mark
/////////////////////////////////////// CRON IS SET: /////////////////////////////////////// su mailman (venv) mailman:/$ crontab -e ... 5 6 * * * /opt/mailman/venv/bin/mailman digests --periodic ...
/////////////////////////////////////// CRON RUNS: /////////////////////////////////////// # tail -100 /var/log/cron.log
^^^^^^^^^^^^^^^^^^^^^^^^^^^^ does your MM3 write logs to /var (outside the venv)??
... 2024-06-21T06:05:01.815948+00:00 deu CRON[3105464]: (mailman) CMD (/opt/mailman/venv/bin/mailman digests --periodic) ...
/////////////////////////////////////// ERROR REPORT EMAILED BY CRON: ///////////////////////////////////////
Traceback (most recent call last): File "/opt/mailman/venv/bin/mailman", line 8, in <module> sys.exit(main()) ^^^^^^ File "/opt/mailman/venv/lib/python3.11/site-packages/click/core.py", line 1157, in __call__ return self.main(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib/python3.11/site-packages/click/core.py", line 1078, in main rv = self.invoke(ctx) ^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib/python3.11/site-packages/mailman/bin/mailman.py", line 69, in invoke return super().invoke(ctx) ^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib/python3.11/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib/python3.11/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, **ctx.params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib/python3.11/site-packages/click/core.py", line 783, in invoke return __callback(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func return f(get_current_context(), *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib/python3.11/site-packages/mailman/commands/cli_digests.py",
line 126, in digests maybe_send_digest_now(mlist, force=True) File "/opt/mailman/venv/lib/python3.11/site-packages/mailman/app/digests.py", line 106, in maybe_send_digest_now os.rename(mailbox_path, mailbox_dest) FileNotFoundError: [Errno 2] No such file or directory: *<================================================== Does the two files below exist? Can you show their permissions/ownership?* '/opt/mailman/mm/var/lists/LISTNAME.LISTDOMAIN/digest.mmdf' -> '/opt/mailman/mm/var/lists/LISTNAME.LISTDOMAIN/digest.252.14.mmdf'
I not sure what the answer is, until I go RTFM, but I would not expect the files to exist if there have been no posts to the list.
I hope this: https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/thread/C... can help you think about the issue.
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 In an Internet failure case, the #1 suspect is a constant: DNS. "Oh, the cruft.", egrep -v '^$|^.*#' ¯\_(ツ)_/¯ :-) [How to ask smart questions: http://www.catb.org/~esr/faqs/smart-questions.html]
Setting the list's folder to 777 didn't help.
/opt/mailman is owned by the right user:group (or NO??) and so doesn't require to be accessed by others, IMHO, so that move of 777 is just dangerous, if not useless.
The user:group is mailman:mailman and the list's directory is also owned by mailman:mailman. Setting the directory to 777 was temporary and just an experiment.
///////////////////////////////////////////////////////////////////////////////////////
# tail -100 /var/log/cron.log
does your MM3 write logs to /var (outside the venv)??
Yes. Mailman's cron writes to /var/log/cron.log
///////////////////////////////////////////////////////////////////////////////////////
FileNotFoundError: [Errno 2] No such file or directory: ....
Does the two files below exist? Can you show their permissions/ownership?* '/opt/mailman/mm/var/lists/LISTNAME.LISTDOMAIN/digest.mmdf' -> '/opt/mailman/mm/var/lists/LISTNAME.LISTDOMAIN/digest.252.14.mmdf'
ls -la /opt/mailman/mm/var/lists drwxr-xr-x 2 mailman mailman 4096 Jun 19 19:13 LISTNAME.LISTDOMAIN
ls -la /opt/mailman/mm/var/lists/LISTNAME.LISTDOMAIN drwxr-xr-x 2 mailman mailman 4096 Jun 19 19:13 . drwxr-xr-x 49 mailman mailman 4096 Jun 21 01:24 ..
... but I would not expect the files to exist if there have been no posts to the list.
That's the problem. The mmdf files don't exist in the list's directory although there have been many posts to the list throughout the day.
I hope this: https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/thread/C... can help you think about the issue.
Thanks, but that post talks about ownership and permissions. This is a venv installation as per the Mailman docs, and mailman is the owner of the processes.
On Fri, Jun 21, 2024 at 12:08 PM Mark <imark@posteo.net> wrote:
Setting the list's folder to 777 didn't help.
/opt/mailman is owned by the right user:group (or NO??) and so doesn't require to be accessed by others, IMHO, so that move of 777 is just dangerous, if not useless.
The user:group is mailman:mailman and the list's directory is also owned by mailman:mailman. Setting the directory to 777 was temporary and just an experiment.
///////////////////////////////////////////////////////////////////////////////////////
# tail -100 /var/log/cron.log
does your MM3 write logs to /var (outside the venv)??
Yes. Mailman's cron writes to /var/log/cron.log
///////////////////////////////////////////////////////////////////////////////////////
FileNotFoundError: [Errno 2] No such file or directory: ....
Does the two files below exist? Can you show their permissions/ownership?* '/opt/mailman/mm/var/lists/LISTNAME.LISTDOMAIN/digest.mmdf' -> '/opt/mailman/mm/var/lists/LISTNAME.LISTDOMAIN/digest.252.14.mmdf'
ls -la /opt/mailman/mm/var/lists drwxr-xr-x 2 mailman mailman 4096 Jun 19 19:13 LISTNAME.LISTDOMAIN
ls -la /opt/mailman/mm/var/lists/LISTNAME.LISTDOMAIN drwxr-xr-x 2 mailman mailman 4096 Jun 19 19:13 . drwxr-xr-x 49 mailman mailman 4096 Jun 21 01:24 ..
... but I would not expect the files to exist if there have been no posts to the list.
That's the problem. The mmdf files don't exist in the list's directory although there have been many posts to the list throughout the day.
I am not sure what the problem is, for sure. The files should be created. I am not sure why they are not being created.
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 In an Internet failure case, the #1 suspect is a constant: DNS. "Oh, the cruft.", egrep -v '^$|^.*#' ¯\_(ツ)_/¯ :-) [How to ask smart questions: http://www.catb.org/~esr/faqs/smart-questions.html]
On 6/21/24 02:07, Mark wrote:
That's the problem. The mmdf files don't exist in the list's directory although there have been many posts to the list throughout the day.
Assuming your mailman/app/digests.py is unmodified, there is some race condition. The code beginning at https://gitlab.com/mailman/mailman/-/blob/master/src/mailman/app/digests.py?... finds the file and size is > 0, but by the time it gets to https://gitlab.com/mailman/mailman/-/blob/master/src/mailman/app/digests.py?... the file is gone.
What other crons run at the same time?
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 6/21/24 02:07, Mark wrote:
...The mmdf files don't exist in the list's directory although there have been many posts to the list throughout the day.
On 2024-06-21 15:55, Mark Sapiro wrote:
Assuming your mailman/app/digests.py is unmodified, there is some race condition. The code beginning at https://gitlab.com/mailman/mailman/-/blob/master/src/mailman/app/digests.py?... finds the file and size is > 0, but by the time it gets to https://gitlab.com/mailman/mailman/-/blob/master/src/mailman/app/digests.py?... the file is gone.
What other crons run at the same time?
The digest.py file is out-of-the-box and unmodified.
ls -la /opt/mailman/venv/lib/python3.11/site-packages/mailman/app/ ... -rw-r--r-- 1 mailman mailman 4642 Jan 30 08:06 digests.py
The /var/log/cron.log shows the crons running at the same time as digest at 22:30 hours.
# tail -f cron.log ... 2024-06-21T22:30:01.368407+00:00 deu CRON[3422196]: (root) CMD (sh /templates/mailman_custom_templates.sh > /dev/null) 2024-06-21T22:30:01.369050+00:00 deu CRON[3422197]: (mailman) CMD (/opt/mailman/venv/bin/mailman-web runjobs quarter_hourly) 2024-06-21T22:30:01.371586+00:00 deu CRON[3422198]: (mailman) CMD (/opt/mailman/venv/bin/mailman notify) 2024-06-21T22:30:01.371878+00:00 deu CRON[3422199]: (mailman) CMD (/opt/mailman/venv/bin/mailman digests --periodic) 2024-06-21T22:30:01.372190+00:00 deu CRON[3422202]: (mailman) CMD (/opt/mailman/venv/bin/mailman-web runjobs minutely)
///////////////////////////////////////////////////////////////////////////////////////////////////////////////
The mailman crons are more or less out-of-the-box:
MAILTO=imark@posteo.net 30 22 * * * /opt/mailman/venv/bin/mailman digests --periodic 30 22 * * * /opt/mailman/venv/bin/mailman notify #@daily /opt/mailman/venv/bin/mailman digests --periodic #@daily /opt/mailman/venv/bin/mailman notify #
0,15,30,45 * * * * /opt/mailman/venv/bin/mailman-web runjobs quarter_hourly @hourly /opt/mailman/venv/bin/mailman-web runjobs hourly @daily /opt/mailman/venv/bin/mailman-web runjobs daily @weekly /opt/mailman/venv/bin/mailman-web runjobs weekly @monthly /opt/mailman/venv/bin/mailman-web runjobs monthly @yearly /opt/mailman/venv/bin/mailman-web runjobs yearly/opt/mailman/venv/bin/mailman-web runjobs minutely
The root cron just runs a bash script every 5 minutes to copy custom templates from the mailman and mailmanweb databases and write them to files.
...
On 6/21/24 15:57, Mark wrote:
On 6/21/24 02:07, Mark wrote:
...The mmdf files don't exist in the list's directory although there have been many posts to the list throughout the day.
If you look at some point prior to 22:30, are there
/opt/mailman/mm/var/lists/<list_id>/digests.mmdf files at that time? and
if so, what happins if you run mailman digests --periodic manually at
that point?
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 2024-06-22 00:04, Mark Sapiro wrote:
...The mmdf files don't exist in the list's directory although there have been many posts to the list throughout the day.
If you look at some point prior to 22:30, are there /opt/mailman/mm/var/lists/<list_id>/digests.mmdf files at that time? and if so, what happins if you run
mailman digests --periodicmanually at that point?
I changed the digest and notify crons to 05:00.
0 5 * * * @daily /opt/mailman/venv/bin/mailman digests --periodic 0 5 * * * @daily /opt/mailman/venv/bin/mailman notify
At ~04:50 I posted an email to the list. The message was delivered to members and archived.
Nothing showed up in /opt/mailman/mm/var/lists/<LISTID>
# cd /opt/mailman/mm/var/lists/<LISTID> # ls -la total 8 drwxr-xr-x 2 mailman mailman 4096 Jun 22 04:52 . drwxr-xr-x 6 mailman mailman 4096 Mar 25 21:32 ..
Then, as "mailman", I ran "mailman digests --periodic" manually and got the same error (as expected since there is nothing there) ...
$ /opt/mailman/venv/bin/mailman digests --periodic Traceback (most recent call last): File "/opt/mailman/venv/bin/mailman", line 8, in <module> sys.exit(main()) ^^^^^^ File "/opt/mailman/venv/lib/python3.11/site-packages/click/core.py", line 1157, in __call__ return self.main(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib/python3.11/site-packages/click/core.py", line 1078, in main rv = self.invoke(ctx) ^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib/python3.11/site-packages/mailman/bin/mailman.py", line 69, in invoke return super().invoke(ctx) ^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib/python3.11/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib/python3.11/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, **ctx.params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib/python3.11/site-packages/click/core.py", line 783, in invoke return __callback(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func return f(get_current_context(), *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib/python3.11/site-packages/mailman/commands/cli_digests.py", line 126, in digests maybe_send_digest_now(mlist, force=True) File "/opt/mailman/venv/lib/python3.11/site-packages/mailman/app/digests.py", line 106, in maybe_send_digest_now os.rename(mailbox_path, mailbox_dest) FileNotFoundError: [Errno 2] No such file or directory: '/opt/mailman/mm/var/lists/<LISTID>/digest.mmdf' -> '/opt/mailman/mm/var/lists/<LISTID>/digest.6.3.mmdf'
On 6/21/24 22:04, Mark wrote:
At ~04:50 I posted an email to the list. The message was delivered to members and archived.
Nothing showed up in /opt/mailman/mm/var/lists/<LISTID>
Probably because (in Postorius) Settings->Digest->Enable Digests is No
Then, as "mailman", I ran "mailman digests --periodic" manually and got the same error (as expected since there is nothing there) ...
$ /opt/mailman/venv/bin/mailman digests --periodic Traceback (most recent call last): ... FileNotFoundError: [Errno 2] No such file or directory: '/opt/mailman/mm/var/lists/<LISTID>/digest.mmdf' -> '/opt/mailman/mm/var/lists/<LISTID>/digest.6.3.mmdf'
This error should not occur if there is no digests.mmdf. It only does occur because you have set Digest size threshold to 0 under the assumption that 0 means unlimited which it doesn't. If you don't want digests sent on size, you need to set Digest size threshold to a very large number.
I've created https://gitlab.com/mailman/mailman/-/issues/1158 to avoid throwing the exception, but setting the threshold to 0 is still wrong as it will cause a digest to be sent on every post.
If the Enable Digests setting is Yes, then the lack of digests needs more investigation.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 6/22/24 17:51, Mark Sapiro wrote:
If the Enable Digests setting is Yes, then the lack of digests needs more investigation.
OTOH, do you know digests are not being sent? With digests enabled and the threshhold set to 0, on each post a digest.mmdf will be created and then immediately moved to digest.<vol>.<num>.mmdf and an entry queued for digest runner to create the digest(s) from that file and remove it, so within probably a few seconds /opt/mailman/mm/var/lists/<LISTID>/ will be empty again.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 6/22/24 17:51, Mark Sapiro wrote:
With digests enabled and the threshhold set to 0, on each post a digest.mmdf will be created and then immediately moved to digest.<vol>.<num>.mmdf and an entry queued for digest runner to create the digest(s) from that file and remove it, so within probably a few seconds /opt/mailman/mm/var/lists/<LISTID>/ will be empty again.
Digests are enabled and the threshold = 0.
Is it possible to temporarily disable the digest runner so that I can see if a post is actually creating the digest.mmdf file (before the runner removes it)?
On 6/22/24 19:54, Mark wrote:
Is it possible to temporarily disable the digest runner so that I can see if a post is actually creating the digest.mmdf file (before the runner removes it)?
Find its PID and sigterm (kill -term) it. It is possible to restart it
with the runner command, but it is safer to just do mailman stop
followed by mailman start (mailman restart will not restart a
sigtermed runner).
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 2024-06-23 00:51, Mark Sapiro wrote:
Nothing showed up in /opt/mailman/mm/var/lists/<LISTID>
Probably because (in Postorius) Settings->Digest->Enable Digests is No
Double checked that Enabled Digest s= Yes
Then, as "mailman", I ran "mailman digests --periodic" manually and got the same error (as expected since there is nothing there) ...
This error should not occur if there is no digests.mmdf. It only does occur because you have set Digest size threshold to 0 under the assumption that 0 means unlimited which it doesn't. If you don't want digests sent on size, you need to set Digest size threshold to a very large number.
I've created https://gitlab.com/mailman/mailman/-/issues/1158 to avoid throwing the exception, but setting the threshold to 0 is still wrong
Oh? I was indeed assuming that 0 meant unlimited. The threshold was set at 0.
My thinking was that meant the file size was not a factor in determining when the digest was sent and that the sending would be triggered by the daily cron.
When I set the threshold very low - 10kB - it sends a digest on every post.
So my understanding of what you are saying is: if the threshold is set very large, say 100,000 kB - then a digest will be be sent by the daily cron even if it is smaller than 100,000 kB.
Am I understanding correctly?
On 6/22/24 20:04, Mark wrote:
Oh? I was indeed assuming that 0 meant unlimited. The threshold was set at 0.
My thinking was that meant the file size was not a factor in determining when the digest was sent and that the sending would be triggered by the daily cron.
If Send Digest Periodically is Yes, digests will be sent by the daily (or whatever schedule) cron. If a digest reaches threshold in between runnings of the cron, one will be sent at that time.
When I set the threshold very low - 10kB - it sends a digest on every post.
This is expected and the same should occur if the threshold is 0.
So my understanding of what you are saying is: if the threshold is set very large, say 100,000 kB - then a digest will be be sent by the daily cron even if it is smaller than 100,000 kB. Correct. If Send Digest Periodically is Yes, any accumulated digest will be sent when
mailman digests --periodicruns, regardless of size.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 2024-06-23 03:24, Mark Sapiro wrote:
On 6/22/24 20:04, Mark wrote:
Oh? I was indeed assuming that 0 meant unlimited. The threshold was set at 0.
My thinking was that meant the file size was not a factor in determining when the digest was sent and that the sending would be triggered by the daily cron.
So my understanding of what you are saying is: if the threshold is set very large, say 100,000 kB - then a digest will be be sent by the daily cron even if it is smaller than 100,000 kB.
Correct. If Send Digest Periodically is Yes, any accumulated digest will be sent when
mailman digests --periodicruns, regardless of size.
Mark - Thank you very much for setting me straight on this. I had been stumbling around in the dark for days.
With the threshold set extremely high, the daily digests are on track and list members are happy again.
Best, Mark
participants (3)
-
Mark -
Mark Sapiro -
Odhiambo Washington