Hi together,
as I found some mails in queue/shunt, I thought who should be running them.
On my (standard debian) reference system, there are no cron entries for mailman3 but only for mailman3-web.
Looking in the source code, there is a ./cron/crontab.in.in however references to scripts in port_me/ directory.
Should I (still) install (parts of) the crontab? Or are these functions handled elsewhere?
Also, who picks up mails in queue/shunt? Or are they lost somehow?
(Or did I miss the documentation somewhere?)
Regards, Andi
On 12/31/20 2:49 AM, Andreas Barth wrote:
Hi together,
as I found some mails in queue/shunt, I thought who should be running them.
See below.
On my (standard debian) reference system, there are no cron entries for mailman3 but only for mailman3-web.
The crons for Mailman core are discussed at <https://docs.mailman3.org/en/latest/config-core.html#configuring-cron-jobs>.
Looking in the source code, there is a ./cron/crontab.in.in however references to scripts in port_me/ directory.
That's all old Mailman 2.1 stuff. ignore it.
Should I (still) install (parts of) the crontab? Or are these functions handled elsewhere?
Again, see <https://docs.mailman3.org/en/latest/config-core.html#configuring-cron-jobs>.
Also, who picks up mails in queue/shunt? Or are they lost somehow?
When processing of a message in Core throws an uncaught exception, the
error is logged with a traceback in var/logs/mailman.log and the
message's queue entry is saved in the /var/queue/shunt/ queue. These
messages can be retried via the mailman unshunt command, but unless
the underlying cause has been fixed, they will throw the same exception
and be shunted again.
Each shunted message requires manual inspection of the
var/logs/mailman.log entry and possibly examination of the message via
mailman qfile /var/queue/shunt/....pck.
Some of these are spam messages which are somehow defective causing the exception, and these can just be manually removed from the shunt queue.
I've seen a couple that are transient errors[1] that can just be successfully unshunted.
The rest require some underlying issue to be fixed before they can be successfully unshunted.
[1] In particular, I've seen an apparent race condition in building the list of recipients where we try to determine the delivery status for a member that is no longer a member.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
- Mark Sapiro (mark@msapiro.net) [201231 19:28]:
On 12/31/20 2:49 AM, Andreas Barth wrote:
Looking in the source code, there is a ./cron/crontab.in.in however references to scripts in port_me/ directory.
That's all old Mailman 2.1 stuff. ignore it.
Can we replace that by a mailman3 cron tab? (I'm happy to write that as merge request if useful, but I assume that should be done rather by opening a bug report and using gitlabs features and not inline per mail)
Andi
On 1/1/21 12:37 AM, Andreas Barth wrote:
- Mark Sapiro (mark@msapiro.net) [201231 19:28]:
On 12/31/20 2:49 AM, Andreas Barth wrote:
Looking in the source code, there is a ./cron/crontab.in.in however references to scripts in port_me/ directory.
That's all old Mailman 2.1 stuff. ignore it.
Can we replace that by a mailman3 cron tab? (I'm happy to write that as merge request if useful, but I assume that should be done rather by opening a bug report and using gitlabs features and not inline per mail)
Yes, the ideal workflow is to open an issue on GitLab and then open a merge request to fix it.
However, I don't think we need a sample crontab as we have <https://docs.mailman3.org/en/latest/config-core.html#configuring-cron-jobs> which should be enough.
I think the real solution is to just remove the mailman/cron directory and at least some of the mailman/port_me/ files. The cron/crontab.in.in file is just a distraction.
For the port_me files, we have:
checkdbs.py - this function is handled by mailman notify
disabled.py - handled by core's bounce processing
show_config.py - handled by mailman conf
and these for which no MM 3 equivalent exists and which could be helpful
in implementing a corresponding mailman subcommand:
config_list.py export.py find_member.py
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
- Mark Sapiro (mark@msapiro.net) [210102 01:07]:
I think the real solution is to just remove the mailman/cron directory and at least some of the mailman/port_me/ files. The cron/crontab.in.in file is just a distraction.
Well, or replace it with just a README.rst pointing to the web page (having something called "cron" in the source tree doesn't sound like a mistake to me).
Regards, Andi
Andreas Barth writes:
as I found some mails in queue/shunt, I thought who should be running them.
Mark covered this quite completely, but I do have one remark. In my experience, most shunted emails are spam (that's *not* how we normally handle spam, it's just that spam is far more likely to be sufficiently malformed to cause an uncaught exception than legit mail). You can just delete the corresponding .pck.
Steve
- Stephen J. Turnbull (turnbull.stephen.fw@u.tsukuba.ac.jp) [210101 03:49]:
Andreas Barth writes:
as I found some mails in queue/shunt, I thought who should be running them.
Mark covered this quite completely, but I do have one remark. In my experience, most shunted emails are spam (that's *not* how we normally handle spam, it's just that spam is far more likely to be sufficiently malformed to cause an uncaught exception than legit mail). You can just delete the corresponding .pck.
Thanks to you both.
In my case, this was however Uncaught runner exception: (raised as a result of Query-invoked autoflush; consider using a session.no_autoflush block if this flush is occurring prematurely) (sqlite3.OperationalError) database is locked [SQL: 'UPDATE mailinglist SET last_post_at=?, post_id=? WHERE mailinglist.id = ?'] [parameters: ('2020-12-23 10:18:41.030593', 10, 11)] (Background on this error at: http://sqlalche.me/e/e3q8
The 'fix' for that was turning it off and on again (i.e. rebooting the VM), but now I have a few messages around.
(Perhaps I should also migrate to a real database, but that's a different problem)
Andi
On 1/1/21 12:17 AM, Andreas Barth wrote:
In my case, this was however Uncaught runner exception: (raised as a result of Query-invoked autoflush; consider using a session.no_autoflush block if this flush is occurring prematurely) (sqlite3.OperationalError) database is locked [SQL: 'UPDATE mailinglist SET last_post_at=?, post_id=? WHERE mailinglist.id = ?'] [parameters: ('2020-12-23 10:18:41.030593', 10, 11)] (Background on this error at: http://sqlalche.me/e/e3q8
The 'fix' for that was turning it off and on again (i.e. rebooting the VM), but now I have a few messages around.
Do you mean you have messages in the shunt queue that were shunted because of this error?
If so, just running Mailman's mailman unshunt command should reprocess
and deliver those.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
- Mark Sapiro (mark@msapiro.net) [210102 00:40]:
Do you mean you have messages in the shunt queue that were shunted because of this error?
yes (but my mail was just "I have the rare case where it isn't spam" :) )
If so, just running Mailman's
mailman unshuntcommand should reprocess and deliver those.
Yes, thanks, that's what I already understood from your last mail and did (after removing duplicates).
Thanks for all for your great help.
Andi
participants (3)
-
Andreas Barth -
Mark Sapiro -
Stephen J. Turnbull