On 12/28/24 05:19, monochromec via Mailman-users wrote:
Following the root cause analysis which Tobias started all those weeks ago we (the admin team behind the installation) are still struggling with the following phenomenon: messages on average take more than 24 hours to be processed, more precisely, the average lifetime of a pickled message object in
/var/lib/mailman3/queue/pipelineclocks in at around 26 hours).
This issue is with Mailman core's pipeline runner. See https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/docs/8-mile... for an overview of message handling and queues in Mailman core.
It seems that the pipeline runner(s) are taking an inordinately long time to process a message through the pipeline of handlers causing the pipeline queue to become backlogged.
Couple of stats of the installation: standard installation from Bookworm OS repos, Hyperkitty as archiver and Postorius as web frontend as explained above, running Python 3.11.2 from the standard systemd service as packaged with the Bookworm deb file. All backends (Core + Hyperkitty) are supported by Postgres version 15+248. The MTA is a standard Postfix installation, again from OS packages.
The primary support for Debian packages is Debian. See https://wiki.list.org/x/12812344
The underlying VM has 7 cores with just under 24 GB of main memory. This production instance is handling less than 130 mailing list (MLs) with an average of less than 10 postings per day per ML. CPU core utilisation hovers around 50% with the lion share allocated to the four pipeline runners as part of the MM configuration.
This is an average of less than 1 message per minute. Processing a message through the pipeline should take nowhere near a minute. Possibly there is an issue in the rfc_2369.py handler. This handler communicates with HyperKitty via mailman_hyperkitty to obtain values for the List-Archive: and Archived-At: headers added to the message. Possibly there is some configuration issue with mailman_hyperkitty that causes these queries to take a long time and possibly eventually time out.
Are there any Connection to HyperKitty failed:, HyperKitty failure on ... or Invalid response from HyperKitty type messages in mailman.log?
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan