REST Runner (and other runners) taking long time to boot up
splitting this discussion into it's own thread from "MM3 occasionally chokes (?)".
Re-iterating background info for posterity:
we have been experiencing increasingly long boot ups from mailman-core over the past year to the point where it's at about 45min for the sequence to complete. We run couple of mailing lists (20+) that are anywhere between 100 and 10K+ members and most of them are low traffic.
From what I can tell all that time is spent doing some sort of indexing operations, I'm writing from memory since I can't restart the system to confirm that.
sizes of directories that containers use: 3.6G core 352M database 285M web
state of my queues (as far as I understand how to evaluate them:
$ for q in core/var/queue/*; do echo ${q}; ls ${q}/ | wc -l ; done core/var/queue/archive 1 core/var/queue/bad 3845 core/var/queue/bounces 0 core/var/queue/command 0 core/var/queue/digest 0 core/var/queue/in 0 core/var/queue/nntp 0 core/var/queue/out 7 core/var/queue/pipeline 0 core/var/queue/retry 1 core/var/queue/shunt 958 core/var/queue/virgin 0
what kind of tuning should we be looking into to speed up our boot-ups?
-- Sr System and DevOps Engineer SoM IRT
On Mon, Mar 18, 2019, at 10:46 AM, Dmitry Makovey wrote:
splitting this discussion into it's own thread from "MM3 occasionally chokes (?)".
Re-iterating background info for posterity:
we have been experiencing increasingly long boot ups from mailman-core over the past year to the point where it's at about 45min for the sequence to complete. We run couple of mailing lists (20+) that are anywhere between 100 and 10K+ members and most of them are low traffic.
From what I can tell all that time is spent doing some sort of indexing operations, I'm writing from memory since I can't restart the system to confirm that.
Can you go back and check the logs from the last time you re-started?
Also, can you check the output of docker logs mailman-core
and
docker logs mailman-web
to see what happens before you see
"Rest Runner was started" or similar message?
I wonder if it is waiting for Postgres to come up.
Core shouldn't be doing any indexing. We compiles the static files during bootup, but that should take no more than few seconds.
sizes of directories that containers use: 3.6G core 352M database 285M web
state of my queues (as far as I understand how to evaluate them:
$ for q in core/var/queue/*; do echo ${q}; ls ${q}/ | wc -l ; done core/var/queue/archive 1 core/var/queue/bad 3845 core/var/queue/bounces 0 core/var/queue/command 0 core/var/queue/digest 0 core/var/queue/in 0 core/var/queue/nntp 0 core/var/queue/out 7 core/var/queue/pipeline 0 core/var/queue/retry 1 core/var/queue/shunt 958 core/var/queue/virgin 0
what kind of tuning should we be looking into to speed up our boot-ups?
-- Sr System and DevOps Engineer SoM IRT
Attachments:
- signature.asc
-- thanks, Abhilash Raj (maxking)
participants (2)
-
Abhilash Raj
-
Dmitry Makovey