On Mon, Mar 18, 2019, at 9:46 AM, Dmitry Makovey wrote:
Thanks for fast response Abhilash!
See inline for responses:
On 3/18/19 9:24 AM, Abhilash Raj wrote:
On Mon, Mar 18, 2019, at 9:14 AM, Dmitry Makovey wrote:
In the last couple of month we've observed some odd behavior from MM3 (namely the core) where it seems like email that is about to be sent to a large-ish mailing list (10K users) has been processed yet never gets sent out until we restart the core.
This is indeed weird, I don't think 10k is too much, but it also depends on your MTA. How fast do they allow sending emails and if there is any rate limiting.
I'm using Mailu.io for MTA. We're controlling it so there's not much throttling as I've whitelisted mailman to send anything over LMTP.
Mailman sends out email using SMTP and receives it from MTA using LMTP.
I wouldn't necessarily consider this a performance problem in Core, since it is still just processing only one email. All it needs to do is send them all out, which is where the problem should be.
inferred assumption is that mailman-core loses LMTP connection with Mailu stack. but then upon restart of service it regains it?
Mailman acts as am SMTP client sending out emails via MTA. AFAIK, Mailman 3 recycles SMTP connections after certain number of emails are sent out.
You can configure how many emails are sent-out per-connection and when should each connection be recycled. You can also disable the recycling altogether and always keep a connection open to the MTA. See 1 here for detailed docs.
Again, copying from the docs, these are some extra MTA related settings that you may want to change:
# Ceiling on the number of recipients that can be specified in a single SMTP # transaction. Set to 0 to submit the entire recipient list in one # transaction. max_recipients: 500
# Ceiling on the number of SMTP sessions to perform on a single socket # connection. Some MTAs have limits. Set this to 0 to do as many as we like # (i.e. your MTA has no limits). Set this to some number great than 0 and # Mailman will close the SMTP connection and re-open it after this number of # consecutive sessions. max_sessions_per_connection: 0
# Maximum number of simultaneous subthreads that will be used for SMTP # delivery. After the recipients list is chunked according to max_recipients, # each chunk is handed off to the SMTP server by a separate such thread. If # your Python interpreter was not built for threads, this feature is disabled. # You can explicitly disable it in all cases by setting max_delivery_threads # to 0. max_delivery_threads: 0
max_sessions_per_connection is by default set to 0, so that shouldn't be causing any problems.
For a list of 10k members with max_delivery_threads set to to default(500),
Mailman could be sequentially sending out emails in 20 (10k/500) connections, which
could be causing the delay that you see.
You could increase the threads or if your MTA can handle it, set the max_delivery_threads to 20 and see if that helps?
I'd imagine debug logging on outgoing runner should give some more pointers on what is going on.
You can configure the smtp runner to log extra information (copied from docs)1
awesome - I'll give it a go. thanks for the pointer.
I'd also recommend looking into MTA logs for anything un-usual.
we briefly did , but nothing stood out.
troubleshooting/digging before I just restart the core (which by the way takes about 45min to fully start... which is another puzzle to solve at
How do you measure that it takes 45 mins to start?
normally it's "docker ps" command output along with "docker top" so once I see all the sub-processes kicked off I'm assuming core is started and from that point it's anywhere between 35 and 45min before I can get any *response* from core (i.e. mailman-web complains, REST commands fail, etc.)
Maybe we should discuss this in another thread? REST Runner (and other runners) shouldn't be taking this long to boot up.
What database backend are you using?
Postgres. The whole setup is docker-setup (0.1.7 version of containers at this point).
-- Sr System and DevOps Engineer SoM IRT
Attachments:
- signature.asc
-- thanks, Abhilash Raj (maxking)