Problem with list creation
Hi,
We want to create thousand of lists. We use a simple curl command
curl --user restadmin:restpass -X POST http://localhost:8001/3.1/lists -d fqdn_listname=nameOfTheList@example.com
After creating ~2k of them we started getting errors in Mailman logs:
[2020-09-16 17:11:10 +0200] [4564] [DEBUG] POST /3.1/lists
[2020-09-16 17:11:41 +0200] [1339] [CRITICAL] WORKER TIMEOUT (pid:4564)
[2020-09-16 17:11:41 +0200] [4564] [INFO] Worker exiting (pid: 4564)
[2020-09-16 17:11:41 +0200] [5019] [INFO] Booting worker with pid: 5019
Since then creation fails in most cases.
We increased timeout in mailman.cfg
[gunicorn]
workers: 4
timeout: 900
And we increased timeout in nginx
http {
# MODIF UNICORN
proxy_connect_timeout 300s;
proxy_read_timeout 300s;
# MODIF UNICORN
Where might be the problem? How to start investigating and fix it?
Thanks, Paweł
On 9/16/20 8:25 AM, Pawel Grzywaczewski wrote:
We increased timeout in mailman.cfg
[gunicorn] workers: 4 timeout: 900
This doesn't do it. In order to pass configuration settings to the REST gunicorn servers, you need to put in mailman.cfg
[webservice]
configuration: /path/to/gunicorn.cfg
and then put settings like these
[gunicorn]
workers: 4
timeout: 900
in /path/to/gunicorn.cfg. Some settings, but only hostname, port and workers can be put in mailman.cfg, but in the [webservice] section, not in a [gunicorn] section. See <https://gitlab.com/mailman/mailman/-/blob/master/src/mailman/rest/gunicorn.py#L48> for more detail.
And we increased timeout in nginx
http { # MODIF UNICORN proxy_connect_timeout 300s; proxy_read_timeout 300s; # MODIF UNICORN
nginx is not involved. These are not proxied requests through nginx. They are requests to Mailman's REST API which use a separate gunicorn server.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mark Sapiro writes:
On 9/16/20 8:25 AM, Pawel Grzywaczewski wrote:
We increased timeout in mailman.cfg
This looks like the issue in Shashikanth's post, too: https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/message/... They are also creating more than 1000 lists.
Do we know why these timeouts are happening? (Search for "list creation times out" returns no results in either core or Postorius issues.)
On Thu, Sep 17, 2020, at 4:17 AM, Stephen J. Turnbull wrote:
Mark Sapiro writes:
On 9/16/20 8:25 AM, Pawel Grzywaczewski wrote:
We increased timeout in mailman.cfg
This looks like the issue in Shashikanth's post, too: https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/message/... They are also creating more than 1000 lists.
Do we know why these timeouts are happening? (Search for "list creation times out" returns no results in either core or Postorius issues.)
This is the first time I am actually seeing this. I tried hammering a local server with default number of workers, 2, and created 5000 lists in a loop and went through pretty smooth. I am probably not replicating the exact situation, I am using the sqlite database (which I presumed would be slow and choke up faster but...it didn't).
Creating list isn't one of the slow operations AFAIK, so I am actually surprised that it needs increasing timeout. It could be the interaction with database layer maybe, but I can't be sure without more debugging.
Mailman-users mailing list -- mailman-users@mailman3.org To unsubscribe send an email to mailman-users-leave@mailman3.org https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
-- thanks, Abhilash Raj (maxking)
On Sat, Sep 19, 2020, at 2:39 PM, Pawel Grzywaczewski wrote:
We use Postgres but DB is on remote host.
Do you have some sort of instrumentation in the database layer to get the latency for the requests?
Also, you mentioned that you are creating lists via the curl command, are you doing in sequence via a loop or parallely via multiple threads at a specific requests/second to the Mailman server?
Cheers, Paweł
Mailman-users mailing list -- mailman-users@mailman3.org To unsubscribe send an email to mailman-users-leave@mailman3.org https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
-- thanks, Abhilash Raj (maxking)
Not sure about getting latency for the requests. I will ask.
It's a simple loop (sequence).
Cheers, Paweł
participants (4)
-
Abhilash Raj -
Mark Sapiro -
Pawel Grzywaczewski -
Stephen J. Turnbull