On Thu, Apr 9, 2020, at 3:25 AM, skenny@scss.tcd.ie wrote:

Thanks for that Abhilash. I've since figured out how to add and remove nonmembers using the REST API.

Regarding the web interface, I wonder could you comment on the following: I've created a test list and I have added a large number - 1,800 - of nonmembers to it. I find that if I try to delete a nonmember via the web interface, it seems to hang, and then the following message appears:

Something went wrong
Mailman REST API not available. Please start Mailman core.

The same operation succeeds on other lists, where the number of nonmembers is not large. Do you think this could be an issue of scale?

Could be, I am not sure which operation is it stuck at. Can you check logs to see where is it breaking?

Can you try doing the same with the REST API, just to verify if it is the Core which has the issue or it is the web frontend.

Abhilash

Thanks Stephen

---

Mailman-users mailing list -- mailman-users@mailman3.org To unsubscribe send an email to mailman-users-leave@mailman3.org https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/

-- thanks, Abhilash Raj (maxking)

On Thu, Apr 9, 2020, at 1:57 PM, skenny@scss.tcd.ie wrote:

Abhilash Raj wrote:

...

Could be, I am not sure which operation is it stuck at. Can you check logs to see where is it breaking?

Sure. I'm doing all this under Docker; below is what I see in /opt/mailman-web-data/logs/mailmanweb.log when I try to delete a nonmember via the web interface. Thediagnostics appear 30 seconds after I click on 'delete':

It looks like you are looking at the Mailman-web logs.

From the looks of it, the client side is basically timing out after 30 seconds. Same seems to be true for the script you wrote (which is calling the REST API underneath using mailmanclient), which is trying to fetch the list of non-members.

Can you also look at the server side of the logs? It should be available at /opt/mailman/core/var/logs/mailman.log if you are using the standard docker-compose file from the repository.

Abhilash

ERROR 2020-04-09 20:35:51,482 19 postorius.middleware Mailman REST API not available Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 672, in urlopen chunked=chunked, File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 421, in _make_request six.raise_from(e, None) File "<string>", line 3, in raise_from File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 416, in _make_request httplib_response = conn.getresponse() File "/usr/lib/python3.6/http/client.py", line 1346, in getresponse response.begin() File "/usr/lib/python3.6/http/client.py", line 307, in begin version, status, reason = self._read_status() File "/usr/lib/python3.6/http/client.py", line 276, in _read_status raise RemoteDisconnected("Remote end closed connection without" http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 449, in send timeout=timeout File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 720, in urlopen method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] File "/usr/lib/python3.6/site-packages/urllib3/util/retry.py", line 400, in increment raise six.reraise(type(error), error, _stacktrace) File "/usr/lib/python3.6/site-packages/urllib3/packages/six.py", line 734, in reraise raise value.with_traceback(tb) File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 672, in urlopen chunked=chunked, File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 421, in _make_request six.raise_from(e, None) File "<string>", line 3, in raise_from File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 416, in _make_request httplib_response = conn.getresponse() File "/usr/lib/python3.6/http/client.py", line 1346, in getresponse response.begin() File "/usr/lib/python3.6/http/client.py", line 307, in begin version, status, reason = self._read_status() File "/usr/lib/python3.6/http/client.py", line 276, in _read_status raise RemoteDisconnected("Remote end closed connection without" urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/mailmanclient/restbase/connection.py", line 96, in call headers=headers) File "/usr/lib/python3.6/site-packages/requests/api.py", line 60, in request return session.request(method=method, url=url, **kwargs) File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 533, in request resp = self.send(prep, **send_kwargs) File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 646, in send r = adapter.send(request, **kwargs) File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 498, in send raise ConnectionError(err, request=request) requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/django/core/handlers/base.py", line 113, in _get_response response = wrapped_callback(request, *callback_args, **callback_kwargs) File "/usr/lib/python3.6/site-packages/django/contrib/auth/decorators.py", line 21, in _wrapped_view return view_func(request, *args, **kwargs) File "/usr/lib/python3.6/site-packages/postorius/auth/decorators.py", line 38, in wrapper return fn(*args, **kwargs) File "/usr/lib/python3.6/site-packages/postorius/views/list.py", line 943, in remove_role roster = getattr(the_list, '{}s'.format(role)) File "/usr/lib/python3.6/site-packages/mailmanclient/restobjects/mailinglist.py", line 86, in nonmembers response, content = self._connection.call(url, data) File "/usr/lib/python3.6/site-packages/mailmanclient/restbase/connection.py", line 119, in call raise MailmanConnectionError('Could not connect to Mailman API') mailmanclient.restbase.connection.MailmanConnectionError: Could not connect to Mailman API ERROR 2020-04-09 20:35:51,509 19 django.request Service Unavailable: /postorius/lists/scss-sysadmins.lists.scss.tcd.ie/remove/nonmember/skenny@scss.tcd.ie ERROR 2020-04-09 20:35:51,509 19 django.request Service Unavailable: /postorius/lists/scss-sysadmins.lists.scss.tcd.ie/remove/nonmember/skenny@scss.tcd.ie

The file /opt/mailman-web-data/logs/uwsgi.log contains just one line:

[pid: 19|app: 0|req: 82/96] 172.19.199.5 () {46 vars in 1233 bytes} [Thu Apr 9 20:35:20 2020] GET /postorius/lists/scss-sysadmins.lists.scss.tcd.ie/remove/nonmember/skenny@scss.tcd.ie => generated 5583 bytes in 31411 msecs (HTTP/1.0 503) 5 headers in 180 bytes (1 switches on core 0)

...

Can you try doing the same with the REST API, just to verify if it is the Core which has the issue or it is the web frontend.

Using the REST API, I can successfully remove a named nonmember from the list. But I see the same pause-then-fail behaviour when I try to simply list all the nonmembers. My snippet of code can happily produce a listing of the members, but it gets stuck trying to list nonmembers, and then 30 seconds later it crashes out with exactly the same error messages as when I tried to delete a user in the web interface (see above). My piece of code appears below (II invoke it like this: docker exec -it mailman-core bash -c 'python3 /opt/mailman/local/probe.py scss-sysadmins@lists.scss.tcd.ie').

from mailmanclient import Client from mailmanclient import MailingList import sys

my_list = str(sys.argv[1]) client = Client('http://172.19.199.2:8001/3.0', 'restadmin', 'restpass') this_list = client.get_list(my_list)

print("\n\nMembers:\n-------------------------------------------------") for i in this_list.members: print(i)

print("\nNonmembers:\n-------------------------------------------------") for i in this_list.nonmembers: print(i)

Thanks again Stephen

---

Mailman-users mailing list -- mailman-users@mailman3.org To unsubscribe send an email to mailman-users-leave@mailman3.org https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/

-- thanks, Abhilash Raj (maxking)

On Fri, Apr 10, 2020, at 3:40 AM, skenny@scss.tcd.ie wrote:

Abhilash Raj wrote:

...

...

It looks like you are looking at the Mailman-web logs. From the looks of it, the client side is basically timing out after 30 seconds. Same seems to be true for the script you wrote (which is calling the REST API underneath using mailmanclient), which is trying to fetch the list of non-members. Can you also look at the server side of the logs? It should be available at /opt/mailman/core/var/logs/mailman.log if you are using the standard docker-compose file from the repository.

OK, here's the output from /opt/mailman/core/var/logs/mailman.log when I click on delete in the web interface:

[10/Apr/2020:08:53:31 +0000] "GET /3.1/lists/scss-sysadmins.lists.scss.tcd.ie HTTP/1.1" 200 421 "-" "GNU Mailman REST client v3.3.0" [10/Apr/2020:08:53:31 +0000] "GET /3.1/lists/scss-sysadmins.lists.scss.tcd.ie/roster/owner HTTP/1.1" 200 626 "-" "GNU Mailman REST client v3.3.0" [10/Apr/2020:08:53:31 +0000] "GET /3.1/lists/scss-sysadmins.lists.scss.tcd.ie/roster/moderator HTTP/1.1" 200 90 "-" "GNU Mailman REST client v3.3.0" [10/Apr/2020:08:53:31 +0000] "GET /3.1/members/find?list_id=scss-sysadmins.lists.scss.tcd.ie&role=nonmember&count=0&page=1 HTTP/1.1" 200 92 "-" "GNU Mailman REST client v3.3.0"

This line is weird, why is it sending count=0. I'll investigate more.

[10/Apr/2020:08:53:33 +0000] "GET /3.1/members/find?list_id=scss-sysadmins.lists.scss.tcd.ie&role=nonmember&count=25&page=1 HTTP/1.1" 200 12511 "-" "GNU Mailman REST client v3.3.0" [10/Apr/2020:08:53:33 +0000] "GET /3.1/lists/scss-sysadmins@lists.scss.tcd.ie/requests HTTP/1.1" 200 90 "-" "GNU Mailman REST client v3.3.0" [10/Apr/2020:08:53:33 +0000] "GET /3.1/lists/scss-sysadmins@lists.scss.tcd.ie/held?count=50&page=1 HTTP/1.1" 200 90 "-" "GNU Mailman REST client v3.3.0" [10/Apr/2020:08:53:37 +0000] "GET /3.1/lists/scss-sysadmins.lists.scss.tcd.ie HTTP/1.1" 200 421 "-" "GNU Mailman REST client v3.3.0"

[2020-04-10 08:54:07 +0000] [30] [CRITICAL] WORKER TIMEOUT (pid:36) [2020-04-10 08:54:07 +0000] [36] [INFO] Worker exiting (pid: 36) [2020-04-10 08:54:08 +0000] [38] [INFO] Booting worker with pid: 38 [10/Apr/2020:09:00:07 +0000] "GET /3.0/lists?count=10&page=1 HTTP/1.1" 200 3749 "-" "GNU Mailman REST client v3.3.0" [10/Apr/2020:10:00:07 +0000] "GET /3.0/lists?count=10&page=1 HTTP/1.1" 200 3749 "-" "GNU Mailman REST client v3.3.0"

Stephen

---

Mailman-users mailing list -- mailman-users@mailman3.org To unsubscribe send an email to mailman-users-leave@mailman3.org https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/

-- thanks, Abhilash Raj (maxking)

On Fri, Apr 10, 2020, at 3:45 AM, skenny@scss.tcd.ie wrote:

Out put from my python program is not dissimilar - I am seeing the same WORKER TIMEOUT message:

[10/Apr/2020:10:42:54 +0000] "GET /3.0/lists/scss-sysadmins@lists.scss.tcd.ie HTTP/1.1" 200 421 "-" "GNU Mailman REST client v3.3.0" [10/Apr/2020:10:42:55 +0000] "GET /3.0/lists/scss-sysadmins@lists.scss.tcd.ie/roster/member HTTP/1.1" 200 3766 "-" "GNU Mailman REST client v3.3.0"

This seems like is timing out getting list of members. Without pagination, I do expect it to time out if the list is large. I think .members by default doesn't use pagination.

[2020-04-10 10:43:25 +0000] [30] [CRITICAL] WORKER TIMEOUT (pid:38) [2020-04-10 10:43:25 +0000] [38] [INFO] Worker exiting (pid: 38) [2020-04-10 10:43:26 +0000] [45] [INFO] Booting worker with pid: 45

Stephen

---

Mailman-users mailing list -- mailman-users@mailman3.org To unsubscribe send an email to mailman-users-leave@mailman3.org https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/

-- thanks, Abhilash Raj (maxking)

Abhilash Raj wrote:

This seems like is timing out getting list of members. Without pagination, I do expect it to time out if the list is large. I think .members by default doesn't use pagination.

Hi Abhilash,

I'm just wondering if there was any update on the issue above?

Thanks Stephen

skenny@scss.tcd.ie writes:

Actually, I came across another thread where you suggested increasing the "graceful_timeout" in gunicorn.cfg. In fact this didn't work for me, but I followed your reference to the documentation

Which documentation? An exact reference would be nice, but I'm not even sure if you're talking about Mailman or gunicorn docs here -- whatever you have off the top of your head would help! I'd like to find a place to put this in the Mailman docs.

and tried increasing just "timeout" instead - and that worked!

It's pretty slow, but I can now get all my (very long) mailing list members via the REST API, which is great.

Do you mind saying just how long your very long list is (ie, the number of subscribers)? It's useful to us to have some idea of the scales at which people are operating (and having performance issues :-/ ). (The biggest list I run in Mailman 3 is 17, so I'm not much help with that.)

I'm not happy about your "pretty slow" remark. We've had Mailman 2 lists get seriously big (seven digits worth of subscribers, I think). If we can improve the performance of listing, or help you to improve it by using a different API, it would be good for you, and help with scaling Mailman 3. I forget the context, but let me confirm: you're using the REST interface to get the member list, right?

The main point: I'm wondering if you might not get better performance by getting using the paginated interface, and assembling the list at the receiving end instead of trying to get it all at once.

Steve

On Fri, Apr 17, 2020, at 8:54 AM, skenny@scss.tcd.ie wrote:

Hi Steve, I replied to your personal email address earlier today. - I guess it's no harm to add my reply to the thread here, so here goes:

Hi Stephen,

Thanks for your message. The documentation I was referring to is from Gunicorn, specifically this:

 http://docs.gunicorn.org/en/latest/settings.html#timeout

(I was led to it by Abhilash's reference in the thread).

Did you also add the config file in mailman.cfg

[webservice] # Configuration for webservice. configuration: /path/to/gunicorn.cfg

This is needed to notify Mailman of the new configuration file that you added. And then format and options of the config file is the URL that you already mentioned above. This information should be documented somewhere, I'll create a bug report for that.

As for the length of the list, I'm talking about in the region of 1,800 members.

When I say "pretty slow", I mean that it takes about 2.5 minutes to return the list of members via the REST API.

I'm not sure I understand your last paragraph - "paginated interface" and "at the receiving end" don't make much sense to me at present. I should add however that I am pretty novice when it comes to all of this (and Python programming too). It took me a while to understand the difference between accessing Mailman3 via the REST API versus mailman shell - for example!

Stephen

---

Mailman-users mailing list -- mailman-users@mailman3.org To unsubscribe send an email to mailman-users-leave@mailman3.org https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/

-- thanks, Abhilash Raj (maxking)

Well, it is quite possible that something is seriously wrong with my configuration.

The database backend is Postgres - not something I configured. It is specified in my docker-compose file, which in turn is based on the compose files at https://github.com/maxking/docker-mailman.

(At one point I thought maybe the virtual machine I'm running on was under-resourced. I've since doubled memory and the number of CPUs but that hasn't really improved things.)

I'm wondering where to go next - try tweaking the Postgres config, switch to SQLlite ....

Thanks for giving time to helping me with this, much appreciated.

Stephen

On Sat, Apr 18, 2020, at 8:25 AM, skenny@scss.tcd.ie wrote:

Well, it is quite possible that something is seriously wrong with my configuration.

The database backend is Postgres - not something I configured. It is specified in my docker-compose file, which in turn is based on the compose files at https://github.com/maxking/docker-mailman.

(At one point I thought maybe the virtual machine I'm running on was under-resourced. I've since doubled memory and the number of CPUs but that hasn't really improved things.)

I'm wondering where to go next - try tweaking the Postgres config, switch to SQLlite ....

How many entries do you have in your member table? You can just use psql command in the postgres container to dump a count of all the rows?

Thanks for giving time to helping me with this, much appreciated.

Stephen

---

Mailman-users mailing list -- mailman-users@mailman3.org To unsubscribe send an email to mailman-users-leave@mailman3.org https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/

-- thanks, Abhilash Raj (maxking)

OK, I've done a clean rebuild of my server with Puppet. (I'll post a copy of my docker-compose file and related config files at the end of this message.)

Via the postorius web interface, I've created a new list called scss-sysadmins@scss.tcd.ie, and I've added 1,800 nonmembers to it using this fairly basic piece of code:

---- add_nonmembers.py start --------------------------------------- #!/usr/bin/python3 from mailmanclient import Client

import sys listarg = str(sys.argv[1]) client = Client('http://172.19.199.2:8001/3.0', 'restadmin', 'restpass')

this_list = client.get_list(listarg) this_list_short = this_list.list_name work_dir = "/usr/local/mailman-scss/list_membership/" + this_list_short + "/" nonmembers_add_file = work_dir + "nonmembers.add"

infile = open(nonmembers_add_file, "r")

for addr in infile: if addr.isspace(): continue addr = addr.strip() print("Adding " + addr + "as a nonmember ") this_list.add_role('nonmember', addr, display_name='')

infile.close() ---- add_nonmembers.py end ---------------------------------------

After doing this, a Postgres "select * from member" SQL command executed in the database container prints a total of 1,801 rows (one more than the number of nonmembers added).

In the web interface, I bring up the list of nonmembers, and then click on "delete" to remove an individual nonmember (in all that follows I've replaced the actual username with XXXXXXXX for privacy reasons). This fails after 30 seconds and the following message appears:

Something went wrong
Mailman REST API not available. Please start Mailman core.

Next, in the mailmain-core container, I increase Gunicorn's timeout parameter to 300 seconds (I edited /usr/lib/python3.6/site-packages/mailman/config/gunicorn.cfg and restarted the container), and try the same operation again. Here's what happens: (1) After 2 minutes and 3 seconds approx, the "Confirm remove role" page appears; (2) I click on "Remove XXXXXXXX@tcd.ie", and after a *further* 2 minutes and 3 or so seconds later, the operation completes - the nonmember no longer appears on the page. The URL that appears in the address bar of my browser after clicking on "delete" is http://mailman-web.scss.tcd.ie/postorius/lists/scss-sysadmins.lists.scss.tcd...

Here are the entries in the *.log files from the -web and -core containers generated by the latter (timeout = 300) attempt to remove a nonmember:

==> ./core/var/logs/mailman.log <== [22/Apr/2020:17:17:25 +0000] "GET /3.1/lists/scss-sysadmins.lists.scss.tcd.ie HTTP/1.1" 200 395 "-" "GNU Mailman REST client v3.3.0" [22/Apr/2020:17:19:20 +0000] "POST /3.1/members/find HTTP/1.1" 200 896038 "-" "GNU Mailman REST client v3.3.0"

==> ./web/logs/uwsgi.log <== [pid: 21|app: 0|req: 65/198] 109.255.134.153 () {36 vars in 1160 bytes} [Wed Apr 22 17:17:25 2020] GET /postorius/lists/scss-sysadmins.lists.scss.tcd.ie/remove/nonmember/XXXXXXXX@tcd.ie => generated 6217 bytes in 115377 msecs (HTTP/1.1 200) 6 headers in 331 bytes (1 switches on core 0)

==> ./core/var/logs/mailman.log <== [22/Apr/2020:17:19:45 +0000] "GET /3.1/lists/scss-sysadmins.lists.scss.tcd.ie HTTP/1.1" 200 395 "-" "GNU Mailman REST client v3.3.0" [22/Apr/2020:17:21:48 +0000] "POST /3.1/members/find HTTP/1.1" 200 896038 "-" "GNU Mailman REST client v3.3.0" [22/Apr/2020:17:21:48 +0000] "DELETE /3.1/lists/scss-sysadmins@lists.scss.tcd.ie/nonmember/XXXXXXXX%40tcd.ie HTTP/1.1" 204 0 "-" "GNU Mailman REST client v3.3.0"

==> ./web/logs/uwsgi.log <== [pid: 21|app: 0|req: 66/199] 109.255.134.153 () {44 vars in 1320 bytes} [Wed Apr 22 17:19:45 2020] POST /postorius/lists/scss-sysadmins.lists.scss.tcd.ie/remove/nonmember/XXXXXXXX@tcd.ie => generated 0 bytes in 122966 msecs (HTTP/1.1 302) 7 headers in 450 bytes (1 switches on core 0)

==> ./core/var/logs/mailman.log <== [22/Apr/2020:17:21:48 +0000] "GET /3.1/lists/scss-sysadmins.lists.scss.tcd.ie HTTP/1.1" 200 395 "-" "GNU Mailman REST client v3.3.0" [22/Apr/2020:17:21:48 +0000] "GET /3.1/lists/scss-sysadmins.lists.scss.tcd.ie/roster/owner HTTP/1.1" 200 626 "-" "GNU Mailman REST client v3.3.0" [22/Apr/2020:17:21:48 +0000] "GET /3.1/lists/scss-sysadmins.lists.scss.tcd.ie/roster/moderator HTTP/1.1" 200 90 "-" "GNU Mailman REST client v3.3.0" [22/Apr/2020:17:21:48 +0000] "GET /3.1/members/find?list_id=scss-sysadmins.lists.scss.tcd.ie&role=nonmember&count=0&page=1 HTTP/1.1" 200 93 "-" "GNU Mailman REST client v3.3.0" [22/Apr/2020:17:21:50 +0000] "GET /3.1/members/find?list_id=scss-sysadmins.lists.scss.tcd.ie&role=nonmember&count=25&page=1 HTTP/1.1" 200 12506 "-" "GNU Mailman REST client v3.3.0" [22/Apr/2020:17:21:50 +0000] "GET /3.1/lists/scss-sysadmins@lists.scss.tcd.ie/requests HTTP/1.1" 200 90 "-" "GNU Mailman REST client v3.3.0" [22/Apr/2020:17:21:50 +0000] "GET /3.1/lists/scss-sysadmins@lists.scss.tcd.ie/held?count=50&page=1 HTTP/1.1" 200 90 "-" "GNU Mailman REST client v3.3.0"

==> ./web/logs/uwsgi.log <== [pid: 21|app: 0|req: 67/200] 109.255.134.153 () {38 vars in 1339 bytes} [Wed Apr 22 17:21:48 2020] GET /postorius/lists/scss-sysadmins.lists.scss.tcd.ie/members/nonmember/ => generated 41068 bytes in 2266 msecs (HTTP/1.1 200) 7 headers in 416 bytes (1 switches on core 0) [pid: 21|app: -1|req: -1/201] 109.255.134.153 () {36 vars in 1031 bytes} [Wed Apr 22 17:21:51 2020] GET /static/postorius/img/mailman_logo_small_trans.png => generated 0 bytes in 0 msecs (HTTP/1.1 304) 0 headers in 29 bytes (0 switches on core 0)

My configuration uses two Docker compose files, one for a Postfix mail server container, and the other for all the mailman stuff. (As you'll see, my browser connection to the mailman-web container is facilitated by exposing port 8000 to port 80 on the Docker host. I'll front everything with an nginx reverse proxy later.) Here is the latter, followed by my mailman-extra.cfg and settings_local.py:

---- mailman-docker-compose.yml start ------------------------------------------------------ version: '2'

services:

mailman-core: image: maxking/mailman-core:0.3 container_name: mailman-core hostname: mailman-core volumes: - /opt/mailman/core:/opt/mailman/ - /etc/scss-mailman-config/mailman-extra.cfg:/opt/mailman/mailman-extra.cfg - /usr/local/mailman-scss:/opt/mailman/scss stop_grace_period: 30s links: - database:database depends_on: - database environment: - DATABASE_URL=postgres://mailman:mailmanpass@database/mailmandb - DATABASE_TYPE=postgres - DATABASE_CLASS=mailman.database.postgresql.PostgreSQLDatabase - HYPERKITTY_API_KEY=someapikey networks: mailman: ipv4_address: 172.19.199.2 restart: always

mailman-web: image: maxking/mailman-web:0.3 container_name: mailman-web hostname: mailman-web ports: - 80:8000 depends_on: - database links: - mailman-core:mailman-core - database:database volumes: - /opt/mailman/web:/opt/mailman-web-data - /etc/scss-mailman-config/settings_local.py:/opt/mailman-web/settings_local.py environment: - DATABASE_TYPE=postgres - DATABASE_URL=postgres://mailman:mailmanpass@database/mailmandb - HYPERKITTY_API_KEY=someapikey - SECRET_KEY=shhitsasecret - SMTP_HOST=172.19.199.10 - MAILMAN_ADMIN_USER=skenny - MAILMAN_ADMIN_EMAIL=skenny@scss.tcd.ie - UWSGI_STATIC_MAP=/static=/opt/mailman-web-data/static - SERVE_FROM_DOMAIN=mailman-web.scss.tcd.ie - DEFAULT_FROM_DOMAIN=mailman-web.scss.tcd.ie networks: mailman: ipv4_address: 172.19.199.3 restart: always

database: environment: POSTGRES_DB: mailmandb POSTGRES_USER: mailman POSTGRES_PASSWORD: mailmanpass restart: always image: postgres:9.6-alpine volumes: - /opt/mailman/database:/var/lib/postgresql/data networks: mailman: ipv4_address: 172.19.199.4

networks: mailman: driver: bridge ipam: driver: default config: - subnet: 172.19.199.0/24 ---- mailman-docker-compose.yml end ------------------------------------------------------

---- mailman-extra.cfg start ------------------------------------------------------ # mailman-extra.cfg

[mta] incoming: mailman.mta.postfix.LMTP outgoing: mailman.mta.deliver.deliver lmtp_host: 172.19.199.2 lmtp_port: 8024 smtp_host: 172.19.199.10 smtp_port: 25 configuration: /etc/postfix-mailman.cfg

[mailman] # This address is the "site owner" address. Certain messages which must be # delivered to a human, but which can't be delivered to a list owner (e.g. a # bounce from a list owner), will be sent to this address. It should point to # a human. site_owner: mrrobot@lists2.scss.tcd.ie

[logging.smtp] level: debug path: smtp.log ---- mailman-extra.cfg end ------------------------------------------------------

---- settings_local.py start ------------------------------------------------------ # Hosts/domain names that are valid for this site; required if DEBUG is False # See https://docs.djangoproject.com/en/1.8/ref/settings/#allowed-hosts ALLOWED_HOSTS = [ "localhost", # Archiving API from Mailman, keep it. # "lists.your-domain.org", # Add here all production URLs you may have. "mailman-web", "mailman-web.scss.tcd.ie", "172.19.199.3", ####sk os.environ.get('SERVE_FROM_DOMAIN'), ####sk os.environ.get('DJANGO_ALLOWED_HOSTS'), ]

INSTALLED_APPS = [ 'hyperkitty', 'postorius', 'django_mailman3', # Uncomment the next line to enable the admin: 'django.contrib.admin', # Uncomment the next line to enable admin documentation: # 'django.contrib.admindocs', 'django.contrib.auth', 'django.contrib.contenttypes', 'django.contrib.sessions', 'django.contrib.sites', 'django.contrib.messages', 'django.contrib.staticfiles', 'rest_framework', 'django_gravatar', 'compressor', 'haystack', 'django_extensions', 'django_q', 'allauth', 'allauth.account', 'allauth.socialaccount', # -------------------------------------------------------------------------- # We're killing the following ... (tip from brian@emwd.com on this page: # https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/thread/L... # But it doesn't stop people signing up. We need to deal with that via # Nginx .... # -------------------------------------------------------------------------- #'django_mailman3.lib.auth.fedora', #'allauth.socialaccount.providers.openid', #'allauth.socialaccount.providers.github', #'allauth.socialaccount.providers.gitlab', #'allauth.socialaccount.providers.google', ---- settings_local.py end ------------------------------------------------------

Thanks again for helping Stephen

Hi Abhilash,

I was just wondering if you got a chance to look at the diagnostics I sent regarding the delay I'm seeing when deleting a nonmember via the web interface

Thanks a lot Stephen

On Thu, Apr 30, 2020, at 7:45 AM, Brian Carpenter wrote:

Any chance a developer can answer Stephen? We are also seeing significant delays working with list members via the REST api and core in our development of Affinity.

We have been discussing this in

https://gitlab.com/mailman/mailman/-/issues/700

The crux of the problem is that calculating a Memeber resources is a complex operation and requires several database queries. With my calculation, it takes about 5 database queries per member resource and so if the list of large, the API request can time out.

This isn't straight forward to fix since there isn't any logical bug. We would need to re-architect how the data is loaded from database and Member resource is calculated, possibly loading child resources in the same query to build the Member resource with fewer database queries.

Thanks, Brian

---

Mailman-users mailing list -- mailman-users@mailman3.org To unsubscribe send an email to mailman-users-leave@mailman3.org https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/

-- thanks, Abhilash Raj (maxking)