OK, I've done a clean rebuild of my server with Puppet. (I'll post a copy of my docker-compose file and related config files at the end of this message.)
Via the postorius web interface, I've created a new list called scss-sysadmins@scss.tcd.ie, and I've added 1,800 nonmembers to it using this fairly basic piece of code:
---- add_nonmembers.py start --------------------------------------- #!/usr/bin/python3 from mailmanclient import Client
import sys listarg = str(sys.argv[1]) client = Client('http://172.19.199.2:8001/3.0', 'restadmin', 'restpass')
this_list = client.get_list(listarg) this_list_short = this_list.list_name work_dir = "/usr/local/mailman-scss/list_membership/" + this_list_short + "/" nonmembers_add_file = work_dir + "nonmembers.add"
infile = open(nonmembers_add_file, "r")
for addr in infile: if addr.isspace(): continue addr = addr.strip() print("Adding " + addr + "as a nonmember ") this_list.add_role('nonmember', addr, display_name='')
infile.close() ---- add_nonmembers.py end ---------------------------------------
After doing this, a Postgres "select * from member" SQL command executed in the database container prints a total of 1,801 rows (one more than the number of nonmembers added).
In the web interface, I bring up the list of nonmembers, and then click on "delete" to remove an individual nonmember (in all that follows I've replaced the actual username with XXXXXXXX for privacy reasons). This fails after 30 seconds and the following message appears:
Something went wrong
Mailman REST API not available. Please start Mailman core.Next, in the mailmain-core container, I increase Gunicorn's timeout parameter to 300 seconds (I edited /usr/lib/python3.6/site-packages/mailman/config/gunicorn.cfg and restarted the container), and try the same operation again. Here's what happens: (1) After 2 minutes and 3 seconds approx, the "Confirm remove role" page appears; (2) I click on "Remove XXXXXXXX@tcd.ie", and after a *further* 2 minutes and 3 or so seconds later, the operation completes - the nonmember no longer appears on the page. The URL that appears in the address bar of my browser after clicking on "delete" is http://mailman-web.scss.tcd.ie/postorius/lists/scss-sysadmins.lists.scss.tcd...
Here are the entries in the *.log files from the -web and -core containers generated by the latter (timeout = 300) attempt to remove a nonmember:
==> ./core/var/logs/mailman.log <== [22/Apr/2020:17:17:25 +0000] "GET /3.1/lists/scss-sysadmins.lists.scss.tcd.ie HTTP/1.1" 200 395 "-" "GNU Mailman REST client v3.3.0" [22/Apr/2020:17:19:20 +0000] "POST /3.1/members/find HTTP/1.1" 200 896038 "-" "GNU Mailman REST client v3.3.0"
==> ./web/logs/uwsgi.log <== [pid: 21|app: 0|req: 65/198] 109.255.134.153 () {36 vars in 1160 bytes} [Wed Apr 22 17:17:25 2020] GET /postorius/lists/scss-sysadmins.lists.scss.tcd.ie/remove/nonmember/XXXXXXXX@tcd.ie => generated 6217 bytes in 115377 msecs (HTTP/1.1 200) 6 headers in 331 bytes (1 switches on core 0)
==> ./core/var/logs/mailman.log <== [22/Apr/2020:17:19:45 +0000] "GET /3.1/lists/scss-sysadmins.lists.scss.tcd.ie HTTP/1.1" 200 395 "-" "GNU Mailman REST client v3.3.0" [22/Apr/2020:17:21:48 +0000] "POST /3.1/members/find HTTP/1.1" 200 896038 "-" "GNU Mailman REST client v3.3.0" [22/Apr/2020:17:21:48 +0000] "DELETE /3.1/lists/scss-sysadmins@lists.scss.tcd.ie/nonmember/XXXXXXXX%40tcd.ie HTTP/1.1" 204 0 "-" "GNU Mailman REST client v3.3.0"
==> ./web/logs/uwsgi.log <== [pid: 21|app: 0|req: 66/199] 109.255.134.153 () {44 vars in 1320 bytes} [Wed Apr 22 17:19:45 2020] POST /postorius/lists/scss-sysadmins.lists.scss.tcd.ie/remove/nonmember/XXXXXXXX@tcd.ie => generated 0 bytes in 122966 msecs (HTTP/1.1 302) 7 headers in 450 bytes (1 switches on core 0)
==> ./core/var/logs/mailman.log <== [22/Apr/2020:17:21:48 +0000] "GET /3.1/lists/scss-sysadmins.lists.scss.tcd.ie HTTP/1.1" 200 395 "-" "GNU Mailman REST client v3.3.0" [22/Apr/2020:17:21:48 +0000] "GET /3.1/lists/scss-sysadmins.lists.scss.tcd.ie/roster/owner HTTP/1.1" 200 626 "-" "GNU Mailman REST client v3.3.0" [22/Apr/2020:17:21:48 +0000] "GET /3.1/lists/scss-sysadmins.lists.scss.tcd.ie/roster/moderator HTTP/1.1" 200 90 "-" "GNU Mailman REST client v3.3.0" [22/Apr/2020:17:21:48 +0000] "GET /3.1/members/find?list_id=scss-sysadmins.lists.scss.tcd.ie&role=nonmember&count=0&page=1 HTTP/1.1" 200 93 "-" "GNU Mailman REST client v3.3.0" [22/Apr/2020:17:21:50 +0000] "GET /3.1/members/find?list_id=scss-sysadmins.lists.scss.tcd.ie&role=nonmember&count=25&page=1 HTTP/1.1" 200 12506 "-" "GNU Mailman REST client v3.3.0" [22/Apr/2020:17:21:50 +0000] "GET /3.1/lists/scss-sysadmins@lists.scss.tcd.ie/requests HTTP/1.1" 200 90 "-" "GNU Mailman REST client v3.3.0" [22/Apr/2020:17:21:50 +0000] "GET /3.1/lists/scss-sysadmins@lists.scss.tcd.ie/held?count=50&page=1 HTTP/1.1" 200 90 "-" "GNU Mailman REST client v3.3.0"
==> ./web/logs/uwsgi.log <== [pid: 21|app: 0|req: 67/200] 109.255.134.153 () {38 vars in 1339 bytes} [Wed Apr 22 17:21:48 2020] GET /postorius/lists/scss-sysadmins.lists.scss.tcd.ie/members/nonmember/ => generated 41068 bytes in 2266 msecs (HTTP/1.1 200) 7 headers in 416 bytes (1 switches on core 0) [pid: 21|app: -1|req: -1/201] 109.255.134.153 () {36 vars in 1031 bytes} [Wed Apr 22 17:21:51 2020] GET /static/postorius/img/mailman_logo_small_trans.png => generated 0 bytes in 0 msecs (HTTP/1.1 304) 0 headers in 29 bytes (0 switches on core 0)
My configuration uses two Docker compose files, one for a Postfix mail server container, and the other for all the mailman stuff. (As you'll see, my browser connection to the mailman-web container is facilitated by exposing port 8000 to port 80 on the Docker host. I'll front everything with an nginx reverse proxy later.) Here is the latter, followed by my mailman-extra.cfg and settings_local.py:
---- mailman-docker-compose.yml start ------------------------------------------------------ version: '2'
services:
mailman-core: image: maxking/mailman-core:0.3 container_name: mailman-core hostname: mailman-core volumes: - /opt/mailman/core:/opt/mailman/ - /etc/scss-mailman-config/mailman-extra.cfg:/opt/mailman/mailman-extra.cfg - /usr/local/mailman-scss:/opt/mailman/scss stop_grace_period: 30s links: - database:database depends_on: - database environment: - DATABASE_URL=postgres://mailman:mailmanpass@database/mailmandb - DATABASE_TYPE=postgres - DATABASE_CLASS=mailman.database.postgresql.PostgreSQLDatabase - HYPERKITTY_API_KEY=someapikey networks: mailman: ipv4_address: 172.19.199.2 restart: always
mailman-web: image: maxking/mailman-web:0.3 container_name: mailman-web hostname: mailman-web ports: - 80:8000 depends_on: - database links: - mailman-core:mailman-core - database:database volumes: - /opt/mailman/web:/opt/mailman-web-data - /etc/scss-mailman-config/settings_local.py:/opt/mailman-web/settings_local.py environment: - DATABASE_TYPE=postgres - DATABASE_URL=postgres://mailman:mailmanpass@database/mailmandb - HYPERKITTY_API_KEY=someapikey - SECRET_KEY=shhitsasecret - SMTP_HOST=172.19.199.10 - MAILMAN_ADMIN_USER=skenny - MAILMAN_ADMIN_EMAIL=skenny@scss.tcd.ie - UWSGI_STATIC_MAP=/static=/opt/mailman-web-data/static - SERVE_FROM_DOMAIN=mailman-web.scss.tcd.ie - DEFAULT_FROM_DOMAIN=mailman-web.scss.tcd.ie networks: mailman: ipv4_address: 172.19.199.3 restart: always
database: environment: POSTGRES_DB: mailmandb POSTGRES_USER: mailman POSTGRES_PASSWORD: mailmanpass restart: always image: postgres:9.6-alpine volumes: - /opt/mailman/database:/var/lib/postgresql/data networks: mailman: ipv4_address: 172.19.199.4
networks: mailman: driver: bridge ipam: driver: default config: - subnet: 172.19.199.0/24 ---- mailman-docker-compose.yml end ------------------------------------------------------
---- mailman-extra.cfg start ------------------------------------------------------ # mailman-extra.cfg
[mta] incoming: mailman.mta.postfix.LMTP outgoing: mailman.mta.deliver.deliver lmtp_host: 172.19.199.2 lmtp_port: 8024 smtp_host: 172.19.199.10 smtp_port: 25 configuration: /etc/postfix-mailman.cfg
[mailman] # This address is the "site owner" address. Certain messages which must be # delivered to a human, but which can't be delivered to a list owner (e.g. a # bounce from a list owner), will be sent to this address. It should point to # a human. site_owner: mrrobot@lists2.scss.tcd.ie
[logging.smtp] level: debug path: smtp.log ---- mailman-extra.cfg end ------------------------------------------------------
---- settings_local.py start ------------------------------------------------------ # Hosts/domain names that are valid for this site; required if DEBUG is False # See https://docs.djangoproject.com/en/1.8/ref/settings/#allowed-hosts ALLOWED_HOSTS = [ "localhost", # Archiving API from Mailman, keep it. # "lists.your-domain.org", # Add here all production URLs you may have. "mailman-web", "mailman-web.scss.tcd.ie", "172.19.199.3", ####sk os.environ.get('SERVE_FROM_DOMAIN'), ####sk os.environ.get('DJANGO_ALLOWED_HOSTS'), ]
INSTALLED_APPS = [ 'hyperkitty', 'postorius', 'django_mailman3', # Uncomment the next line to enable the admin: 'django.contrib.admin', # Uncomment the next line to enable admin documentation: # 'django.contrib.admindocs', 'django.contrib.auth', 'django.contrib.contenttypes', 'django.contrib.sessions', 'django.contrib.sites', 'django.contrib.messages', 'django.contrib.staticfiles', 'rest_framework', 'django_gravatar', 'compressor', 'haystack', 'django_extensions', 'django_q', 'allauth', 'allauth.account', 'allauth.socialaccount', # -------------------------------------------------------------------------- # We're killing the following ... (tip from brian@emwd.com on this page: # https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/thread/L... # But it doesn't stop people signing up. We need to deal with that via # Nginx .... # -------------------------------------------------------------------------- #'django_mailman3.lib.auth.fedora', #'allauth.socialaccount.providers.openid', #'allauth.socialaccount.providers.github', #'allauth.socialaccount.providers.gitlab', #'allauth.socialaccount.providers.google', ---- settings_local.py end ------------------------------------------------------
Thanks again for helping Stephen