Ok, now that I got that out of the way. Maybe I can get some help here for a Docker Installation of Mailman 3.
I updated a Maxking's Docker installation to Postorius 1.3.3. It made my client's list unusable to a continual stream of removals due to ghost bounce processes. So, I want to totally move away from the use of Docker (mainly because I have found it buggy and the support is simply lacking) and install Mailman 3 for this client from source. However I need to get an export of their list members. But of course, I can't. Rest API times out and I don't know how to apply the instructions I was given to fix it and I don't have days to way for one. So please help.
I increased the gunicorn workers to 4, hoping that would allow me to do the CSV export. No. So was given these instructions:
### The issues seems to be due to timeout not due to high load. You want to increase the timeout for gunicorn and you can do that by specifying a gunicorn.ini config file:
[webservice] configuration: /path/to/gunicorn.cfg
and then in the gunicorn.cfg you can add
[gunicorn] timeout = 90 graceful_timeout = 30 ###
I applied the above changes to /opt/mailman/core/mailman-extra.cfg and created the gunicorn.cfg file. Restarted the docker container (mailman-core) and no access to Postorius due to no REST api.
After restarting docker-mailman, I can't access Postorius due to the following error:
ERROR 2020-07-20 20:16:27,351 15 postorius Mailman REST API not available Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 160, in _new_conn (self._dns_host, self.port), self.timeout, **extra_kw File "/usr/lib/python3.6/site-packages/urllib3/util/connection.py", line 84, in create_connection raise err File "/usr/lib/python3.6/site-packages/urllib3/util/connection.py", line 74, in create_connection sock.connect(sa) ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 677, in urlopen chunked=chunked, File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 392, in _make_request conn.request(method, url, **httplib_request_kw) File "/usr/lib/python3.6/http/client.py", line 1254, in request self._send_request(method, url, body, headers, encode_chunked) File "/usr/lib/python3.6/http/client.py", line 1300, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/usr/lib/python3.6/http/client.py", line 1249, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/lib/python3.6/http/client.py", line 1036, in _send_output self.send(msg) File "/usr/lib/python3.6/http/client.py", line 974, in send self.connect() File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 187, in connect conn = self._new_conn() File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 172, in _new_conn self, "Failed to establish a new connection: %s" % e urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f96340a2e10>: Failed to establish a new con$
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 449, in send timeout=timeout File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 725, in urlopen
Commenting out "configuration: /opt/mailman/gunicorn.cfg" brings Postorius back up.
I am really frustrated here. All I want to do is export a 3000 member list and I can't. Updating Mailman 3 using new Docker images shouldn't break a stable installation either.
Please help.
Brian
On 7/21/20 5:40 PM, Brian Carpenter wrote:
Ok, now that I got that out of the way. Maybe I can get some help here for a Docker Installation of Mailman 3.
Abhilash (@maxking) is one person and he has a $DayJob. We're all doing the best we can.
I updated a Maxking's Docker installation to Postorius 1.3.3. It made my client's list unusable to a continual stream of removals due to ghost bounce processes.
This is not a Docker issue per se. It is a consequence of upgrading Mailman core to 3.3.1+ and processing many old, stale, previously unprocessed bounces. See <https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/thread/U5ZW6X5QLQRXUOMD2DN77CYCOLKVMUXJ/>
So, I want to totally move away from the use of Docker (mainly because I have found it buggy and the support is simply lacking) and install Mailman 3 for this client from source. However I need to get an export of their list members. But of course, I can't. Rest API times out and I don't know how to apply the instructions I was given to fix it and I don't have days to way for one. So please help.
I increased the gunicorn workers to 4, hoping that would allow me to do the CSV export. No. So was given these instructions:
### The issues seems to be due to timeout not due to high load. You want to increase the timeout for gunicorn and you can do that by specifying a gunicorn.ini config file:
[webservice] configuration: /path/to/gunicorn.cfg
and then in the gunicorn.cfg you can add
[gunicorn] timeout = 90 graceful_timeout = 30 ###
I applied the above changes to /opt/mailman/core/mailman-extra.cfg and created the gunicorn.cfg file. Restarted the docker container (mailman-core) and no access to Postorius due to no REST api.
After restarting docker-mailman, I can't access Postorius due to the following error:
ERROR 2020-07-20 20:16:27,351 15 postorius Mailman REST API not available
Is Mailman core running? If not, whats in core's mailman.log as to why not?
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 7/21/20 8:55 PM, Mark Sapiro wrote:
On 7/21/20 5:40 PM, Brian Carpenter wrote:
Abhilash (@maxking) is one person and he has a $DayJob. We're all doing the best we can.
Well, I still need support.
This is not a Docker issue per se. It is a consequence of upgrading Mailman core to 3.3.1+ and processing many old, stale, previously unprocessed bounces. See <https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/thread/U5ZW6X5QLQRXUOMD2DN77CYCOLKVMUXJ/>
I am not worried about the bounce issues at this point. I am just trying to get a membership list exported for a client.
So, I want to totally move away from the use of Docker (mainly because I have found it buggy and the support is simply lacking) and install Mailman 3 for this client from source. However I need to get an export of their list members. But of course, I can't. Rest API times out and I don't know how to apply the instructions I was given to fix it and I don't have days to way for one. So please help.
I increased the gunicorn workers to 4, hoping that would allow me to do the CSV export. No. So was given these instructions:
### The issues seems to be due to timeout not due to high load. You want to increase the timeout for gunicorn and you can do that by specifying a gunicorn.ini config file:
[webservice] configuration: /path/to/gunicorn.cfg
and then in the gunicorn.cfg you can add
[gunicorn] timeout = 90 graceful_timeout = 30 ###
I applied the above changes to /opt/mailman/core/mailman-extra.cfg and created the gunicorn.cfg file. Restarted the docker container (mailman-core) and no access to Postorius due to no REST api.
After restarting docker-mailman, I can't access Postorius due to the following error:
ERROR 2020-07-20 20:16:27,351 15 postorius Mailman REST API not available Is Mailman core running? If not, whats in core's mailman.log as to why not?
It looks like but no error message in mailman.log. However I noted that the gunicorn process did not start.
This is with the configuration line enabled in mailman-extra.cfg:
# docker-compose restart mailman-core Restarting mailman-core ... done [root@nicu-net docker-mailman]# tail -f /opt/mailman/core/var/logs/mailman.log Jul 22 01:25:17 2020 (22) in runner caught SIGTERM. Stopping. Jul 22 01:25:17 2020 (25) out runner caught SIGTERM. Stopping. Jul 22 01:25:17 2020 (25) out runner exiting. Jul 22 01:25:17 2020 (21) command runner exiting. Jul 22 01:25:17 2020 (22) in runner exiting. Jul 22 01:25:17 2020 (24) nntp runner caught SIGTERM. Stopping. Jul 22 01:25:17 2020 (24) nntp runner exiting. [2020-07-22 01:25:19 +0000] [27] [INFO] Shutting down: Master Jul 22 01:25:21 2020 (1) Master stopped Jul 22 01:25:26 2020 (1) Master started Jul 22 01:25:44 2020 (27) rest runner started. Jul 22 01:25:44 2020 (26) pipeline runner started. Jul 22 01:25:44 2020 (20) bounces runner started. Jul 22 01:25:45 2020 (22) in runner started. Jul 22 01:25:45 2020 (28) retry runner started. Jul 22 01:25:46 2020 (29) virgin runner started. Jul 22 01:25:46 2020 (25) out runner started. Jul 22 01:25:46 2020 (23) lmtp runner started. Jul 22 01:25:46 2020 (24) nntp runner started. Jul 22 01:25:47 2020 (30) digest runner started. Jul 22 01:25:47 2020 (19) archive runner started. Jul 22 01:25:47 2020 (21) command runner started.
This is with the configuration line disabled in mailman-extra.cfg:
# docker-compose restart mailman-core Restarting mailman-core ... done [root@nicu-net docker-mailman]# tail -f /opt/mailman/core/var/logs/mailman.log Jul 22 01:27:49 2020 (1) Master stopped Jul 22 01:27:53 2020 (1) Master started Jul 22 01:28:10 2020 (30) digest runner started. Jul 22 01:28:10 2020 (29) virgin runner started. Jul 22 01:28:10 2020 (24) nntp runner started. Jul 22 01:28:10 2020 (25) out runner started. Jul 22 01:28:10 2020 (22) in runner started. Jul 22 01:28:11 2020 (28) retry runner started. Jul 22 01:28:11 2020 (27) rest runner started. [2020-07-22 01:28:11 +0000] [27] [INFO] Starting gunicorn 19.9.0 [2020-07-22 01:28:11 +0000] [27] [INFO] Listening at: http://172.19.199.2:8001 (27) [2020-07-22 01:28:11 +0000] [27] [INFO] Using worker: sync [2020-07-22 01:28:11 +0000] [32] [INFO] Booting worker with pid: 32 [2020-07-22 01:28:11 +0000] [33] [INFO] Booting worker with pid: 33 [2020-07-22 01:28:11 +0000] [34] [INFO] Booting worker with pid: 34 [2020-07-22 01:28:11 +0000] [35] [INFO] Booting worker with pid: 35 Jul 22 01:28:11 2020 (19) archive runner started. Jul 22 01:28:11 2020 (21) command runner started. Jul 22 01:28:12 2020 (23) lmtp runner started. Jul 22 01:28:12 2020 (20) bounces runner started. Jul 22 01:28:12 2020 (26) pipeline runner started.
-- Please let me know if you need further assistance.
Thank you for your business. We appreciate our clients. Brian Carpenter EMWD.com
-- EMWD's Knowledgebase: https://clientarea.emwd.com/index.php/knowledgebase
EMWD's Community Forums http://discourse.emwd.com/
Hi Brian,
You have been one of the good and generous people providing support. I hope i can repay a bit with this: These are one-line bash scripts I wrote after moving to MM3, so I could do things not yet in MM3, such as letting my moderators request a list’s members from me. I need one more script that will sync a text file to an existing list, deleting and adding as necessary.
The scripts are for public use, reuse, misuse and overuse. I claim no rights and no responsibilities.
mm3Members.bash #!/bin/bash # mm3Members # # Obtain a list of members of a given list # # Usage: # mm3Members.bash <listName> # # # Revisions: # Date Name Reason for change # 28-Mar-2020 Allan Hansen Initial # exec /opt/mailman/mm/venv/bin/mailman -C "/opt/mailman/mm/mailman.cfg" members $1 |grep -v -e "has no members"|sed -e 's/.*/\L&/‘
mm3ASddMembers.bash #!/bin/bash # mm3AddMembers # # Add members from a given list. # # Usage: # mm3AddMembers.bash <fileName> <listName> # <fileName>: text file with addresses to add. Same legal formats as in mass subscribe # # # Revisions: # Date Name Reason for change # 28-Mar-2020 Allan Hansen Initial # exec /opt/mailman/mm/venv/bin/mailman -C "/opt/mailman/mm/mailman.cfg" members --add $1 $2
mm3DeleteMembers.bash #!/bin/bash # mm3DeleteMembers # # Deletes members from a given list # <fileName> List of members to delete. Legal format the same a formats used for mass subscribes # # Usage: # mm3DeleteMembers.bash <fileName> <listName> # # Revisions: # # Date Name Reason for change # 28-Mar-2020 Allan Hansen Initial # exec /opt/mailman/mm/venv/bin/mailman -C "/opt/mailman/mm/mailman.cfg" members --delete $1 $2
mm3MemberCount.bash #!/bin/bash # mm3MemberCount # # Obtain count of the members of a given list # # Usage: # mm3MemberCount.bash <listName> # # Revisions: # Date Name Reason for change # 28-Mar-2020 Allan Hansen Initial # /home/hansen/scripts/mm3Members.bash $1|wc
Yours,
Allan Hansen hansen@rc.org
On Jul 21, 2020, at 18:31 , Brian Carpenter <brian_carpenter@emwd.com> wrote:
On 7/21/20 8:55 PM, Mark Sapiro wrote:
On 7/21/20 5:40 PM, Brian Carpenter wrote:
Abhilash (@maxking) is one person and he has a $DayJob. We're all doing the best we can.
Well, I still need support.
This is not a Docker issue per se. It is a consequence of upgrading Mailman core to 3.3.1+ and processing many old, stale, previously unprocessed bounces. See <https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/thread/U5ZW6X5QLQRXUOMD2DN77CYCOLKVMUXJ/>
I am not worried about the bounce issues at this point. I am just trying to get a membership list exported for a client.
So, I want to totally move away from the use of Docker (mainly because I have found it buggy and the support is simply lacking) and install Mailman 3 for this client from source. However I need to get an export of their list members. But of course, I can't. Rest API times out and I don't know how to apply the instructions I was given to fix it and I don't have days to way for one. So please help.
I increased the gunicorn workers to 4, hoping that would allow me to do the CSV export. No. So was given these instructions:
### The issues seems to be due to timeout not due to high load. You want to increase the timeout for gunicorn and you can do that by specifying a gunicorn.ini config file:
[webservice] configuration: /path/to/gunicorn.cfg
and then in the gunicorn.cfg you can add
[gunicorn] timeout = 90 graceful_timeout = 30 ###
I applied the above changes to /opt/mailman/core/mailman-extra.cfg and created the gunicorn.cfg file. Restarted the docker container (mailman-core) and no access to Postorius due to no REST api.
After restarting docker-mailman, I can't access Postorius due to the following error:
ERROR 2020-07-20 20:16:27,351 15 postorius Mailman REST API not available Is Mailman core running? If not, whats in core's mailman.log as to why not?
It looks like but no error message in mailman.log. However I noted that the gunicorn process did not start.
This is with the configuration line enabled in mailman-extra.cfg:
# docker-compose restart mailman-core Restarting mailman-core ... done [root@nicu-net docker-mailman]# tail -f /opt/mailman/core/var/logs/mailman.log Jul 22 01:25:17 2020 (22) in runner caught SIGTERM. Stopping. Jul 22 01:25:17 2020 (25) out runner caught SIGTERM. Stopping. Jul 22 01:25:17 2020 (25) out runner exiting. Jul 22 01:25:17 2020 (21) command runner exiting. Jul 22 01:25:17 2020 (22) in runner exiting. Jul 22 01:25:17 2020 (24) nntp runner caught SIGTERM. Stopping. Jul 22 01:25:17 2020 (24) nntp runner exiting. [2020-07-22 01:25:19 +0000] [27] [INFO] Shutting down: Master Jul 22 01:25:21 2020 (1) Master stopped Jul 22 01:25:26 2020 (1) Master started Jul 22 01:25:44 2020 (27) rest runner started. Jul 22 01:25:44 2020 (26) pipeline runner started. Jul 22 01:25:44 2020 (20) bounces runner started. Jul 22 01:25:45 2020 (22) in runner started. Jul 22 01:25:45 2020 (28) retry runner started. Jul 22 01:25:46 2020 (29) virgin runner started. Jul 22 01:25:46 2020 (25) out runner started. Jul 22 01:25:46 2020 (23) lmtp runner started. Jul 22 01:25:46 2020 (24) nntp runner started. Jul 22 01:25:47 2020 (30) digest runner started. Jul 22 01:25:47 2020 (19) archive runner started. Jul 22 01:25:47 2020 (21) command runner started.
This is with the configuration line disabled in mailman-extra.cfg:
# docker-compose restart mailman-core Restarting mailman-core ... done [root@nicu-net docker-mailman]# tail -f /opt/mailman/core/var/logs/mailman.log Jul 22 01:27:49 2020 (1) Master stopped Jul 22 01:27:53 2020 (1) Master started Jul 22 01:28:10 2020 (30) digest runner started. Jul 22 01:28:10 2020 (29) virgin runner started. Jul 22 01:28:10 2020 (24) nntp runner started. Jul 22 01:28:10 2020 (25) out runner started. Jul 22 01:28:10 2020 (22) in runner started. Jul 22 01:28:11 2020 (28) retry runner started. Jul 22 01:28:11 2020 (27) rest runner started. [2020-07-22 01:28:11 +0000] [27] [INFO] Starting gunicorn 19.9.0 [2020-07-22 01:28:11 +0000] [27] [INFO] Listening at: http://172.19.199.2:8001 (27) [2020-07-22 01:28:11 +0000] [27] [INFO] Using worker: sync [2020-07-22 01:28:11 +0000] [32] [INFO] Booting worker with pid: 32 [2020-07-22 01:28:11 +0000] [33] [INFO] Booting worker with pid: 33 [2020-07-22 01:28:11 +0000] [34] [INFO] Booting worker with pid: 34 [2020-07-22 01:28:11 +0000] [35] [INFO] Booting worker with pid: 35 Jul 22 01:28:11 2020 (19) archive runner started. Jul 22 01:28:11 2020 (21) command runner started. Jul 22 01:28:12 2020 (23) lmtp runner started. Jul 22 01:28:12 2020 (20) bounces runner started. Jul 22 01:28:12 2020 (26) pipeline runner started.
-- Please let me know if you need further assistance.
Thank you for your business. We appreciate our clients. Brian Carpenter EMWD.com
-- EMWD's Knowledgebase: https://clientarea.emwd.com/index.php/knowledgebase
EMWD's Community Forums http://discourse.emwd.com/
Mailman-users mailing list -- mailman-users@mailman3.org To unsubscribe send an email to mailman-users-leave@mailman3.org https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
On 7/21/20 6:31 PM, Brian Carpenter wrote:
This is with the configuration line enabled in mailman-extra.cfg:
# docker-compose restart mailman-core Restarting mailman-core ... done [root@nicu-net docker-mailman]# tail -f /opt/mailman/core/var/logs/mailman.log Jul 22 01:25:17 2020 (22) in runner caught SIGTERM. Stopping. Jul 22 01:25:17 2020 (25) out runner caught SIGTERM. Stopping. Jul 22 01:25:17 2020 (25) out runner exiting. Jul 22 01:25:17 2020 (21) command runner exiting. Jul 22 01:25:17 2020 (22) in runner exiting. Jul 22 01:25:17 2020 (24) nntp runner caught SIGTERM. Stopping. Jul 22 01:25:17 2020 (24) nntp runner exiting. [2020-07-22 01:25:19 +0000] [27] [INFO] Shutting down: Master Jul 22 01:25:21 2020 (1) Master stopped Jul 22 01:25:26 2020 (1) Master started Jul 22 01:25:44 2020 (27) rest runner started. Jul 22 01:25:44 2020 (26) pipeline runner started. Jul 22 01:25:44 2020 (20) bounces runner started. Jul 22 01:25:45 2020 (22) in runner started. Jul 22 01:25:45 2020 (28) retry runner started. Jul 22 01:25:46 2020 (29) virgin runner started. Jul 22 01:25:46 2020 (25) out runner started. Jul 22 01:25:46 2020 (23) lmtp runner started. Jul 22 01:25:46 2020 (24) nntp runner started. Jul 22 01:25:47 2020 (30) digest runner started. Jul 22 01:25:47 2020 (19) archive runner started. Jul 22 01:25:47 2020 (21) command runner started.
This is with the configuration line disabled in mailman-extra.cfg:
# docker-compose restart mailman-core Restarting mailman-core ... done [root@nicu-net docker-mailman]# tail -f /opt/mailman/core/var/logs/mailman.log Jul 22 01:27:49 2020 (1) Master stopped Jul 22 01:27:53 2020 (1) Master started Jul 22 01:28:10 2020 (30) digest runner started. Jul 22 01:28:10 2020 (29) virgin runner started. Jul 22 01:28:10 2020 (24) nntp runner started. Jul 22 01:28:10 2020 (25) out runner started. Jul 22 01:28:10 2020 (22) in runner started. Jul 22 01:28:11 2020 (28) retry runner started. Jul 22 01:28:11 2020 (27) rest runner started. [2020-07-22 01:28:11 +0000] [27] [INFO] Starting gunicorn 19.9.0 [2020-07-22 01:28:11 +0000] [27] [INFO] Listening at: http://172.19.199.2:8001 (27) [2020-07-22 01:28:11 +0000] [27] [INFO] Using worker: sync [2020-07-22 01:28:11 +0000] [32] [INFO] Booting worker with pid: 32 [2020-07-22 01:28:11 +0000] [33] [INFO] Booting worker with pid: 33 [2020-07-22 01:28:11 +0000] [34] [INFO] Booting worker with pid: 34 [2020-07-22 01:28:11 +0000] [35] [INFO] Booting worker with pid: 35 Jul 22 01:28:11 2020 (19) archive runner started. Jul 22 01:28:11 2020 (21) command runner started. Jul 22 01:28:12 2020 (23) lmtp runner started. Jul 22 01:28:12 2020 (20) bounces runner started. Jul 22 01:28:12 2020 (26) pipeline runner started.
Is there perhaps some permissions issue in reading your added gonicorn.cfg?
Also, note that gunicorn is set-up to log to the logging.http path, the
default for which is mailman.log in the configured log_dir, but if this
is changed to something other than mailman.log (see mailman conf -k path -s logging.http), it could be elsewhere. I'd expect gunicorn to
log something if it couldn't start.
Also, as Alan Hansen notes, you can export a list of members with
mailman members
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
participants (3)
-
Allan Hansen -
Brian Carpenter -
Mark Sapiro