mailman-web: ResponseNotReady
We've had an odd experience with mailman3 containers:
ERROR 2017-10-18 15:47:09,448 60 django.request Internal Server Error: /postorius/lists/ Traceback (most recent call last): File "/usr/local/lib/python2.7/site-packages/django/core/handlers/exception.py", line 39, in inner response = get_response(request) File "/usr/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 249, in _legacy_get_response response = self._get_response(request) File "/usr/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 187, in _get_response response = self.process_exception_by_middleware(e, request) File "/usr/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 185, in _get_response response = wrapped_callback(request, *callback_args, **callback_kwargs) File "/usr/local/lib/python2.7/site-packages/postorius/views/list.py", line 590, in list_index choosable_domains = _get_choosable_domains(request) File "/usr/local/lib/python2.7/site-packages/postorius/views/list.py", line 524, in _get_choosable_domains return [(d.mail_host, d.mail_host) for d in domains] File "/usr/local/lib/python2.7/site-packages/mailmanclient/_client.py", line 217, in __getattr__ return self._get(name) File "/usr/local/lib/python2.7/site-packages/mailmanclient/_client.py", line 183, in _get return self.rest_data.get(key) File "/usr/local/lib/python2.7/site-packages/mailmanclient/_client.py", line 172, in rest_data response, content = self._connection.call(self._url) File "/usr/local/lib/python2.7/site-packages/mailmanclient/_client.py", line 109, in call response, content = Http().request(url, method, data_str, headers) File "/usr/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1659, in request (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey) File "/usr/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1399, in _request (response, content) = self._conn_request(conn, request_uri, method, body, headers) File "/usr/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1355, in _conn_request response = conn.getresponse() File "/usr/local/lib/python2.7/httplib.py", line 1108, in getresponse raise ResponseNotReady() ResponseNotReady
It seems like mailman-core is still operating properly, but for some reason mailman-web is not. Can I reset mailman-web somehow to re-initiate?
-- Sr System and DevOps Engineer SoM IRT
This looks similar to what I got when I installed mailman using sudo but then tried to start it using my standard user. I don't have a working install of mailman 3 at the moment to try it again for sure.
-Erin
On Oct 18, 2017, at 9:00 AM, Dmitry Makovey <dmakovey@stanford.edu> wrote:
We've had an odd experience with mailman3 containers:
ERROR 2017-10-18 15:47:09,448 60 django.request Internal Server Error: /postorius/lists/ Traceback (most recent call last): File "/usr/local/lib/python2.7/site-packages/django/core/handlers/exception.py", line 39, in inner response = get_response(request) File "/usr/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 249, in _legacy_get_response response = self._get_response(request) File "/usr/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 187, in _get_response response = self.process_exception_by_middleware(e, request) File "/usr/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 185, in _get_response response = wrapped_callback(request, *callback_args, **callback_kwargs) File "/usr/local/lib/python2.7/site-packages/postorius/views/list.py", line 590, in list_index choosable_domains = _get_choosable_domains(request) File "/usr/local/lib/python2.7/site-packages/postorius/views/list.py", line 524, in _get_choosable_domains return [(d.mail_host, d.mail_host) for d in domains] File "/usr/local/lib/python2.7/site-packages/mailmanclient/_client.py", line 217, in __getattr__ return self._get(name) File "/usr/local/lib/python2.7/site-packages/mailmanclient/_client.py", line 183, in _get return self.rest_data.get(key) File "/usr/local/lib/python2.7/site-packages/mailmanclient/_client.py", line 172, in rest_data response, content = self._connection.call(self._url) File "/usr/local/lib/python2.7/site-packages/mailmanclient/_client.py", line 109, in call response, content = Http().request(url, method, data_str, headers) File "/usr/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1659, in request (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey) File "/usr/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1399, in _request (response, content) = self._conn_request(conn, request_uri, method, body, headers) File "/usr/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1355, in _conn_request response = conn.getresponse() File "/usr/local/lib/python2.7/httplib.py", line 1108, in getresponse raise ResponseNotReady() ResponseNotReady
It seems like mailman-core is still operating properly, but for some reason mailman-web is not. Can I reset mailman-web somehow to re-initiate?
-- Sr System and DevOps Engineer SoM IRT
Mailman-users mailing list mailman-users@mailman3.org https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
On 10/18/2017 09:59 AM, Erin Justice wrote:
This looks similar to what I got when I installed mailman using sudo but then tried to start it using my standard user. I don't have a working install of mailman 3 at the moment to try it again for sure.
thanks for the hint Erin, unfortunately it doesn't seem like that is the case - checked all the file permissions within container - they are all OK. For the off-chance I even forced permission change - no go :(
-- Sr System and DevOps Engineer SoM IRT
On 10/18/2017 09:00 AM, Dmitry Makovey wrote:
It seems like mailman-core is still operating properly, but for some reason mailman-web is not. Can I reset mailman-web somehow to re-initiate?
I'm not very well versed in the docker images, but I think mailman-web uses uWsgi and sending the uwsgi process a SIGHUP should do it.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 10/18/2017 12:28 PM, Mark Sapiro wrote:
On 10/18/2017 09:00 AM, Dmitry Makovey wrote:
It seems like mailman-core is still operating properly, but for some reason mailman-web is not. Can I reset mailman-web somehow to re-initiate?
I'm not very well versed in the docker images, but I think mailman-web uses uWsgi and sending the uwsgi process a SIGHUP should do it.
I think I didn't express myself clearly - I want to reset any saved state for mailman-web - not just in-memory stuff. I've restarted my setup several times with mailman-web having exact same problem every time.
-- Sr System and DevOps Engineer SoM IRT
Hi,
$ docker-compose rm -f mailman-web $ docker-compose up -d mailman-web
This will basically create a new container for Mailman-web, but, since the state of mailman-web is stored completely in database (database container), I am not sure how to reset *just* mailman-web without dropping mailman-core related tables too (they share the same database but have their own separate tables). The above command will get you a fresh copy of code running.
You can potentially get the list of tables and drop them, or maybe Diango has some way to reset it's state from Django shell (python manage.py shell, from inside mailman-web will get you shell). I may have to look it up.
Also, just by looking at the traceback, I am not sure about the exact error. I may have to debug more to arrive at a better conclusion, which I won't able to do until after this weekend.
Thanks, Abhilash
On October 18, 2017 1:27:21 PM PDT, Dmitry Makovey <dmakovey@stanford.edu> wrote:
On 10/18/2017 12:28 PM, Mark Sapiro wrote:
On 10/18/2017 09:00 AM, Dmitry Makovey wrote:
It seems like mailman-core is still operating properly, but for some reason mailman-web is not. Can I reset mailman-web somehow to
re-initiate?
I'm not very well versed in the docker images, but I think mailman-web uses uWsgi and sending the uwsgi process a SIGHUP should do it.
I think I didn't express myself clearly - I want to reset any saved state for mailman-web - not just in-memory stuff. I've restarted my setup several times with mailman-web having exact same problem every time.
-- Sr System and DevOps Engineer SoM IRT
-- Sent from my Android device with K-9 Mail. Please excuse my brevity.
On 10/18/2017 01:41 PM, Abhilash Raj wrote:
Hi,
$ docker-compose rm -f mailman-web $ docker-compose up -d mailman-web
This will basically create a new container for Mailman-web, but, since the state of mailman-web is stored completely in database (database container), I am not sure how to reset *just* mailman-web without dropping mailman-core related tables too (they share the same database but have their own separate tables). The above command will get you a fresh copy of code running.
You can potentially get the list of tables and drop them, or maybe Diango has some way to reset it's state from Django shell (python manage.py <http://manage.py> shell, from inside mailman-web will get you shell). I may have to look it up.
Also, just by looking at the traceback, I am not sure about the exact error. I may have to debug more to arrive at a better conclusion, which I won't able to do until after this weekend.
thanks for the suggestions. I'll try to dig in direction outlined by you, may end up producing some code to automate such task in the future :)
-- Sr System and DevOps Engineer SoM IRT
On 10/18/2017 01:41 PM, Abhilash Raj wrote:
Hi,
$ docker-compose rm -f mailman-web $ docker-compose up -d mailman-web
This will basically create a new container for Mailman-web, but, since the state of mailman-web is stored completely in database (database container), I am not sure how to reset *just* mailman-web without dropping mailman-core related tables too (they share the same database but have their own separate tables). The above command will get you a fresh copy of code running.
You can potentially get the list of tables and drop them, or maybe Diango has some way to reset it's state from Django shell (python manage.py <http://manage.py> shell, from inside mailman-web will get you shell). I may have to look it up.
Also, just by looking at the traceback, I am not sure about the exact error. I may have to debug more to arrive at a better conclusion, which I won't able to do until after this weekend.
after a full remove of the DB - I still get this error. I left some of the directories intact but for the most part it's a fresh new install that gets the same error where previous didn't give us as much trouble. Something else must be messed up. logs show very little detail other than originally posted exception. I'll try to dig up some more info...
-- Sr System and DevOps Engineer SoM IRT
On 10/19/2017 10:14 AM, Dmitry Makovey wrote:
after a full remove of the DB - I still get this error. I left some of the directories intact but for the most part it's a fresh new install that gets the same error where previous didn't give us as much trouble. Something else must be messed up. logs show very little detail other than originally posted exception. I'll try to dig up some more info...
running mailman-web with DEBUG=True it looks like issue is rising from here:
does it mean we're setting domains somehow improperly?
-- Sr System and DevOps Engineer SoM IRT
On 10/19/2017 01:30 PM, Dmitry Makovey wrote:
On 10/19/2017 10:14 AM, Dmitry Makovey wrote:
after a full remove of the DB - I still get this error. I left some of the directories intact but for the most part it's a fresh new install that gets the same error where previous didn't give us as much trouble. Something else must be messed up. logs show very little detail other than originally posted exception. I'll try to dig up some more info...
running mailman-web with DEBUG=True it looks like issue is rising from here:
does it mean we're setting domains somehow improperly?
Adding info here for posterity:
turns out problem stems from much deeper. Since we're running containerized mailman deployment previously I've posted that for mailman-core I've had to resort to a trick making it think it runs off the public FQDN for the host:
services: mailman-core: image: maxking/mailman-core:0.1 container_name: mailman-core hostname: $CORE_HOSTNAME domainname: $CORE_DOMAINNAME
however it turns out that REST calls to mailman-core are adding abovementioned FQDN to the responses. For some reason Docker/iptables dind't allow mailman-web to talk to the public IP of the host directly, so the solution was to extend above trick a bit further:
and add to mailman-core definition an alias:
services: mailman-core: image: maxking/mailman-core:0.1 container_name: mailman-core hostname: $CORE_HOSTNAME domainname: $CORE_DOMAINNAME ... networks: mailman: aliases: - $CORE_HOSTNAME.$CORE_DOMAINNAME
now mailman-web can properly communicate with mailman-core using either of two names: mailman-core and $CORE_HOSTNAME.$CORE_DOMAINNAME
-- Sr System and DevOps Engineer SoM IRT
participants (4)
-
Abhilash Raj -
Dmitry Makovey -
Erin Justice -
Mark Sapiro