I'm working on a container based Mailman3 solution for my university. We have a help desk tool which needs to access the API remotely for creating and deleting Mailman lists. The API will be restricted to a private network, behind a firewall, etc.
What is the simplest way to expose the API to an external host?
At the moment, I seem to have this partially working via an Apache virtual host configuration which simply proxies via api.example.edu:80 to localhost:8001 on the container.
I'm able to remotely use the mailmanclient module for the client.lists, client.system and client.members commands. However running client.domains generates an API connection error. So it seems unstable.
Any thoughts or suggestions?
Thanks.
Dan Caballero Systems Administrator Academic Computing Solutions IMSS - Caltech https://imss.caltech.edu
dancab@caltech.edu writes:
What is the simplest way to expose the API to an external host?
At the moment, I seem to have this partially working via an Apache virtual host configuration which simply proxies via api.example.edu:80 to localhost:8001 on the container.
An alternative would be an ssh tunnel. That might not be as simply for your users though.
I'm able to remotely use the mailmanclient module for the client.lists, client.system and client.members commands. However running client.domains generates an API connection error. So it seems unstable.
That doesn't sound like instability, that sounds like an API mismatch. Have you tried logging in on the container and using mailman shell and/or the Python interpreter (and importing mailmanclient)?
Have you updated either core or mailmanclient but not both?
If you can't think of anything you may have done to create an API mismatch (such as partial update or patching), please file this as a bug on gitlab.
Steve
Thanks Stephen. I'm able to run the mailmanclient with no issues from the container via localhost:8001.
So I will look into this API mismatch issue you suggest. My tests so far have been from an iMac where I used brew install to get Python 3 and pip3 running to set up mailmanclient.
"That doesn't sound like instability, that sounds like an API mismatch. Have you tried logging in on the container and using mailman shell and/or the Python interpreter (and importing mailmanclient)?"
Dan Caballero Systems Administrator Academic Computing Solutions IMSS - Caltech https://imss.caltech.edu
From: Stephen J. Turnbull <turnbull.stephen.fw@u.tsukuba.ac.jp> Sent: Monday, July 8, 2019 8:42 PM To: Caballero, Danny (Dan) Cc: mailman-users@mailman3.org Subject: [MM3-users] exposing Mailman3 API securely
dancab@caltech.edu writes:
What is the simplest way to expose the API to an external host?
At the moment, I seem to have this partially working via an Apache virtual host configuration which simply proxies via api.example.edu:80 to localhost:8001 on the container.
An alternative would be an ssh tunnel. That might not be as simply for your users though.
I'm able to remotely use the mailmanclient module for the client.lists, client.system and client.members commands. However running client.domains generates an API connection error. So it seems unstable.
That doesn't sound like instability, that sounds like an API mismatch. Have you tried logging in on the container and using mailman shell and/or the Python interpreter (and importing mailmanclient)?
Have you updated either core or mailmanclient but not both?
If you can't think of anything you may have done to create an API mismatch (such as partial update or patching), please file this as a bug on gitlab.
Steve
On Mon, Jul 8, 2019, at 3:12 PM, dancab@caltech.edu wrote:
I'm working on a container based Mailman3 solution for my university. We have a help desk tool which needs to access the API remotely for creating and deleting Mailman lists. The API will be restricted to a private network, behind a firewall, etc.
What is the simplest way to expose the API to an external host?
I do want to stress the fact that we wouldn't advise exposing the API to internet since there is a very basic authentication in the API an the API is an administrative API.
At the moment, I seem to have this partially working via an Apache virtual host configuration which simply proxies via
api.example.edu:80 to localhost:8001 on the container.I'm able to remotely use the mailmanclient module for the client.lists, client.system and client.members commands. However running client.domains generates an API connection error. So it seems unstable.
Does it *always* happen when you run client.domains? Did you get
an error traceback?
Any thoughts or suggestions?
Thanks.
Dan Caballero Systems Administrator Academic Computing Solutions IMSS - Caltech https://imss.caltech.edu
Mailman-users mailing list -- mailman-users@mailman3.org To unsubscribe send an email to mailman-users-leave@mailman3.org https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
-- thanks, Abhilash Raj (maxking)
Thanks Abhilash. The plan is to only expose the API via our private subnets in AWS and using a security group to restrict access to a specific instance(s).
I just tried setting up mailmanclient on a Linux system running Python 3.6.3 and had a similar issue trying to run client.domains via mailmanclient. Note: My test instance hosting the Mailman API is running Python 3.6.8.
See client.system output below.
client.system {'mailman_version': 'GNU Mailman 3.2.2 (La Villa Strangiato)', 'python_version': '3.6.8 (default, Apr 25 2019, 21:02:35) \n[GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]', 'api_version': '3.1', 'self_link': 'http://localhost:8001/3.1/system/versions', 'http_etag': '"a4e539145192f2cd02e837a2b905583fe4649fdc"'}
Here's the full error traceback for client.domains. The significant lines indicate some kind of socket error.
client.domains Traceback (most recent call last): File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/mailmanclient/restbase/connection.py", line 95, in call response, content = Http().request(url, method, data_str, headers) File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/httplib2/__init__.py", line 1926, in request cachekey, File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/httplib2/__init__.py", line 1595, in _request conn, request_uri, method, body, headers File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/httplib2/__init__.py", line 1501, in _conn_request conn.connect() File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/httplib2/__init__.py", line 1207, in connect raise socket_err File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/httplib2/__init__.py", line 1181, in connect self.sock.connect((self.host, self.port) + sa[2:]) ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/mailmanclient/restobjects/domain.py", line 35, in __repr__ return '<Domain {0!r}>'.format(self.mail_host) File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/mailmanclient/restbase/base.py", line 121, in __getattr__ return self._get(name) File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/mailmanclient/restbase/base.py", line 87, in _get return self.rest_data.get(key) File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/mailmanclient/restbase/base.py", line 76, in rest_data response, content = self._connection.call(self._url) File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/mailmanclient/restbase/connection.py", line 109, in call raise MailmanConnectionError('Could not connect to Mailman API') mailmanclient.restbase.connection.MailmanConnectionError: Could not connect to Mailman API
It seems my Python output messages didn't post fully. I'll try quoting them instead.
">>> client.system {'mailman_version': 'GNU Mailman 3.2.2 (La Villa Strangiato)', 'python_version': '3.6.8 (default, Apr 25 2019, 21:02:35) \n[GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]', 'api_version': '3.1', 'self_link': 'http://localhost:8001/3.1/system/versions', 'http_etag': '"a4e539145192f2cd02e837a2b905583fe4649fdc"'}"
">>> client.domains Traceback (most recent call last): File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/mailmanclient/restbase/connection.py", line 95, in call response, content = Http().request(url, method, data_str, headers) File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/httplib2/__init__.py", line 1926, in request cachekey, File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/httplib2/__init__.py", line 1595, in _request conn, request_uri, method, body, headers File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/httplib2/__init__.py", line 1501, in _conn_request conn.connect() File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/httplib2/__init__.py", line 1207, in connect raise socket_err File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/httplib2/__init__.py", line 1181, in connect self.sock.connect((self.host, self.port) + sa[2:]) ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/mailmanclient/restobjects/domain.py", line 35, in __repr__ return '<Domain {0!r}>'.format(self.mail_host) File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/mailmanclient/restbase/base.py", line 121, in __getattr__ return self._get(name) File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/mailmanclient/restbase/base.py", line 87, in _get return self.rest_data.get(key) File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/mailmanclient/restbase/base.py", line 76, in rest_data response, content = self._connection.call(self._url) File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/mailmanclient/restbase/connection.py", line 109, in call raise MailmanConnectionError('Could not connect to Mailman API') mailmanclient.restbase.connection.MailmanConnectionError: Could not connect to Mailman API"
On Wed, Jul 10, 2019, at 10:55 AM, dancab@caltech.edu wrote:
It seems my Python output messages didn't post fully. I'll try quoting them instead.
I came out fine on my mail client the first time.
Anyway, this seems like a weird problem because everything is same when
running client.system and client.domains, the connection path, the API
call etc.
Can see if you can call the API using curl or something? The command I *think* should be:
$ curl -u <user>:<password> http://localhost:8001/3.1/domains">>> client.system {'mailman_version': 'GNU Mailman 3.2.2 (La Villa Strangiato)', 'python_version': '3.6.8 (default, Apr 25 2019, 21:02:35) \n[GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]', 'api_version': '3.1', 'self_link': 'http://localhost:8001/3.1/system/versions', 'http_etag': '"a4e539145192f2cd02e837a2b905583fe4649fdc"'}"
">>> client.domains Traceback (most recent call last): File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/mailmanclient/restbase/connection.py", line 95, in call response, content = Http().request(url, method, data_str, headers) File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/httplib2/__init__.py", line 1926, in request cachekey, File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/httplib2/__init__.py", line 1595, in _request conn, request_uri, method, body, headers File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/httplib2/__init__.py", line 1501, in _conn_request conn.connect() File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/httplib2/__init__.py", line 1207, in connect raise socket_err File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/httplib2/__init__.py", line 1181, in connect self.sock.connect((self.host, self.port) + sa[2:]) ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/mailmanclient/restobjects/domain.py", line 35, in __repr__ return '<Domain {0!r}>'.format(self.mail_host) File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/mailmanclient/restbase/base.py", line 121, in __getattr__ return self._get(name) File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/mailmanclient/restbase/base.py", line 87, in _get return self.rest_data.get(key) File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/mailmanclient/restbase/base.py", line 76, in rest_data response, content = self._connection.call(self._url) File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/mailmanclient/restbase/connection.py", line 109, in call raise MailmanConnectionError('Could not connect to Mailman API') mailmanclient.restbase.connection.MailmanConnectionError: Could not connect to Mailman API"
Mailman-users mailing list -- mailman-users@mailman3.org To unsubscribe send an email to mailman-users-leave@mailman3.org https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
-- thanks, Abhilash Raj (maxking)
Accessing the API via localhost works fine and I haven't encountered any issues. It's only with my 'proxy' solution for remote access that I have problems.
However, using curl seems to work either way.
This is the curl output via the Docker container connecting directly to localhost:8001.
{"start": 0, "total_size": 1, "entries": [{"alias_domain": "", "description": "Caltech Mailing List Service", "mail_host": "caltech.edu", "self_link": "http://localhost:8001/3.1/domains/caltech.edu", "http_etag": "\"e855d96f0362738a0f174fd24c8595b62761d496\""}], "http_etag": "\"333b74d5a650c199065262450afc1f5521f87db2\""}
This is the curl output connecting from my desktop to the same container remotely via Apache virtual host proxy. It seems fine!
{"start": 0, "total_size": 1, "entries": [{"alias_domain": "", "description": "Caltech Mailing List Service", "mail_host": "caltech.edu", "self_link": "http://localhost:8001/3.1/domains/caltech.edu", "http_etag": "\"e855d96f0362738a0f174fd24c8595b62761d496\""}], "http_etag": "\"333b74d5a650c199065262450afc1f5521f87db2\""}
Any ideas?
On Tue, Jul 16, 2019, at 3:06 PM, dancab@caltech.edu wrote:
Accessing the API via localhost works fine and I haven't encountered any issues. It's only with my 'proxy' solution for remote access that I have problems.
However, using curl seems to work either way.
This is the curl output via the Docker container connecting directly to localhost:8001.
{"start": 0, "total_size": 1, "entries": [{"alias_domain": "", "description": "Caltech Mailing List Service", "mail_host": "caltech.edu", "self_link": "http://localhost:8001/3.1/domains/caltech.edu", "http_etag": "\"e855d96f0362738a0f174fd24c8595b62761d496\""}], "http_etag": "\"333b74d5a650c199065262450afc1f5521f87db2\""}
This is the curl output connecting from my desktop to the same container remotely via Apache virtual host proxy. It seems fine!
{"start": 0, "total_size": 1, "entries": [{"alias_domain": "", "description": "Caltech Mailing List Service", "mail_host": "caltech.edu", "self_link": "http://localhost:8001/3.1/domains/caltech.edu", "http_etag": "\"e855d96f0362738a0f174fd24c8595b62761d496\""}], "http_etag": "\"333b74d5a650c199065262450afc1f5521f87db2\""}
Does it *always* happen when running client.domains or does it
happen randomly on any client. call?
I was thinking if there could be an issue with total number of open connections over the ssh tunnel that you have setup, causing connection refused after a limit has been reached. max_sessions and max_startup options could help with that. I'd have to read up more on how the sessions work with ssh tunnel.
Any ideas?
This is a totally wild guess, but maybe using a different HTTP library could change something. The latest master of mailmanclient has moved to using requests as compared to Httplib2 from your installed version.
You can install a test version from Gitlab's master branch[1]. I will soon cut out a release, but that should help debug the problem.
[1] https://gitlab.com/mailman/mailmanclient
-- thanks, Abhilash Raj (maxking)
Ok, I will try to test this newer version of Mailman Client.
For what it's worth, yes I have tried a variety of commands... client.lists, client.member, client.system, etc. via my remote connection. But only client.domain has given me a problem.
I was out on vacation for a couple of weeks, so just getting back around to this.
I downloaded the latest Mailman Client from Gitlab and installed it.
I'm now getting slightly different errors. See output below.
client.system {'mailman_version': 'GNU Mailman 3.2.2 (La Villa Strangiato)', 'python_version': '3.6.8 (default, Apr 25 2019, 21:02:35) \n[GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]', 'api_version': '3.1', 'self_link': 'http://localhost:8001/3.1/system/versions', 'http_etag': '"a4e539145192f2cd02e837a2b905583fe4649fdc"'}
">>> client.domains Traceback (most recent call last): File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/urllib3/connection.py", line 159, in _new_conn (self._dns_host, self.port), self.timeout, **extra_kw) File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/urllib3/util/connection.py", line 80, in create_connection raise err File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/urllib3/util/connection.py", line 70, in create_connection sock.connect(sa) ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen chunked=chunked) File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 354, in _make_request conn.request(method, url, **httplib_request_kw) File "/opt/rh/rh-python36/root/usr/lib64/python3.6/http/client.py", line 1239, in request self._send_request(method, url, body, headers, encode_chunked) File "/opt/rh/rh-python36/root/usr/lib64/python3.6/http/client.py", line 1285, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/opt/rh/rh-python36/root/usr/lib64/python3.6/http/client.py", line 1234, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/opt/rh/rh-python36/root/usr/lib64/python3.6/http/client.py", line 1026, in _send_output self.send(msg) File "/opt/rh/rh-python36/root/usr/lib64/python3.6/http/client.py", line 964, in send self.connect() File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/urllib3/connection.py", line 181, in connect conn = self._new_conn() File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/urllib3/connection.py", line 168, in _new_conn self, "Failed to establish a new connection: %s" % e) urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f0df9030dd8>: Failed to establish a new connection: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/requests/adapters.py", line 449, in send timeout=timeout File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen _stacktrace=sys.exc_info()[2]) File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/urllib3/util/retry.py", line 398, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=8001): Max retries exceeded with url: /3.1/domains/caltech.edu (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f0df9030dd8>: Failed to establish a new connection: [Errno 111] Connection refused',))
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/mailmanclient/restbase/connection.py", line 96, in call headers=headers) File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/requests/api.py", line 60, in request return session.request(method=method, url=url, **kwargs) File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/requests/sessions.py", line 533, in request resp = self.send(prep, **send_kwargs) File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/requests/sessions.py", line 646, in send r = adapter.send(request, **kwargs) File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/requests/adapters.py", line 516, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=8001): Max retries exceeded with url: /3.1/domains/caltech.edu (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f0df9030dd8>: Failed to establish a new connection: [Errno 111] Connection refused',))
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/mailmanclient/restobjects/domain.py", line 35, in __repr__ return '<Domain {0!r}>'.format(self.mail_host) File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/mailmanclient/restbase/base.py", line 119, in __getattr__ return self._get(name) File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/mailmanclient/restbase/base.py", line 85, in _get return self.rest_data.get(key) File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/mailmanclient/restbase/base.py", line 74, in rest_data response, content = self._connection.call(self._url) File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/mailmanclient/restbase/connection.py", line 119, in call raise MailmanConnectionError('Could not connect to Mailman API') mailmanclient.restbase.connection.MailmanConnectionError: Could not connect to Mailman API"
participants (4)
-
Abhilash Raj -
Caballero, Danny (Dan) -
dancab＠caltech.edu
-
Stephen J. Turnbull