Hi,
I try to connect mailman3 to hyperkitty, but there is no success so far. You can see the details below. How can I solve the problem?
As I have several domain name (like lists.xxx.hu, xxx.hu, yyy.hu) on the same server, so I guess localhost won't work.
In mailman-hyperkitty.cfg there is a line:
base_url: http://lists.xxx.hu/hyperkitty/
/etc/mailman3/mailman-web.py includes "*" in ALLOWED_HOSTS
I have an error message in /var/log/mailman3/mailman.log
Dec 30 21:41:23 2021 (126976) HyperKitty failure on http://lists.xxx.hu/hyperkitty/api/mailman/archive: (405) Dec 30 21:41:23 2021 (126976) Exception in the HyperKitty archiver: Dec 30 21:41:23 2021 (126976) Traceback (most recent call last): File "/usr/lib/python3/dist-packages/mailman_hyperkitty/__init__.py", line 154, in _archive_message url = self._send_message(mlist, msg) File "/usr/lib/python3/dist-packages/mailman_hyperkitty/__init__.py", line 210, in _send_message raise ValueError(result.text) ValueError
In /var/log/mailman3/web/mailman-web.log
WARNING 2021-12-30 20:41:23,816 127032 django.request Method Not Allowed (GET): /hyperkitty/api/mailman/archive [pid: 127032|app: 0|req: 101/101] 109.230.236.204 () {38 vars in 592 bytes} [Thu Dec 30 20:41:23 2021] GET /hyperkitty/api/mailman/archive?key=lHzJB%2FzfSWUrbPKZ84JNSlLXHVyCDt8T => generated 0 bytes in 1 msecs (HTTP/1.1 405) 6 headers in 189 bytes (1 switches on core 0) WARNING 2021-12-30 20:41:23,854 127032 django.request Method Not Allowed (GET): /hyperkitty/api/mailman/archive
On Debian 11, I have these packages
ii mailman3 3.3.3-1 all Mailing list management system ii mailman3-full 3.3.3-1 all Full Mailman3 mailing list management suite (metapackage) ii mailman3-web 0+20200530-2 all Django project integrating Mailman3 Postorius and HyperKitty ii python3-django-mailman3 1.3.5-2 all Django library to help interaction with Mailman3 (Python 3 version) ii python3-mailman-hyperkitty 1.1.0-10 all Mailman3 plugin to archive emails with HyperKitty
Thanks, Arpad
On Thu, 2021-12-30 at 21:19 +0000, horvath.arpad.szfvar@gmail.com wrote:
Hi,
I try to connect mailman3 to hyperkitty, but there is no success so far. You can see the details below. How can I solve the problem?
As I have several domain name (like lists.xxx.hu, xxx.hu, yyy.hu) on the same server, so I guess localhost won't work.
In mailman-hyperkitty.cfg there is a line:
base_url: http://lists.xxx.hu/hyperkitty/
/etc/mailman3/mailman-web.py includes "*" in ALLOWED_HOSTS
I have an error message in /var/log/mailman3/mailman.log
Dec 30 21:41:23 2021 (126976) HyperKitty failure on http://lists.xxx.hu/hyperkitty/api/mailman/archive: (405) Dec 30 21:41:23 2021 (126976) Exception in the HyperKitty archiver: Dec 30 21:41:23 2021 (126976) Traceback (most recent call last): File "/usr/lib/python3/dist-packages/mailman_hyperkitty/__init__.py", line 154, in _archive_message url = self._send_message(mlist, msg) File "/usr/lib/python3/dist-packages/mailman_hyperkitty/__init__.py", line 210, in _send_message raise ValueError(result.text) ValueError
[snip]
I'm no expert, but here's a previous discussion I found:
https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/thread/P...
I don't know if it applies, but it looks close.
billo
On 12/30/21 1:30 PM, William Oliver wrote:
On Thu, 2021-12-30 at 21:19 +0000, horvath.arpad.szfvar@gmail.com wrote:
Hi,
I try to connect mailman3 to hyperkitty, but there is no success so far. You can see the details below. How can I solve the problem?
As I have several domain name (like lists.xxx.hu, xxx.hu, yyy.hu) on the same server, so I guess localhost won't work.
In mailman-hyperkitty.cfg there is a line:
base_url: http://lists.xxx.hu/hyperkitty/
As noted in the thread that William references and also the thread at https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/thread/L... the issue is probably that the above http url is being redirected to https and losing POST data. Set the above to
base_url: https://lists.xxx.hu/hyperkitty/
[snip]
I'm no expert, but here's a previous discussion I found:
https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/thread/P...
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
I changed base_url to the https one, reloaded nginx, restarted mailman3-web.
Now there is an other error message in mailman.log (I can't find the a meaningful result for InvalidChunkLength error message in this list and in google search)
Jan 01 10:54:40 2022 (126976) Connection to HyperKitty failed: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read)) Jan 01 10:54:40 2022 (126976) Exception in the HyperKitty archiver: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read)) Jan 01 10:54:40 2022 (126976) Traceback (most recent call last): File "/usr/lib/python3/dist-packages/urllib3/response.py", line 697, in _update_chunk_length self.chunk_left = int(line, 16) ValueError: invalid literal for int() with base 16: b''
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/usr/lib/python3/dist-packages/urllib3/response.py", line 438, in _error_catcher yield File "/usr/lib/python3/dist-packages/urllib3/response.py", line 764, in read_chunked self._update_chunk_length() File "/usr/lib/python3/dist-packages/urllib3/response.py", line 701, in _update_chunk_length raise InvalidChunkLength(self, line) urllib3.exceptions.InvalidChunkLength: InvalidChunkLength(got length b'', 0 bytes read)
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/usr/lib/python3/dist-packages/requests/models.py", line 753, in generate for chunk in self.raw.stream(chunk_size, decode_content=True): File "/usr/lib/python3/dist-packages/urllib3/response.py", line 572, in stream for line in self.read_chunked(amt, decode_content=decode_content): File "/usr/lib/python3/dist-packages/urllib3/response.py", line 793, in read_chunked self._original_response.close() File "/usr/lib/python3.9/contextlib.py", line 135, in __exit__ self.gen.throw(type, value, traceback) File "/usr/lib/python3/dist-packages/urllib3/response.py", line 455, in _error_catcher raise ProtocolError("Connection broken: %r" % e, e) urllib3.exceptions.ProtocolError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/usr/lib/python3/dist-packages/mailman_hyperkitty/__init__.py", line 154, in _archive_message url = self._send_message(mlist, msg) File "/usr/lib/python3/dist-packages/mailman_hyperkitty/__init__.py", line 198, in _send_message result = requests.post( File "/usr/lib/python3/dist-packages/requests/api.py", line 119, in post return request('post', url, data=data, json=json, **kwargs) File "/usr/lib/python3/dist-packages/requests/api.py", line 61, in request return session.request(method=method, url=url, **kwargs) File "/usr/lib/python3/dist-packages/requests/sessions.py", line 542, in request resp = self.send(prep, **send_kwargs) File "/usr/lib/python3/dist-packages/requests/sessions.py", line 697, in send r.content File "/usr/lib/python3/dist-packages/requests/models.py", line 831, in content self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b'' File "/usr/lib/python3/dist-packages/requests/models.py", line 756, in generate raise ChunkedEncodingError(e) requests.exceptions.ChunkedEncodingError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read)) Jan 01 10:54:40 2022 (126976) Connection to HyperKitty failed: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read)) Jan 01 10:54:40 2022 (126976) Exception in the HyperKitty archiver: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read)) Jan 01 10:54:40 2022 (126976) Traceback (most recent call last): File "/usr/lib/python3/dist-packages/urllib3/response.py", line 697, in _update_chunk_length self.chunk_left = int(line, 16) ValueError: invalid literal for int() with base 16: b''
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/usr/lib/python3/dist-packages/urllib3/response.py", line 438, in _error_catcher yield File "/usr/lib/python3/dist-packages/urllib3/response.py", line 764, in read_chunked self._update_chunk_length() File "/usr/lib/python3/dist-packages/urllib3/response.py", line 701, in _update_chunk_length raise InvalidChunkLength(self, line) urllib3.exceptions.InvalidChunkLength: InvalidChunkLength(got length b'', 0 bytes read)
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/usr/lib/python3/dist-packages/requests/models.py", line 753, in generate for chunk in self.raw.stream(chunk_size, decode_content=True): File "/usr/lib/python3/dist-packages/urllib3/response.py", line 572, in stream for line in self.read_chunked(amt, decode_content=decode_content): File "/usr/lib/python3/dist-packages/urllib3/response.py", line 793, in read_chunked self._original_response.close() File "/usr/lib/python3.9/contextlib.py", line 135, in __exit__ self.gen.throw(type, value, traceback) File "/usr/lib/python3/dist-packages/urllib3/response.py", line 455, in _error_catcher raise ProtocolError("Connection broken: %r" % e, e) urllib3.exceptions.ProtocolError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/usr/lib/python3/dist-packages/mailman_hyperkitty/__init__.py", line 154, in _archive_message url = self._send_message(mlist, msg) File "/usr/lib/python3/dist-packages/mailman_hyperkitty/__init__.py", line 198, in _send_message result = requests.post( File "/usr/lib/python3/dist-packages/requests/api.py", line 119, in post return request('post', url, data=data, json=json, **kwargs) File "/usr/lib/python3/dist-packages/requests/api.py", line 61, in request return session.request(method=method, url=url, **kwargs) File "/usr/lib/python3/dist-packages/requests/sessions.py", line 542, in request resp = self.send(prep, **send_kwargs) File "/usr/lib/python3/dist-packages/requests/sessions.py", line 697, in send r.content File "/usr/lib/python3/dist-packages/requests/models.py", line 831, in content self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b'' File "/usr/lib/python3/dist-packages/requests/models.py", line 756, in generate raise ChunkedEncodingError(e) requests.exceptions.ChunkedEncodingError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))
In the /var/log/mailman3/smtp.log I have at the same time:
Jan 01 10:54:37 2022 (126980) Available AUTH mechanisms: LOGIN(builtin) PLAIN(builtin) Jan 01 10:54:37 2022 (126980) Peer: ('127.0.0.1', 54816) Jan 01 10:54:37 2022 (126980) ('127.0.0.1', 54816) handling connection Jan 01 10:54:37 2022 (126980) ('127.0.0.1', 54816) Data: b'LHLO yyy.hu' Jan 01 10:54:37 2022 (126980) ('127.0.0.1', 54816) Data: b'MAIL FROM:horvath.arpad.szfvar.nospam@gmail.com' Jan 01 10:54:37 2022 (126980) ('127.0.0.1', 54816) sender: horvath.arpad.szfvar.nospam@gmail.com Jan 01 10:54:37 2022 (126980) ('127.0.0.1', 54816) Data: b'RCPT TO:test@lists.xxx.hu' Jan 01 10:54:37 2022 (126980) ('127.0.0.1', 54816) recip: test@lists.xxx.hu Jan 01 10:54:37 2022 (126980) ('127.0.0.1', 54816) Data: b'DATA' Jan 01 10:54:37 2022 (126980) ('127.0.0.1', 54816) Data: b'QUIT' Jan 01 10:54:37 2022 (126980) ('127.0.0.1', 54816) connection lost Jan 01 10:54:37 2022 (126980) Connection lost during _handle_client()
I have these package versions
ii python3-urllib3 1.26.5-1~exp1 all HTTP library with thread-safe connection pooling for Python3 ii python3-requests 2.25.1+dfsg-2 all elegant and simple HTTP library for Python3, built for human beings ii python3-requests-oauthlib 1.0.0-1.1 all module providing OAuthlib auth support for requests (Python 3) ii python3-requests-toolbelt 0.9.1-1 all Utility belt for advanced users of python3-requests
On 1/1/22 2:31 AM, horvath.arpad.szfvar@gmail.com wrote:
In the /var/log/mailman3/smtp.log I have at the same time:
Jan 01 10:54:37 2022 (126980) Available AUTH mechanisms: LOGIN(builtin) PLAIN(builtin) Jan 01 10:54:37 2022 (126980) Peer: ('127.0.0.1', 54816) Jan 01 10:54:37 2022 (126980) ('127.0.0.1', 54816) handling connection Jan 01 10:54:37 2022 (126980) ('127.0.0.1', 54816) Data: b'LHLO yyy.hu' Jan 01 10:54:37 2022 (126980) ('127.0.0.1', 54816) Data: b'MAIL FROM:horvath.arpad.szfvar.nospam@gmail.com' Jan 01 10:54:37 2022 (126980) ('127.0.0.1', 54816) sender: horvath.arpad.szfvar.nospam@gmail.com Jan 01 10:54:37 2022 (126980) ('127.0.0.1', 54816) Data: b'RCPT TO:test@lists.xxx.hu' Jan 01 10:54:37 2022 (126980) ('127.0.0.1', 54816) recip: test@lists.xxx.hu Jan 01 10:54:37 2022 (126980) ('127.0.0.1', 54816) Data: b'DATA' Jan 01 10:54:37 2022 (126980) ('127.0.0.1', 54816) Data: b'QUIT' Jan 01 10:54:37 2022 (126980) ('127.0.0.1', 54816) connection lost Jan 01 10:54:37 2022 (126980) Connection lost during _handle_client()
This is all normal and expected for a post from horvath.arpad.szfvar.nospam@gmail.com to test@lists.xxx.hu.
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 1/1/22 2:14 AM, horvath.arpad.szfvar@gmail.com wrote:
I changed base_url to the https one, reloaded nginx, restarted mailman3-web.
Now there is an other error message in mailman.log (I can't find the a meaningful result for InvalidChunkLength error message in this list and in google search)
Jan 01 10:54:40 2022 (126976) Connection to HyperKitty failed: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))
This exception is raised by urllib3 and is documented at https://urllib3.readthedocs.io/en/stable/reference/urllib3.exceptions.html#u...
...
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/usr/lib/python3/dist-packages/mailman_hyperkitty/__init__.py", line 154, in _archive_message url = self._send_message(mlist, msg) File "/usr/lib/python3/dist-packages/mailman_hyperkitty/__init__.py", line 198, in _send_message result = requests.post( File "/usr/lib/python3/dist-packages/requests/api.py", line 119, in post return request('post', url, data=data, json=json, **kwargs) File "/usr/lib/python3/dist-packages/requests/api.py", line 61, in request return session.request(method=method, url=url, **kwargs) File "/usr/lib/python3/dist-packages/requests/sessions.py", line 542, in request resp = self.send(prep, **send_kwargs) File "/usr/lib/python3/dist-packages/requests/sessions.py", line 697, in send r.content File "/usr/lib/python3/dist-packages/requests/models.py", line 831, in content self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b'' File "/usr/lib/python3/dist-packages/requests/models.py", line 756, in generate raise ChunkedEncodingError(e) requests.exceptions.ChunkedEncodingError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))
We are trying to archive the message via a POST request to the URL https://lists.xxx.hu/hyperkitty/api/mailman/archive and the response is chunked with an invalid chunk length.
What happens if you go to https://lists.xxx.hu/hyperkitty/api/ in a web browser. That should return a page of documentation of the API. Does it? Can you go to https://lists.xxx.hu/hyperkitty/api/lists and get a response from Django REST framework which is a json representation of your mailing lists info?
I don't know what could be causing this error, but I suspect something in the configuration of the web server that is serving this request
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Can you go to https://lists.xxx.hu/hyperkitty/api/lists and get a response from Django REST framework which is a json representation of your mailing lists info?
In this list I have just one list that I created for testing. And I can't see there the two lists I've imported from mailman2, e.g. korus. I guess because of admin is not a list list member there. I can see the korus at https://lists.xxx.hu/hyperkitty/ , but not at https://lists.xxx.hu/postorius/lists/ .
I guess the setting below is not good.
MAILMAN_ARCHIVER_FROM = ('127.0.0.1', '::1')
Should I set it to ('https://lists.xxx.hu/hyperkitty', '::1') ? Doesn't seem to work, like that.
And what about MAILMAN_REST_API_URL? How can I found its proper value? It's currently the default value but if I try to reach it with wget http://localhost:8001 I get a 404. not found. How can I find the right value?
8001 is listening, so the latter value might be good. horvatha@pyEdu /etc/mailman3 (main) 12:34:13$ sudo lsof -i -P -n | grep LISTEN (...) python3 216185 list 25u IPv4 1079013 0t0 TCP 127.0.0.1:8001 (LISTEN) python3 216186 list 25u IPv4 1079013 0t0 TCP 127.0.0.1:8001 (LISTEN)
On 1/2/22 4:34 AM, horvath.arpad.szfvar@gmail.com wrote:
Can you go to https://lists.xxx.hu/hyperkitty/api/lists and get a response from Django REST framework which is a json representation of your mailing lists info?
In this list I have just one list that I created for testing. And I can't see there the two lists I've imported from mailman2, e.g. korus. I guess because of admin is not a list list member there. I can see the korus at https://lists.xxx.hu/hyperkitty/ , but not at https://lists.xxx.hu/postorius/lists/ .
when you view https://lists.xxx.hu/postorius/lists/ is there a Filter by
Role dropdown at the top of the listing? If so, if you select All do
you then see the lists. Postorius by default shows only those lists in
which you have a role.
If you still don't see the lists, what does the command mailman lists
show?
I guess the setting below is not good.
MAILMAN_ARCHIVER_FROM = ('127.0.0.1', '::1')
Should I set it to ('https://lists.xxx.hu/hyperkitty', '::1') ? Doesn't seem to work, like that.
It needs to be an IP. Set it to ('xxx.xxx.xxx.xxx', '127.0.0.1', '::1') where xxx.xxx.xxx.xxx is the IP of lists.xxx.hu or just
('xxx.xxx.xxx.xxx',) would probably work.
And what about MAILMAN_REST_API_URL? How can I found its proper value? It's currently the default value but if I try to reach it with wget http://localhost:8001 I get a 404. not found. How can I find the right value?
If Postorius works, It's good. wget http://localhost:8001 returns a
404 as it should because this is a REST API and / is not a known
endpoint. Something like
wget --user restadmin --password restpass http://localhost:8001/3.1/lists
should work.
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
participants (3)
-
horvath.arpad.szfvar＠gmail.com -
Mark Sapiro -
William Oliver