HyperKitty failure

horvath.arpad.szfvar＠gmail.com

Dec. 30, 2021

9:19 p.m.

Hi,

I try to connect mailman3 to hyperkitty, but there is no success so far. You can see the details below. How can I solve the problem?

As I have several domain name (like lists.xxx.hu, xxx.hu, yyy.hu) on the same server, so I guess localhost won't work.

In mailman-hyperkitty.cfg there is a line:

base_url: http://lists.xxx.hu/hyperkitty/

/etc/mailman3/mailman-web.py includes "*" in ALLOWED_HOSTS

I have an error message in /var/log/mailman3/mailman.log

Dec 30 21:41:23 2021 (126976) HyperKitty failure on http://lists.xxx.hu/hyperkitty/api/mailman/archive: (405) Dec 30 21:41:23 2021 (126976) Exception in the HyperKitty archiver: Dec 30 21:41:23 2021 (126976) Traceback (most recent call last): File "/usr/lib/python3/dist-packages/mailman_hyperkitty/__init__.py", line 154, in _archive_message url = self._send_message(mlist, msg) File "/usr/lib/python3/dist-packages/mailman_hyperkitty/__init__.py", line 210, in _send_message raise ValueError(result.text) ValueError

In /var/log/mailman3/web/mailman-web.log

WARNING 2021-12-30 20:41:23,816 127032 django.request Method Not Allowed (GET): /hyperkitty/api/mailman/archive [pid: 127032|app: 0|req: 101/101] 109.230.236.204 () {38 vars in 592 bytes} [Thu Dec 30 20:41:23 2021] GET /hyperkitty/api/mailman/archive?key=lHzJB%2FzfSWUrbPKZ84JNSlLXHVyCDt8T => generated 0 bytes in 1 msecs (HTTP/1.1 405) 6 headers in 189 bytes (1 switches on core 0) WARNING 2021-12-30 20:41:23,854 127032 django.request Method Not Allowed (GET): /hyperkitty/api/mailman/archive

On Debian 11, I have these packages

ii mailman3 3.3.3-1 all Mailing list management system ii mailman3-full 3.3.3-1 all Full Mailman3 mailing list management suite (metapackage) ii mailman3-web 0+20200530-2 all Django project integrating Mailman3 Postorius and HyperKitty ii python3-django-mailman3 1.3.5-2 all Django library to help interaction with Mailman3 (Python 3 version) ii python3-mailman-hyperkitty 1.1.0-10 all Mailman3 plugin to archive emails with HyperKitty

Thanks, Arpad

Show replies by date

William Oliver

December 2021

9:30 p.m.

On Thu, 2021-12-30 at 21:19 +0000, horvath.arpad.szfvar@gmail.com wrote:

...

Hi,

I try to connect mailman3 to hyperkitty, but there is no success so far. You can see the details below. How can I solve the problem?

As I have several domain name (like lists.xxx.hu, xxx.hu, yyy.hu) on the same server, so I guess localhost won't work.

In mailman-hyperkitty.cfg there is a line:

base_url: http://lists.xxx.hu/hyperkitty/

/etc/mailman3/mailman-web.py includes "*" in ALLOWED_HOSTS

I have an error message in /var/log/mailman3/mailman.log

Dec 30 21:41:23 2021 (126976) HyperKitty failure on http://lists.xxx.hu/hyperkitty/api/mailman/archive: (405) Dec 30 21:41:23 2021 (126976) Exception in the HyperKitty archiver: Dec 30 21:41:23 2021 (126976) Traceback (most recent call last): File "/usr/lib/python3/dist-packages/mailman_hyperkitty/__init__.py", line 154, in _archive_message url = self._send_message(mlist, msg) File "/usr/lib/python3/dist-packages/mailman_hyperkitty/__init__.py", line 210, in _send_message raise ValueError(result.text) ValueError

[snip]

I'm no expert, but here's a previous discussion I found:

https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/thread/P...

I don't know if it applies, but it looks close.

billo

Mark Sapiro

January 2022

12:36 a.m.

On 12/30/21 1:30 PM, William Oliver wrote:

...

On Thu, 2021-12-30 at 21:19 +0000, horvath.arpad.szfvar@gmail.com wrote:

...
Hi,

I try to connect mailman3 to hyperkitty, but there is no success so far. You can see the details below. How can I solve the problem?

As I have several domain name (like lists.xxx.hu, xxx.hu, yyy.hu) on the same server, so I guess localhost won't work.

In mailman-hyperkitty.cfg there is a line:

base_url: http://lists.xxx.hu/hyperkitty/

As noted in the thread that William references and also the thread at https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/thread/L... the issue is probably that the above http url is being redirected to https and losing POST data. Set the above to

base_url: https://lists.xxx.hu/hyperkitty/

...

[snip]

I'm no expert, but here's a previous discussion I found:

https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/thread/P...

-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

horvath.arpad.szfvar＠gmail.com

10:14 a.m.

I changed base_url to the https one, reloaded nginx, restarted mailman3-web.

Now there is an other error message in mailman.log (I can't find the a meaningful result for InvalidChunkLength error message in this list and in google search)

Jan 01 10:54:40 2022 (126976) Connection to HyperKitty failed: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read)) Jan 01 10:54:40 2022 (126976) Exception in the HyperKitty archiver: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read)) Jan 01 10:54:40 2022 (126976) Traceback (most recent call last): File "/usr/lib/python3/dist-packages/urllib3/response.py", line 697, in _update_chunk_length self.chunk_left = int(line, 16) ValueError: invalid literal for int() with base 16: b''

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/lib/python3/dist-packages/urllib3/response.py", line 438, in _error_catcher yield File "/usr/lib/python3/dist-packages/urllib3/response.py", line 764, in read_chunked self._update_chunk_length() File "/usr/lib/python3/dist-packages/urllib3/response.py", line 701, in _update_chunk_length raise InvalidChunkLength(self, line) urllib3.exceptions.InvalidChunkLength: InvalidChunkLength(got length b'', 0 bytes read)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/lib/python3/dist-packages/requests/models.py", line 753, in generate for chunk in self.raw.stream(chunk_size, decode_content=True): File "/usr/lib/python3/dist-packages/urllib3/response.py", line 572, in stream for line in self.read_chunked(amt, decode_content=decode_content): File "/usr/lib/python3/dist-packages/urllib3/response.py", line 793, in read_chunked self._original_response.close() File "/usr/lib/python3.9/contextlib.py", line 135, in __exit__ self.gen.throw(type, value, traceback) File "/usr/lib/python3/dist-packages/urllib3/response.py", line 455, in _error_catcher raise ProtocolError("Connection broken: %r" % e, e) urllib3.exceptions.ProtocolError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/lib/python3/dist-packages/mailman_hyperkitty/__init__.py", line 154, in _archive_message url = self._send_message(mlist, msg) File "/usr/lib/python3/dist-packages/mailman_hyperkitty/__init__.py", line 198, in _send_message result = requests.post( File "/usr/lib/python3/dist-packages/requests/api.py", line 119, in post return request('post', url, data=data, json=json, **kwargs) File "/usr/lib/python3/dist-packages/requests/api.py", line 61, in request return session.request(method=method, url=url, **kwargs) File "/usr/lib/python3/dist-packages/requests/sessions.py", line 542, in request resp = self.send(prep, **send_kwargs) File "/usr/lib/python3/dist-packages/requests/sessions.py", line 697, in send r.content File "/usr/lib/python3/dist-packages/requests/models.py", line 831, in content self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b'' File "/usr/lib/python3/dist-packages/requests/models.py", line 756, in generate raise ChunkedEncodingError(e) requests.exceptions.ChunkedEncodingError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read)) Jan 01 10:54:40 2022 (126976) Connection to HyperKitty failed: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read)) Jan 01 10:54:40 2022 (126976) Exception in the HyperKitty archiver: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read)) Jan 01 10:54:40 2022 (126976) Traceback (most recent call last): File "/usr/lib/python3/dist-packages/urllib3/response.py", line 697, in _update_chunk_length self.chunk_left = int(line, 16) ValueError: invalid literal for int() with base 16: b''

During handling of the above exception, another exception occurred:

horvath.arpad.szfvar＠gmail.com

10:31 a.m.

In the /var/log/mailman3/smtp.log I have at the same time:

Jan 01 10:54:37 2022 (126980) Available AUTH mechanisms: LOGIN(builtin) PLAIN(builtin) Jan 01 10:54:37 2022 (126980) Peer: ('127.0.0.1', 54816) Jan 01 10:54:37 2022 (126980) ('127.0.0.1', 54816) handling connection Jan 01 10:54:37 2022 (126980) ('127.0.0.1', 54816) Data: b'LHLO yyy.hu' Jan 01 10:54:37 2022 (126980) ('127.0.0.1', 54816) Data: b'MAIL FROM:<horvath.arpad.szfvar.nospam@gmail.com>' Jan 01 10:54:37 2022 (126980) ('127.0.0.1', 54816) sender: horvath.arpad.szfvar.nospam@gmail.com Jan 01 10:54:37 2022 (126980) ('127.0.0.1', 54816) Data: b'RCPT TO:<test@lists.xxx.hu>' Jan 01 10:54:37 2022 (126980) ('127.0.0.1', 54816) recip: test@lists.xxx.hu Jan 01 10:54:37 2022 (126980) ('127.0.0.1', 54816) Data: b'DATA' Jan 01 10:54:37 2022 (126980) ('127.0.0.1', 54816) Data: b'QUIT' Jan 01 10:54:37 2022 (126980) ('127.0.0.1', 54816) connection lost Jan 01 10:54:37 2022 (126980) Connection lost during _handle_client()

I have these package versions

ii python3-urllib3 1.26.5-1~exp1 all HTTP library with thread-safe connection pooling for Python3 ii python3-requests 2.25.1+dfsg-2 all elegant and simple HTTP library for Python3, built for human beings ii python3-requests-oauthlib 1.0.0-1.1 all module providing OAuthlib auth support for requests (Python 3) ii python3-requests-toolbelt 0.9.1-1 all Utility belt for advanced users of python3-requests

Mark Sapiro

11:32 p.m.

On 1/1/22 2:31 AM, horvath.arpad.szfvar@gmail.com wrote:

...

In the /var/log/mailman3/smtp.log I have at the same time:

Jan 01 10:54:37 2022 (126980) Available AUTH mechanisms: LOGIN(builtin) PLAIN(builtin) Jan 01 10:54:37 2022 (126980) Peer: ('127.0.0.1', 54816) Jan 01 10:54:37 2022 (126980) ('127.0.0.1', 54816) handling connection Jan 01 10:54:37 2022 (126980) ('127.0.0.1', 54816) Data: b'LHLO yyy.hu' Jan 01 10:54:37 2022 (126980) ('127.0.0.1', 54816) Data: b'MAIL FROM:<horvath.arpad.szfvar.nospam@gmail.com>' Jan 01 10:54:37 2022 (126980) ('127.0.0.1', 54816) sender: horvath.arpad.szfvar.nospam@gmail.com Jan 01 10:54:37 2022 (126980) ('127.0.0.1', 54816) Data: b'RCPT TO:<test@lists.xxx.hu>' Jan 01 10:54:37 2022 (126980) ('127.0.0.1', 54816) recip: test@lists.xxx.hu Jan 01 10:54:37 2022 (126980) ('127.0.0.1', 54816) Data: b'DATA' Jan 01 10:54:37 2022 (126980) ('127.0.0.1', 54816) Data: b'QUIT' Jan 01 10:54:37 2022 (126980) ('127.0.0.1', 54816) connection lost Jan 01 10:54:37 2022 (126980) Connection lost during _handle_client()

This is all normal and expected for a post from horvath.arpad.szfvar.nospam@gmail.com to test@lists.xxx.hu.

-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

Mark Sapiro

11:57 p.m.

On 1/1/22 2:14 AM, horvath.arpad.szfvar@gmail.com wrote:

...

I changed base_url to the https one, reloaded nginx, restarted mailman3-web.

Now there is an other error message in mailman.log (I can't find the a meaningful result for InvalidChunkLength error message in this list and in google search)

Jan 01 10:54:40 2022 (126976) Connection to HyperKitty failed: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))

This exception is raised by urllib3 and is documented at https://urllib3.readthedocs.io/en/stable/reference/urllib3.exceptions.html#u...

...

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/lib/python3/dist-packages/mailman_hyperkitty/__init__.py", line 154, in _archive_message url = self._send_message(mlist, msg) File "/usr/lib/python3/dist-packages/mailman_hyperkitty/__init__.py", line 198, in _send_message result = requests.post( File "/usr/lib/python3/dist-packages/requests/api.py", line 119, in post return request('post', url, data=data, json=json, **kwargs) File "/usr/lib/python3/dist-packages/requests/api.py", line 61, in request return session.request(method=method, url=url, **kwargs) File "/usr/lib/python3/dist-packages/requests/sessions.py", line 542, in request resp = self.send(prep, **send_kwargs) File "/usr/lib/python3/dist-packages/requests/sessions.py", line 697, in send r.content File "/usr/lib/python3/dist-packages/requests/models.py", line 831, in content self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b'' File "/usr/lib/python3/dist-packages/requests/models.py", line 756, in generate raise ChunkedEncodingError(e) requests.exceptions.ChunkedEncodingError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))

We are trying to archive the message via a POST request to the URL https://lists.xxx.hu/hyperkitty/api/mailman/archive and the response is chunked with an invalid chunk length.

What happens if you go to https://lists.xxx.hu/hyperkitty/api/ in a web browser. That should return a page of documentation of the API. Does it? Can you go to https://lists.xxx.hu/hyperkitty/api/lists and get a response from Django REST framework which is a json representation of your mailing lists info?

I don't know what could be causing this error, but I suspect something in the configuration of the web server that is serving this request

-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

horvath.arpad.szfvar＠gmail.com

12:34 p.m.

...

Can you go to https://lists.xxx.hu/hyperkitty/api/lists and get a response from Django REST framework which is a json representation of your mailing lists info?

In this list I have just one list that I created for testing. And I can't see there the two lists I've imported from mailman2, e.g. korus. I guess because of admin is not a list list member there. I can see the korus at https://lists.xxx.hu/hyperkitty/ , but not at https://lists.xxx.hu/postorius/lists/ .

I guess the setting below is not good.

MAILMAN_ARCHIVER_FROM = ('127.0.0.1', '::1')

Should I set it to ('https://lists.xxx.hu/hyperkitty', '::1') ? Doesn't seem to work, like that.

And what about MAILMAN_REST_API_URL? How can I found its proper value? It's currently the default value but if I try to reach it with wget http://localhost:8001 I get a 404. not found. How can I find the right value?

8001 is listening, so the latter value might be good. horvatha@pyEdu /etc/mailman3 (main) 12:34:13$ sudo lsof -i -P -n | grep LISTEN (...) python3 216185 list 25u IPv4 1079013 0t0 TCP 127.0.0.1:8001 (LISTEN) python3 216186 list 25u IPv4 1079013 0t0 TCP 127.0.0.1:8001 (LISTEN)

Mark Sapiro

9:18 p.m.

On 1/2/22 4:34 AM, horvath.arpad.szfvar@gmail.com wrote:

...

...
Can you go to https://lists.xxx.hu/hyperkitty/api/lists and get a response from Django REST framework which is a json representation of your mailing lists info?

In this list I have just one list that I created for testing. And I can't see there the two lists I've imported from mailman2, e.g. korus. I guess because of admin is not a list list member there. I can see the korus at https://lists.xxx.hu/hyperkitty/ , but not at https://lists.xxx.hu/postorius/lists/ .

when you view https://lists.xxx.hu/postorius/lists/ is there a Filter by Role dropdown at the top of the listing? If so, if you select All do you then see the lists. Postorius by default shows only those lists in which you have a role.

If you still don't see the lists, what does the command mailman lists show?

...

I guess the setting below is not good.

MAILMAN_ARCHIVER_FROM = ('127.0.0.1', '::1')

Should I set it to ('https://lists.xxx.hu/hyperkitty', '::1') ? Doesn't seem to work, like that.

It needs to be an IP. Set it to ('xxx.xxx.xxx.xxx', '127.0.0.1', '::1') where xxx.xxx.xxx.xxx is the IP of lists.xxx.hu or just ('xxx.xxx.xxx.xxx',) would probably work.

...

And what about MAILMAN_REST_API_URL? How can I found its proper value? It's currently the default value but if I try to reach it with wget http://localhost:8001 I get a 404. not found. How can I find the right value?

If Postorius works, It's good. wget http://localhost:8001 returns a 404 as it should because this is a REST API and / is not a known endpoint. Something like wget --user restadmin --password restpass http://localhost:8001/3.1/lists should work.

-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

1163

Age (days ago)

1166

Last active (days ago)

List overview

Download

8 comments

3 participants

참여자 (3)

horvath.arpad.szfvar＠gmail.com
Mark Sapiro
William Oliver

HyperKitty failure

tags

참여자 (3)