Re: Hyperkitty 403 Forbidden Error
Dear all,
Sorry for coming back to a topic that seems to be done for 18 months now.
I'm staring at this issue for a few days now (on and off). But I still get these messages:
Jan 19 22:43:27 2022 (149439) HyperKitty failure on https://lists.[...]/hyperkitty/api/mailman/urls: <html><title>Forbidden</title><body>
<h1>Access is forbidden</h1><p>Please check the IP addresses
assigned to MAILMAN_ARCHIVER_FROM in the settings file.
</p></body></html> (403). and consequently, nothing ends up in the archive.
I have checked (among others):
MAILMAN_ARCHIVER_FROM - doesn't make a difference. 127.0.0.1, external IP address, different orders, '*'.
(Asterisk makes a difference, see at the bottom)MAILMAN_ARCHIVER_KEY in mailman-web.py matches api_key in mailman-hyperkitty.cfg (except for the quotes). According to the editor's
color coding, the quotes are "right", too. I have removed special characters to circumvent diverging treatment in different environments
I guess it doesn't matter, but wget --user restadmin --password [.] http://localhost:8001/3.1/lists works, too.
ACCOUNT_DEFAULT_HTTP_PROTOCOL = "https" (probably also doesn't matter
I'm running a fresh package installation from Debian Bullseye with certificates from Certbot with nginx, so Certbot may have messed with my nginx config, but even that seems to be OK: Yes, Certbot has inserted http->https redirection, but I'd expect that a https base_url and ACCOUNT_DEFAULT_HTTP_PROTOCOL = "https" circumvents that.
Mailman info says
GNU Mailman 3.3.3 (Tom Sawyer)
Python 3.9.2 (default, Feb 28 2021, 17:03:44)
[GCC 10.2.1 20210110]
.
REST root url: http://localhost:8001/3.1/
Hyperkitty is 1.3.4, Postorious is 1.3.4, too.
To provide all possible information, I'm also adding the nginx log that seems to correspond to that request (no other requests at that time, give or take five minutes): Yes, everything yields 403 response codes.
[.] - - [19/Jan/2022:22:43:27 +0000] "GET /hyperkitty/api/mailman/urls?mlist=demo%40lists.pro-digi-par.de&key=[.] HTTP/1.1" 403 175 "-" "python-requests/2.25.1"
[.] - - [19/Jan/2022:22:43:27 +0000] "GET /hyperkitty/api/mailman/urls?mlist=demo%40lists.pro-digi-par.de&msgid=AM0PR0 4MB478573911C2A81B8898527E6C2599%40AM0PR04MB4785.eurprd04.prod.outlook.com&k ey=[.] HTTP/1.1
" 403 175 "-" "python-requests/2.25.1"
[.] - - [19/Jan/2022:22:43:28 +0000] "GET /hyperkitty/api/mailman/urls?mlist=demo%40lists.pro-digi-par.de&msgid=AM0PR0 4MB478573911C2A81B8898527E6C2599%40AM0PR04MB4785.eurprd04.prod.outlook.com&k ey=[.] HTTP/1.1
" 403 175 "-" "python-requests/2.25.1"
[.] - - [19/Jan/2022:22:43:28 +0000] "POST /hyperkitty/api/mailman/archive?key=[.] HTTP/1.1" 403 166 "-" "python-requests/2.25.1"
[.] - - [19/Jan/2022:22:43:28 +0000] "POST /hyperkitty/api/mailman/archive?key=[.] HTTP/1.1" 403 166 "-" "python-requests/2.25.1"
. repeated many times, there seems to be quite a few testmessages in the queue .
[.] - - [19/Jan/2022:22:46:50 +0000] "GET /hyperkitty/api/mailman/urls?mlist=demo%40lists.pro-digi-par.de&key=[.] HTTP/1.1" 403 175 "-" "python-requests/2.25.1"
[.] - - [19/Jan/2022:22:46:50 +0000] "GET /hyperkitty/api/mailman/urls?mlist=demo%40lists.pro-digi-par.de&msgid=AM0PR0 4MB4785430188B9C0E8DF248231C2599%40AM0PR04MB4785.eurprd04.prod.outlook.com&k ey=6xpENHLEHQ0kD53vXt5RpKzPDOsKbcN9 HTTP/1.1
" 403 175 "-" "python-requests/2.25.1"
[.] - - [19/Jan/2022:22:46:50 +0000] "POST /hyperkitty/api/mailman/archive?key=[.] HTTP/1.1" 403 166 "-" "python-requests/2.25.1"
. again repeated many times .
[.] - - [19/Jan/2022:22:46:51 +0000] "GET /hyperkitty/api/mailman/urls?mlist=demo%40lists.pro-digi-par.de&msgid=AM0PR0 4MB4785430188B9C0E8DF248231C2599%40AM0PR04MB4785.eurprd04.prod.outlook.com&k ey=[.] HTTP/1.1" 403 175 "-" "python-requests/2.25.1"
I'd appreciate a new idea where I could look for clues.
Thanks a lot,
Josef
On 1/20/22 11:40 PM, Josef Dietl wrote:
Dear all,
Sorry for coming back to a topic that seems to be done for 18 months now.
I'm staring at this issue for a few days now (on and off). But I still get these messages:
Jan 19 22:43:27 2022 (149439) HyperKitty failure on https://lists.[...]/hyperkitty/api/mailman/urls: <html><title>Forbidden</title><body>
<h1>Access is forbidden</h1><p>Please check the IP addresses assigned to MAILMAN_ARCHIVER_FROM in the settings file. </p></body></html> (403). and consequently, nothing ends up in the archive.
I have checked (among others):
- MAILMAN_ARCHIVER_FROM - doesn't make a difference. 127.0.0.1, external IP address, different orders, '*'.
OK.
(Asterisk makes a difference, see at the bottom)
??? I didn't see what.
- MAILMAN_ARCHIVER_KEY in mailman-web.py matches api_key in mailman-hyperkitty.cfg (except for the quotes). According to the editor's
color coding, the quotes are "right", too. I have removed special characters to circumvent diverging treatment in different environments
- I guess it doesn't matter, but wget --user restadmin --password [.] http://localhost:8001/3.1/lists works, too.
That is Mailman core's REST API which is not the issue here.
- ACCOUNT_DEFAULT_HTTP_PROTOCOL = "https" (probably also doesn't matter
I'm running a fresh package installation from Debian Bullseye with certificates from Certbot with nginx, so Certbot may have messed with my nginx config, but even that seems to be OK: Yes, Certbot has inserted http->https redirection, but I'd expect that a https base_url and ACCOUNT_DEFAULT_HTTP_PROTOCOL = "https" circumvents that.
It should.
Note that your primary resource for help with issue with Debian packages should be Debian.
Mailman info says ... To provide all possible information, I'm also adding the nginx log that seems to correspond to that request (no other requests at that time, give or take five minutes): Yes, everything yields 403 response codes.
[.] - - [19/Jan/2022:22:43:27 +0000] "GET /hyperkitty/api/mailman/urls?mlist=demo%40lists.pro-digi-par.de&key=[.] HTTP/1.1" 403 175 "-" "python-requests/2.25.1"
The issue here is accessing the hyperkitty API. The base URL for these
accesses is configured in mailman-hyperkitty.cfg. From what I see here,
that is something like http(s)://host.example.com/hyperkitty/ If you
append api/ to that base URL and go there in a web browser do you get
a page of documentation of the HyperKitty API? If so, then figure out
why it works from a web browser and not when Mailman core does it. If
you get the 403, figure out why. This is probably an nginx issue.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Dear Mark, Dear all,
(Asterisk makes a difference, see at the bottom)??? I didn't see what.
Sorry, my bad. While chasing the issue, I had thought I had seen a difference but can't reproduce it anymore. Then I forgot to remove the pointer.
I'm running a fresh package installation from Debian Bullseye with certificates from Certbot with nginx, so Certbot may have messed with my nginx config, but even that seems to be OK: Yes, Certbot has inserted http->https redirection, but I'd expect that a https base_url and ACCOUNT_DEFAULT_HTTP_PROTOCOL = "https" circumvents that.
It should.
Note that your primary resource for help with issue with Debian packages should be Debian.
The state of the Debian Bullseye packages suggest a certain feeling of priority on that matter ☹ If I can't get this going "soon", I'll wipe the entire server, install a fresh Bullseye and install mailman3 through pip. Out of bad experience, I'd like to avoid mixing software lifecycle managers, but it seems to be the lesser evil.
To provide all possible information, I'm also adding the nginx log that seems to correspond to that request (no other requests at that time, give or take five minutes): Yes, everything yields 403 response codes.
[.] - - [19/Jan/2022:22:43:27 +0000] "GET /hyperkitty/api/mailman/urls?mlist=demo%40lists.pro-digi-par.de&key=[.] HTTP/1.1" 403 175 "-" "python-requests/2.25.1"
The issue here is accessing the hyperkitty API. The base URL for these accesses is configured in mailman-hyperkitty.cfg. From what I see here, that is something like
http(s)://host.example.com/hyperkitty/If you appendapi/to that base URL and go there in a web browser do you get a page of documentation of the HyperKitty API? If so, then figure out why it works from a web browser and not when Mailman core does it. If you get the 403, figure out why. This is probably an nginx issue.
Seems like an nginx issue indeed. However, I'm an apache guy, trying to re-educate myself on nginx.
OK, here's the result of changing mailman-hyperkitty.cfg from http(s)://host.example.com/hyperkitty/ to http(s)://host.example.com/hyperkitty/api/
Yes, if I open things from the web browser, I do get the HyperKitty API doc. With what follows below, I guess it's a timeout: Processing a message takes longer than rendering a HTML document.
Jan 22 20:26:27 2022 (149435) ACCEPT: <AM0PR04MB4785365CA1BB735884E149CEC25C9@AM0PR04MB4785.eurprd04.prod.outlook.com> Jan 22 20:26:28 2022 (149439) HyperKitty failure on https://[FQDN]/hyperkitty/api/api/mailman/urls:
(what follows is a lengthy HTML document, essentially a 404 not found message).
That corresponds to the nginx logfiles:
access.log: [IP4] - - [22/Jan/2022:20:26:29 +0000] "POST /hyperkitty/api/api/mailman/archive?key=6xpENHLEHQ0kD53vXt5RpKzPDOsKbcN9 HTTP/1.1" 404 1342 "-" "python-requests/2.25.1" 9
And error.log: 2022/01/22 20:26:29 [error] 128720#128720: *4375 readv() failed (104: Connection reset by peer) while reading upstream, client: [IP4], server: [FQDN], request: "POST /hyperkitty/api/api/mailman/archive?key=[key] HTTP/1.1", upstream: "uwsgi://unix:/run/mailman3-web/uwsgi.sock:", host: "[FQDN]"
So it all seems consistent in a way, except that error.log suggests that this could be a timeout.
Question: What happens if nginx issues a 403 "Forbidden" in that spot? - Does the mailman system as a whole (whichever subsystem is responsible...) _assume_ that there is an error with MAILMAN_ARCHIVER_FROM (as suggested in the mailman3 logfile) - so other possible reasons for 403 are theoretically possible, or are other possible reasons for 403 explicitly (e.g. a timeout) ruled out somewhere, somehow?
Following that, I've set the timeout in nginx to five minutes, but it doesn't change anything.
I'm sorry that I can't contribute any news. I guess I'll start over, using pip and virtualenv for installation.
Thank you for your help!
On 1/23/22 07:41, Josef Dietl wrote:
The issue here is accessing the hyperkitty API. The base URL for these accesses is configured in mailman-hyperkitty.cfg. From what I see here, that is something like
http(s)://host.example.com/hyperkitty/If you appendapi/to that base URL and go there in a web browser do you get a page of documentation of the HyperKitty API? If so, then figure out why it works from a web browser and not when Mailman core does it. If you get the 403, figure out why. This is probably an nginx issue.Seems like an nginx issue indeed. However, I'm an apache guy, trying to re-educate myself on nginx. OK, here's the result of changing mailman-hyperkitty.cfg from
http(s)://host.example.com/hyperkitty/tohttp(s)://host.example.com/hyperkitty/api/
Do not make this change in mailman-hyperkitty.cfg. That will produce 404's for sure. This was only a suggestion to try from a browser.
Yes, if I open things from the web browser, I do get the HyperKitty API doc.
OK. That means you can access HyperKitty's API from a browser at
http(s)://host.example.com/hyperkitty/api/. This is the same thing
Mailman core is doing if base_url is set to
http(s)://host.example.com/hyperkitty/, so what's the difference
between Mailman core's access via mailman-hyperkitty and your's via the
web browser.
For further diagnosis, you might try wget -O- http(s)://host.example.com/hyperkitty/api/ on the Mailman server.
With what follows below, I guess it's a timeout: Processing a message takes longer than rendering a HTML document.
Jan 22 20:26:27 2022 (149435) ACCEPT: <AM0PR04MB4785365CA1BB735884E149CEC25C9@AM0PR04MB4785.eurprd04.prod.outlook.com> Jan 22 20:26:28 2022 (149439) HyperKitty failure on https://[FQDN]/hyperkitty/api/api/mailman/urls:
Do not include the api/ in base_url. That was just for testing with a web browser or wget/curl.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
participants (2)
-
Josef Dietl -
Mark Sapiro