Hi,
I followed the installation instructions at https://docs.mailman3.org/en/latest/install/virtualenv.html#virtualenv-insta..., which thankfully shielded me from configuring the Django applications for the web interface manually. All Django applications (postorius, hyperkitty, haystack etc) are loaded via mailman_web/settings/base.py, which is in turn referenced by the top-level settings file in /etc/mailman3. uwsgi is linked to mailman_web via /etc/mailman3/uwsgi.ini. So far, so good.
I do not want to dive into the interaction between the various components and just want to switch from a http socket (http-socket = 0.0.0.0:8000) to a file socket (socket = /opt/mailman/mm/var/mailman.sock). After making this change in /etc/mailman3/uwsgi.ini, I applied it in the nginx proxy configuration
proxy_pass http://127.0.0.1:8000; → proxy_pass http://unix:/opt/mailman/mm/var/mailman.sock;
and received an error message (as expected):
2022/11/25 18:39:12 [error] 2978606#2978606: *3 upstream prematurely closed connection while reading response header from upstream, client: 37.201.155.65, server: lists.eden.one, request: "GET /mailman3/lists/ HTTP/2.0", upstream: "http://unix:/opt/mailman/mm/var/mailman.sock:/mailman3/lists/", host: "lists.eden.one"
There are obviously a couple of places where the http-socket on port 8000 is referenced (e.g. mailman_web/settings/mailman.py, /opt/mailman/mm/hyperkitty.cfg). Is there an overview of the required changes when using a different (file) socket?
Related question: My own (much simpler) Django applications use the following (e.g. in mysite_nginx.conf):
upstream django { server unix:///private/tmp/mysite.sock; }
location / { uwsgi_pass django; }
instead of the "http://unix:/..." notation I tried for Mailman above. Is there any difference between the upstream/uwsgi_pass combination and the proxy_pass variant?
Thanks, Jan
On 11/25/22 23:00, Jan Eden via Mailman-users wrote:
proxy_pass http://127.0.0.1:8000; → proxy_pass http://unix:/opt/mailman/mm/var/mailman.sock;
and received an error message (as expected):
Because that won't work. You can't http to a unix socket in that way.
Use
uwsgi_pass unix:/opt/mailman/mm/var/mailman.sock;
There are obviously a couple of places where the http-socket on port 8000 is referenced (e.g. mailman_web/settings/mailman.py, /opt/mailman/mm/hyperkitty.cfg). Is there an overview of the required changes when using a different (file) socket?
If you configure uwsgi with only a unix socket and no http-socket,
nothing will be listening on port 8000. Thus, you need to configure
those references to public facing URLs and let ngnix determine where to go.
e.g., for mailman-hyperkitty.cfg
[general]
base_url: http://example.com/archives/
and in settings.py
POSTORIUS_TEMPLATE_BASE_URL = 'https://example.com'
to override the localhost:8000 reference in
mailman_web/settings/mailman.py.
Those two are the only ones. While you might be able to change the references to something like
http+unix://%2fopt%2fmailman%2fmm%2fvar%2fmailman.sock/...
that may not work, and it's simpler to let ngnix handle it.
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 2022-11-26 12:02, Mark Sapiro wrote:
On 11/25/22 23:00, Jan Eden via Mailman-users wrote:
proxy_pass http://127.0.0.1:8000; → proxy_pass http://unix:/opt/mailman/mm/var/mailman.sock;
and received an error message (as expected):
Because that won't work. You can't http to a unix socket in that way.
Use
uwsgi_pass unix:/opt/mailman/mm/var/mailman.sock;There are obviously a couple of places where the http-socket on port 8000 is referenced (e.g. mailman_web/settings/mailman.py, /opt/mailman/mm/hyperkitty.cfg). Is there an overview of the required changes when using a different (file) socket?
If you configure uwsgi with only a unix
socketand nohttp-socket, nothing will be listening on port 8000. Thus, you need to configure those references to public facing URLs and let ngnix determine where to go.e.g., for mailman-hyperkitty.cfg
[general] base_url: http://example.com/archives/and in settings.py
POSTORIUS_TEMPLATE_BASE_URL = 'https://example.com'to override the
localhost:8000reference in mailman_web/settings/mailman.py.
Thank you! This all makes sense, but uwsgi logged the following error after applying the changes above:
Traceback (most recent call last): File "/opt/mailman/venv/lib/python3.10/site-packages/django/core/handlers/wsgi.py", line 130, in __call__ request = self.request_class(environ) File "/opt/mailman/venv/lib/python3.10/site-packages/django/core/handlers/wsgi.py", line 78, in __init__ self.method = environ["REQUEST_METHOD"].upper() KeyError: 'REQUEST_METHOD' Traceback (most recent call last): File "/opt/mailman/venv/lib/python3.10/site-packages/django/core/handlers/wsgi.py", line 130, in __call__ request = self.request_class(environ) File "/opt/mailman/venv/lib/python3.10/site-packages/django/core/handlers/wsgi.py", line 78, in __init__ self.method = environ["REQUEST_METHOD"].upper() KeyError: 'REQUEST_METHOD'
So I checked my local Django configuration and found an include directive in mysite_nginx.conf (include /etc/nginx/uwsgi_params;) where the referenced file uwsgi_params contained the following:
uwsgi_param QUERY_STRING $query_string; uwsgi_param REQUEST_METHOD $request_method; uwsgi_param CONTENT_TYPE $content_type; uwsgi_param CONTENT_LENGTH $content_length;
uwsgi_param REQUEST_URI $request_uri; uwsgi_param PATH_INFO $document_uri; uwsgi_param DOCUMENT_ROOT $document_root; uwsgi_param SERVER_PROTOCOL $server_protocol; uwsgi_param REQUEST_SCHEME $scheme; uwsgi_param HTTPS $https if_not_empty;
uwsgi_param REMOTE_ADDR $remote_addr; uwsgi_param REMOTE_PORT $remote_port; uwsgi_param SERVER_PORT $server_port; uwsgi_param SERVER_NAME $server_name;
Replicating this setup on the server solved the KeyError problem, everything works now! I just wonder why these uwsgi parameters need to be passed explicitly when using a file socket, but not with a http socket.
Another (unrelated) question: Why is mailman_hyperkitty.cfg placed in /opt/mailman/mm and not in /etc/mailman3 with the other configuration files when using the virtualenv installation instructions (https://docs.mailman3.org/en/latest/install/virtualenv.html#virtualenv-insta...
Thanks again for all your help (and patience)!
- Jan
On 11/26/22 21:29, Jan Eden via Mailman-users wrote:
Replicating this setup on the server solved the KeyError problem, everything works now! I just wonder why these uwsgi parameters need to be passed explicitly when using a file socket, but not with a http socket.
You are doing more than just switching from a tcp socket to a unix socket. You are also switching from communicating with uwsgi via http protocol to uwsgi protocol.
Another (unrelated) question: Why is mailman_hyperkitty.cfg placed in /opt/mailman/mm and not in /etc/mailman3 with the other configuration files when using the virtualenv installation instructions (https://docs.mailman3.org/en/latest/install/virtualenv.html#virtualenv-insta...
mailman_hyperkitty and its mailman-hyperkitty.cfg are part of mailman core, not mailman-web. If you want, you could put mailman-hyperkitty.cfg in /etc/mailman3/ by putting
configuration: /etc/mailman3/mailman-hyperkitty.cfg
in the [archiver.hyperkitty] section of mailman.cfg, but as it really
belongs to core and not mailman-web we choose to put it in /opt/mailman/mm/.
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 2022-11-26 22:59, Mark Sapiro wrote:
Another (unrelated) question: Why is mailman_hyperkitty.cfg placed in /opt/mailman/mm and not in /etc/mailman3 with the other configuration files when using the virtualenv installation instructions (https://docs.mailman3.org/en/latest/install/virtualenv.html#virtualenv-insta...
mailman_hyperkitty and its mailman-hyperkitty.cfg are part of mailman core, not mailman-web. If you want, you could put mailman-hyperkitty.cfg in /etc/mailman3/ by putting
configuration: /etc/mailman3/mailman-hyperkitty.cfgin the
[archiver.hyperkitty]section of mailman.cfg, but as it really belongs to core and not mailman-web we choose to put it in /opt/mailman/mm/.
I see. But the mailman.cfg is placed in /etc/mailman3, and this file also belongs to the core, doesn't it? In any case, I changed the configuration parameter for [archiver.hyperkitty] as you suggested and all settings are in one place now.
I will post two more questions to this list today, and promise to stop spamming you afterwards. :)
Thanks again, Jan
On 11/27/22 00:19, Jan Eden via Mailman-users wrote:
I see. But the mailman.cfg is placed in /etc/mailman3, and this file also belongs to the core, doesn't it? In any case, I changed the configuration parameter for [archiver.hyperkitty] as you suggested and all settings are in one place now.
In that case, it's probably just an oversight. The instructions at https://docs.mailman3.org/en/latest/install/virtualenv.html are an evolution of a process that started with an obsolete mailman-bundler package, the initial iterations of which are documented at https://wiki.list.org/x/17891998 and prior revisions of that page. https://docs.mailman3.org/en/latest/install/virtualenv.html is still a work in progress.
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 2022-11-27 07:08, Mark Sapiro wrote:
On 11/27/22 00:19, Jan Eden via Mailman-users wrote:
I see. But the mailman.cfg is placed in /etc/mailman3, and this file also belongs to the core, doesn't it? In any case, I changed the configuration parameter for [archiver.hyperkitty] as you suggested and all settings are in one place now.
In that case, it's probably just an oversight. The instructions at https://docs.mailman3.org/en/latest/install/virtualenv.html are an evolution of a process that started with an obsolete mailman-bundler package, the initial iterations of which are documented at https://wiki.list.org/x/17891998 and prior revisions of that page. https://docs.mailman3.org/en/latest/install/virtualenv.html is still a work in progress.
Ok – as long as I can consolidate all settings in one place for backup purposes (with symlinks, if need be), any layout is fine.
Thank you, Jan
On 11/27/22 10:38, Jan Eden via Mailman-users wrote:
On 2022-11-27 07:08, Mark Sapiro wrote:
In that case, it's probably just an oversight. The instructions at https://docs.mailman3.org/en/latest/install/virtualenv.html are an evolution of a process that started with an obsolete mailman-bundler package, the initial iterations of which are documented at https://wiki.list.org/x/17891998 and prior revisions of that page. https://docs.mailman3.org/en/latest/install/virtualenv.html is still a work in progress.
Ok – as long as I can consolidate all settings in one place for backup purposes (with symlinks, if need be), any layout is fine.
I think it was an oversight. See https://gitlab.com/mailman/mailman-suite-doc/-/issues/46 which I fixed with https://gitlab.com/mailman/mailman-suite-doc/-/merge_requests/121
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mark Sapiro writes:
On 11/27/22 00:19, Jan Eden via Mailman-users wrote:
I see. But the mailman.cfg is placed in /etc/mailman3, and this file also belongs to the core, doesn't it? In any case, I changed the configuration parameter for [archiver.hyperkitty] as you suggested and all settings are in one place now.
In that case, it's probably just an oversight.
Yes, I think that's what's going on. There's been quite a bit of simplification and unification of the Django configuration over time, which is ongoing and not 100% reflected in all documentation.
Because of the way most Linux distros work (and I think it's probably true for *BSD and other POSIX systems), it makes sense to put users' site configurations in /etc. I don't see a particular reason why with the current relatively small set of config files we'd want to put HyperKitty or Postorius files elsewhere than /etc/mailman3 (although it might make sense to have more separation given the existence of EMWD's Affinity etc).
Steve
participants (3)
-
Jan Eden -
Mark Sapiro -
Stephen J. Turnbull