Oh! I just noticed Jeff and Mark's posts about the solution for this issue.
I just adjusted my systemd unit file and, of course, that worked for me as well.
Thank you so much for your help on this, everyone!
Best, Michela
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Tuesday, May 11, 2021 1:37 PM, Michela via Mailman-users <mailman-users@mailman3.org> wrote:
Hi. Thank you for your replies, everyone!
Perhaps I should first mention that I checked uwsgi.ini and it is set to /etc/mailman3/, which is where my configuration files actually are. I had a typo in my message to the list about that (but not on the server), where I accidentally omitted the "3" in the directory path. So everything was actually in order there. Good catch in any case! :)
The start of the uwsgi.ini file is as follows:
*** Starting uWSGI 2.0.19.1 (64bit) on [Mon May 10 11:11:01 2021] *** compiled with version: 8.3.1 20191121 (Red Hat 8.3.1-5.0.2) on 10 May 2021 00:42:28 os: Linux-5.4.17-2102.200.13.el8uek.x86_64 #2 SMP Sun Mar 28 14:48:36 PDT 2021 nodename: hostname machine: x86_64 clock source: unix detected number of CPU cores: 2 current working directory: / detected binary path: /opt/mailman/venv/bin/uwsgi !!! no internal routing support, rebuild with pcre support !!! your processes number limit is 6642 your memory page size is 4096 bytes detected max file descriptor number: 1024 lock engine: pthread robust mutexes thunder lock: disabled (you can enable it with --thunder-lock) uwsgi socket 0 bound to TCP address 0.0.0.0:8000 fd 10 Python version: 3.6.8 (default, Mar 9 2021, 15:28:46) [GCC 8.3.1 20191121 (Red Hat 8.3.1-5.0.1)] PEP 405 virtualenv detected: /opt/mailman/venv/ Set PythonHome to /opt/mailman/venv/ Python main interpreter initialized at 0x10ccf30 python threads support enabled your server socket listen backlog is limited to 100 connections your mercy for graceful operations on workers is 60 seconds mapped 250032 bytes (244 KB) for 4 cores *** Operational MODE: preforking+threaded *** added /etc/mailman3/ to pythonpath. Traceback (most recent call last): File "/opt/mailman/venv/lib64/python3.6/site-packages/mailman_web/wsgi.py", line 16, in <module> application = get_wsgi_application() File "/opt/mailman/venv/lib64/python3.6/site-packages/django/core/wsgi.py", line 12, in get_wsgi_application django.setup(set_prefix=False) File "/opt/mailman/venv/lib64/python3.6/site-packages/django/init.py", line 19, in setup configure_logging(settings.LOGGING_CONFIG, settings.LOGGING) File "/opt/mailman/venv/lib64/python3.6/site-packages/django/conf/init.py", line 76, in getattr self._setup(name) File "/opt/mailman/venv/lib64/python3.6/site-packages/django/conf/init.py", line 63, in _setup self._wrapped = Settings(settings_module) File "/opt/mailman/venv/lib64/python3.6/site-packages/django/conf/init.py", line 142, in init mod = importlib.import_module(self.SETTINGS_MODULE) File "/usr/lib64/python3.6/importlib/init.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) ModuleNotFoundError: No module named 'settings' unable to load app 0 (mountpoint='') (callable not found or import error) *** no app loaded. going in full dynamic mode *** *** uWSGI is running in multiple interpreter mode *** spawned uWSGI master process (pid: 38878) spawned uWSGI worker 1 (pid: 38882, cores: 2) spawned uWSGI worker 2 (pid: 38883, cores: 2) [uwsgi-daemons] spawning "/opt/mailman/venv/bin/mailman-web qcluster" (uid: 1002 gid: 1002) Traceback (most recent call last): File "/opt/mailman/venv/lib64/python3.6/site-packages/django/core/management/init.py", line 224, in fetch_command app_name = commands[subcommand] KeyError: 'qcluster'
Running qcluster interactively does appear to work properly:
(venv) [mailman@hostname ~]$ /opt/mailman/venv/bin/mailman-web qcluster /opt/mailman/venv/lib64/python3.6/site-packages/django_q/conf.py:138: UserWarning: Retry and timeout are misconfigured. Set retry larger than timeout, failure to do so will cause the tasks to be retriggered before completion. See https://django-q.readthedocs.io/en/latest/configure.html#retry for details. See https://django-q.readthedocs.io/en/latest/configure.html#retry for details.""") 20:16:21 [Q] INFO Q Cluster berlin-angel-south-california starting. 20:16:21 [Q] INFO Process-1:1 ready for work at 57948 20:16:21 [Q] INFO Process-1:2 ready for work at 57949 20:16:21 [Q] INFO Process-1:3 monitoring at 57950 20:16:21 [Q] INFO Process-1 guarding cluster berlin-angel-south-california 20:16:21 [Q] INFO Q Cluster berlin-angel-south-california running. 20:16:21 [Q] INFO Process-1:4 pushing tasks at 57951
My uwsgi.ini file:
/etc/mailman3/uwsgi.ini
========================
==
[uwsgi]
Port on which uwsgi will be listening.
=======================================
uwsgi-socket = 0.0.0.0:8000
If running uwsgi from the virtual environment ...
==================================================
virtualenv = /opt/mailman/venv/ module=mailman_web.wsgi:application
Add to python import path.
===========================
pythonpath = /etc/mailman3/
The default settings module.
=============================
env = DJANGO_SETTINGS_MODULE=settings
Setup default number of processes and threads per process.
===========================================================
master = true processes = 2 threads = 2
Setup the django_q related worker processes.
=============================================
attach-daemon = /opt/mailman/venv/bin/mailman-web qcluster
Setup the request log.
=======================
req-logger = file:/opt/mailman/web/logs/uwsgi.log
Log qcluster commands separately.
==================================
logger = qcluster file:/opt/mailman/web/logs/uwsgi-qcluster.log log-route = qcluster uwsgi-daemons
Last log and it logs the rest of the stuff.
============================================
logger = file:/opt/mailman/web/logs/uwsgi-error.log
Files in /etc/mailman3/
(venv) [mailman@hostname ~]$ ls /etc/mailman3/ mailman.cfg pycache/ settings.py uwsgi.ini
Regarding Steve's comment on WSGI vs. Gunicorn, we used WSGI as the application server for Mailman 3 because that's what the Mailman 3 venv instructions say to use -- I did my best to follow them exactly (the only difference is in the way I installed PostgreSQL -- I used the official EL PostgreSQL repository for that). So, hopefully, the developers consider it okay for us to use UWSGI (at least for now). :D
Thank you for your help!! Michela
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Monday, May 10, 2021 10:08 PM, Abhilash Raj maxking@asynchronous.in wrote:
On May 10, 2021, at 8:11 PM, Stephen J. Turnbull turnbull.stephen.fw@u.tsukuba.ac.jp wrote: Michela via Mailman-users writes:
The log file, /opt/mailman/web/logs/uwsgi-error.log, contains the following stack trace, however. Note the bit toward the bottom: "ModuleNotFoundError: No module named 'settings'"
Given that you can run the applications from the command line as systemd services, this almost certainly localizes the problem to uwsgi itself. Note that uwsgi has been problematic lately. I don't recall the details. I think there's a preference for gunicorn as the SGI provider among the Mailman 3 developers (and it's what core uses internally to provide the REST interface) so uwsgi may not get "beta" testing in production from us. Specifically, is this line pythonpath = /etc/mailman3/ present, correct, and not commented out in uswgi.ini?
My configuration files outside of the Mailman virtual environment are in /etc/mailman/:
Is it possible you have /etc/mailman/ instead of /etc/mailman3/ in uwsgi.ini as in the above quote? (I am pretty sure that's a typo in the post, but gotta check everything.) On the other hand, maybe we did a typo, and there should be no trailing slash on /etc/mailman3 in this statement. (Usually trailing slashes aren't a problem, applications mostly ignore them, but there is no trailing slash on any directory in my sys.path, so it's worth a try.)
Yeah, it would be good to share the full contents of /etc/mailman3/uwsgi.ini, for some reason what is listed in the docs1 is working on one of my test servers. Although, given that there are more than one reports of the same error, it is possible something is wrong or missing in the docs. The trailing slashes in this case isn’t really a problem I think. If you can share some initial parts of your uwsgi-error.log part where uwsgi initializes that would be useful too. I am looking something that looks like this:
*** Starting uWSGI 2.0.19.1 (64bit) on [Thu Jan 21 08:49:16 2021] *** compiled with version: 9.3.0 on 21 January 2021 08:40:15 os: Linux-5.4.0-1036-azure #38-Ubuntu SMP Wed Jan 6 18:26:25 UTC 2021 nodename: mailman machine: x86_64 clock source: unix detected number of CPU cores: 2 current working directory: /opt/mailman3 detected binary path: /opt/mailman3/venv/bin/uwsgi !!! no internal routing support, rebuild with pcre support !!! your processes number limit is 15571 your memory page size is 4096 bytes detected max file descriptor number: 1024 lock engine: pthread robust mutexes thunder lock: disabled (you can enable it with --thunder-lock) uwsgi socket 0 bound to TCP address 127.0.0.1:8000 fd 8 Python version: 3.8.5 (default, Jul 28 2020, 12:59:40) [GCC 9.3.0] PEP 405 virtualenv detected: /opt/mailman3/venv/ Set PythonHome to /opt/mailman3/venv/ Python main interpreter initialized at 0x56002ba3aed0 python threads support enabled your server socket listen backlog is limited to 100 connections your mercy for graceful operations on workers is 60 seconds mapped 166688 bytes (162 KB) for 2 cores *** Operational MODE: threaded *** added /etc/mailman3/ to pythonpath. WSGI app 0 (mountpoint='') ready in 1 seconds on interpreter 0x56002ba3aed0 pid: 715893 (default app) *** uWSGI is running in multiple interpreter mode *** spawned uWSGI master process (pid: 715893) spawned uWSGI worker 1 (pid: 715897, cores: 2) [uwsgi-daemons] spawning "mailman-web qcluster" (uid: 1001 gid: 1001) 08:49:18 [Q] INFO Q Cluster december-nebraska-helium-montana starting. 08:49:18 [Q] INFO Process-1:1 ready for work at 715904 08:49:18 [Q] INFO Process-1:2 ready for work at 715905Please ignore that the paths in my logs are
/opt/mailman3/*, I have multiple installations on my test server, the docs say it should be/opt/mailman/*. It doesn’t really matter as long as they are consistent in all places. Also, what happens if you run the command "/opt/mailman/venv/bin/mailman-web qcluster” as a “mailman” user on your machine? Do you see the traceback like above or does it start a process like this?mailman@mailman:~$ /opt/mailman3/venv/bin/mailman-web qcluster 05:05:29 [Q] INFO Q Cluster april-east-hot-potato starting. 05:05:29 [Q] INFO Process-1:1 ready for work at 3364005 05:05:29 [Q] INFO Process-1:2 ready for work at 3364006 05:05:29 [Q] INFO Process-1 guarding cluster april-east-hot-potato 05:05:29 [Q] INFO Process-1:3 monitoring at 3364007 05:05:29 [Q] INFO Q Cluster april-east-hot-potato running. 05:05:29 [Q] INFO Process-1:4 pushing tasks at 3364008If it's none of the above, I'm out of ideas. Maybe Mark or Abhilash has more experience. Steve Mailman-users mailing list -- mailman-users@mailman3.org To unsubscribe send an email to mailman-users-leave@mailman3.org https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
-- thanks, Abhilash Raj (maxking)
Mailman-users mailing list -- mailman-users@mailman3.org To unsubscribe send an email to mailman-users-leave@mailman3.org https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/