Mailman3 not working following distribution upgrade from Debian 10 to 11.
I performed a distribution upgrade from Debian 10 Buster (which runs mailman 3.2.1) to Debian 11 Bullseye (which runs mailman 3.3.1) with the command "apt-get full-upgrade -y" and everything apparently worked properly except at the end of the upgrade when the following error dialog was generated by the mailman3-web and mailman3-full post-installation scripts:
Setting up mailman3-web (0+20200530-2) ... Determining localhost credentials from /etc/mysql/debian.cnf: succeeded. dbconfig-common: writing config to /etc/dbconfig-common/mailman3-web.conf dbconfig-common: flushing administrative password sed: -e expression #2, char 77: unterminated `s' command dpkg: error processing package mailman3-web (--configure): installed mailman3-web package post-installation script subprocess returned error exit status 1 dpkg: dependency problems prevent configuration of mailman3-full: mailman3-full depends on mailman3-web; however: Package mailman3-web is not configured yet.
dpkg: error processing package mailman3-full (--configure): dependency problems - leaving unconfigured Errors were encountered while processing: mailman3-web mailman3-full E: Sub-process /usr/bin/dpkg returned an error code (1)
Now, every time that I execute "apt-get upgrade" then I continue to get the above error dialog. Also, the command "dpkg-reconfigure mailman3-web" generates this output:
/usr/sbin/dpkg-reconfigure: mailman3-web is broken or not fully installed
So, I am now stuck since apt-get is saying that mailman3-web is not configured yet, however, neither apt-get nor dpkg-reconfigure will successfully configure mailman3-web.
Does anybody have any idea how to fix this? I would very much appreciate some suggestions.
Thanks,
Gordon
"gordon" == gordon <gordon@dickens.com> writes:
gordon> I performed a distribution upgrade from Debian 10 Buster gordon> (which runs mailman 3.2.1) to Debian 11 Bullseye (which runs gordon> mailman 3.3.1) with the command "apt-get full-upgrade -y" and gordon> everything apparently worked properly except at the end of the gordon> upgrade when the following error dialog was generated by the
Suggest reporting this as a bug to the Debian maintainers, as it's to do with their packaging. Use the 'reportbug' command.
Peter C
On 1/6/22 2:40 PM, gordon@dickens.com wrote:
the mailman3-web and mailman3-full post-installation scripts
These look like the Debian packages. Mark Sapiro can speak to this much better than I can, but the packages both on 10 and 11 are outdated and have some components that don't work, or don't work well.
Better IMO to uninstall these and install current versions using the venv and web instructions:
https://docs.mailman3.org/en/latest/install/virtualenv.html
https://docs.mailman3.org/en/latest/config-web.html
There is some overlap between these docs, but I can verify that this setup worked on Debian 10, and continued to work after upgrading to Debian 11.
dn
On 1/6/22 2:40 PM, gordon@dickens.com wrote:
I performed a distribution upgrade from Debian 10 Buster (which runs mailman 3.2.1) to Debian 11 Bullseye (which runs mailman 3.3.1) with the command "apt-get full-upgrade -y" and everything apparently worked properly except at the end of the upgrade when the following error dialog was generated by the mailman3-web and mailman3-full post-installation scripts:
Setting up mailman3-web (0+20200530-2) ... Determining localhost credentials from /etc/mysql/debian.cnf: succeeded. dbconfig-common: writing config to /etc/dbconfig-common/mailman3-web.conf dbconfig-common: flushing administrative password sed: -e expression #2, char 77: unterminated `s' command dpkg: error processing package mailman3-web (--configure): installed mailman3-web package post-installation script subprocess returned error exit status 1 dpkg: dependency problems prevent configuration of mailman3-full: mailman3-full depends on mailman3-web; however: Package mailman3-web is not configured yet.
dpkg: error processing package mailman3-full (--configure): dependency problems - leaving unconfigured Errors were encountered while processing: mailman3-web mailman3-full E: Sub-process /usr/bin/dpkg returned an error code (1)
Now, every time that I execute "apt-get upgrade" then I continue to get the above error dialog. Also, the command "dpkg-reconfigure mailman3-web" generates this output:
/usr/sbin/dpkg-reconfigure: mailman3-web is broken or not fully installed
So, I am now stuck since apt-get is saying that mailman3-web is not configured yet, however, neither apt-get nor dpkg-reconfigure will successfully configure mailman3-web.
Does anybody have any idea how to fix this? I would very much appreciate some suggestions.
This is a Debian packaging issue. It looks like the underlying error is
sed: -e expression #2, char 77: unterminated `s' command
which is an issue in one of the installation scripts in the .deb for mailman3-web. Your best source of help for this is Debian.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Peter, David and Mark, Thanks very much for your prompt replies.
I have reported this issue to the Debian folks using the Debian "reportbug" command.
How do I uninstall the Debian version and then reinstall with the venv and web instructions so that all of my mailing lists and configuration data are preserved? I may elect to go that route if I can preserve all of my mailing list and data and you think that would be a quicker route to a solution. Is there a simple way to do this?
Thanks,
Gordon
On 1/6/22 3:25 PM, gordon@dickens.com wrote:
How do I uninstall the Debian version and then reinstall with the venv and web instructions so that all of my mailing lists and configuration data are preserved? I may elect to go that route if I can preserve all of my mailing list and data and you think that would be a quicker route to a solution. Is there a simple way to do this?
It's probably not "simple", but here's how I would do it.
First, follow
https://docs.mailman3.org/en/latest/install/virtualenv.html to install
Mailman in a virtualenv. You can use list as the Mailman user rather
than mailman if you like.
The doc uses Postgreql as the database manager, but you want to use whatever the current DBM is because you will want to use the existing database which is where all the list configuration, membership and archives are.
One gotcha here is the instructions tell you to create /etc/mailman3/mailman.cfg and the Debian package already puts its mailman.cfg there. The Debian mailman.cfg may actually be appropriate although it puts things in different places, e.g. /var/lib/mailman3 instead of /opt/mailman/mm/var.
The Debian package puts its Django settings in /etc/mailman3/mailman-web.py and the virtualenv doc puts them in /etc/mailman3/settings.py. You need to reconcile the differences keeping in mind that the recommended settings.py imports most settings from the mailman-web project and only contains things you want to change locally.
The basic idea is you want the virtualenv install to use the existing databases and /var/lib/mailman3 stuff so you preserve all your existing list stuff.
It probably involves several gotchas that might be difficult to resolve if you are less than intimate with the details of a Mailman installation.
I actually spun up a small Ubuntu 20.04 droplet and installed the mailman3-full package to see what things looked like, but I destroyed it without examining the contents of some of the files or I might know more.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
gordon@dickens.com writes:
How do I uninstall the Debian version and then reinstall with the venv and web instructions so that all of my mailing lists and configuration data are preserved?
tar czf /tmp/mailmandata.tar.gz $PATH_TO_MAILMAN_DATA
Not sure where Debian puts it but I would guess /var/lib/mailman. You can also try apt-get --dry-run remove ... but I think that provides output at too coarse a level (packages, not files). Generally apt is pretty good about not removing files that are not in the package, though.
I may elect to go that route if I can preserve all of my mailing list and data and you think that would be a quicker route to a solution. Is there a simple way to do this?
Then you can do apt-get remove $MAILMAN_PACKAGES with confidence. Do not do apt-get purge ..., of course (although I suspect even that will leave your archives and mailing lists).
We still generally recommend installing from source, or possibly using the docker containers that Abhilash provides (usually quite promptly after a release). First, you're not dependent on a distro for new features, and Mailman 3 is still by no means feature-complete. (On the other hand, the distros are good about providing security upgrades, and often the easiest route for them is a version bump). Second, if you put stuff where we tell you to, we know where it is when you ask for advice. I'm a Debian user, but I don't use their packages for Mailman itself.
Steve
I agree that this is a Debian packaging issue and I agree with Mark that the underlying issue is the "sed:" error message. I have only had one reply to my Debian bug report. That reply told me what I had to do make apt-get usable again (since it was stuck on the mailman3 error) but that post did not address the "sed:" bug itself and nobody else has responded to the bug report. So, it looks like I am on my own as far as my Debian mailman3 installation is concerned.
So, here is what I plan to do:
- apt-get remove mailman3 mailman3-web mailman3-full. This should not remove any configuration data or database files but I will run "apt-get --dry-run remove" first just to make sure.
- Install mailman3 using the venv and web instructions as suggested in this thread. I plan to use mysql (mariadb) since my database was originally implemented with mysql.
- I will poach as much of the configuration data as I can find from my original Debian mailman3 installation.
Thanks to everyone for your support and replys!
Gordon
On 1/11/22 12:39 PM, gordon@dickens.com wrote:
- Install mailman3 using the venv and web instructions as suggested in this thread. I plan to use mysql (mariadb) since my database was originally implemented with mysql.
I too use MariaDB. Here are the MariaDB-specific variations from the venv and web docs:
- Before going through the venv docs I installed these packages:
libmariadb3 libmariadb-dev default-libmysqlclient-dev
- When installing mailman in venv, I did this:
(venv)$ pip install --upgrade pip (venv)$ pip install wheel mailman pymysql mysqlclient
- In /etc/mailman.cfg, the database config looks like this:
..
[database] class: mailman.database.mysql.MySQLDatabase url: mysql+pymysql://myuser:mypassword@mymysqlhost/mailman?charset=utf8mb4&use_unicode=1
..
The url is all one line (it's wrapping in my mail client).
- The [Unit] section of /etc/systemd/system/mailman3.service looks like this:
[Unit] Description=GNU Mailing List Manager After=syslog.target network.target mariadb.service
..
Note that this is on a Debian 11 system. On Debian 10 and previous, it'd be mysql.service instead of mariadb.service.
- The settings.py file includes this section:
..
# MySQL database setup. DATABASES = { 'default': { 'ENGINE': 'django.db.backends.mysql', 'NAME': 'mailmanweb', 'USER': 'mailman', 'PASSWORD': '<password>', 'HOST': 'localhost', 'PORT': '', 'OPTIONS': {'charset': 'utf8mb4'} # Enable utf8 4-byte encodings. } }
..
Note that your NAME, USER, PASSWORD, and possibly HOST may differ.
- The [Unit] section of /etc/systemd/system/mailmanweb.service looks like this:
[Unit] Description=GNU Mailman Web UI After=syslog.target network.target mariadb.service mailman3.service
..
Same caveat as with step 4 that this is for Debian 11, and you'd use mysql.service instead on Debian 10 and previous.
I think that's all the changes, but if I've omitted something I hope Mark Sapiro or someone else can step in.
dn
Thanks to David and everybody else that have replied to my problem. I have proceeded to uninstall the debian mailman3 packages and then endeavored to install everything with venv. I have implemented all of David's configuration suggestions which were very helpful. Now, mailman3 (core) appears to be working properly in that I can start/stop/restart/status the service. Also, the command "mailman info" from the venv prompt yields output which appears to be correct and the same as in the venv instructions. However, I have not been able to get mailman3-web working.
All of my configuration files may be viewed here:
https://dickens.com/etc-mailman3/
The above files are basically a mirror of the contents of my /etc/mailman3 directory except that I have obfuscated sensitive info such as keys, passwords and email addresses.
Executing the mailman3-web service script with the command "systemctl start mailman3-web" starts everything without errors, however, the command "systemctl status mailman3-web" then generates the following output with errors:
● mailman3-web.service - GNU Mailman Web UI Loaded: loaded (/etc/mailman3/mailman3-web.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Tue 2022-01-25 17:10:30 EST; 6s ago Process: 1718478 ExecStart=/opt/mailman/venv/bin/uwsgi --ini /etc/mailman3/uwsgi.ini (code=exited, status=1/FAILURE) Main PID: 1718478 (code=exited, status=1/FAILURE) CPU: 12ms
Jan 25 17:10:29 slc.mailhub4u.com systemd[1]: Started GNU Mailman Web UI. Jan 25 17:10:30 slc.mailhub4u.com uwsgi[1718478]: [uWSGI] getting INI configuration from /etc/mailman3/uwsgi.ini Jan 25 17:10:30 slc.mailhub4u.com systemd[1]: mailman3-web.service: Main process exited, code=exited, status=1/FAILURE Jan 25 17:10:30 slc.mailhub4u.com systemd[1]: mailman3-web.service: Failed with result 'exit-code'.
Also, /var/log/syslog and "journalctl | grep mailman3-web" contain these two lines of errors:
Jan 25 14:00:46 slc systemd[1]: mailman3-web.service: Main process exited, code=exited, status=1/FAILURE Jan 25 14:00:46 slc systemd[1]: mailman3-web.service: Failed with result 'exit-code'.
Also, other commands in the venv instructions do not work such as "mailman-web migrate" which produces volumes of error dialog starting with:
Traceback (most recent call last): File "/opt/mailman/venv/lib/python3.9/site-packages/django/template/utils.py", line 66, in __getitem__ return self._engines[alias] KeyError: 'django'
Then these two lines are then followed by over 50 lines of django and python error messages.
I have not been able to find any hints or suggestions as to what I am doing wrong in my server's logs. FWIW, I have been working in the Unix/Linux command line environment for over 35 years and I have extensive LAMP, EXIM and Mailman2 experience. However, I have not ever worked with Django. I am usually able to resolve issues like this quickly, however, I am at a total loss as to what to do now with this mailman3 installation. I suspect that my unfamiliarity with Django may be my achilles heel.
So, I am hoping that somebody here will soon be able to offer suggestions that might help me get everything working properly. I have several organization's mailings lists on this server and I need to get this fixed ASAP especially since my mailman installation has been down now for almost 3 weeks. As you might imagine, this is causing me alot of grief with my mailman3 users. So, please let me know any suggestions that you folks may have to get my mailman3 installation working again.
Many Thanks,
Gordon
On 1/25/22 2:59 PM, gordon@dickens.com wrote:
Thanks to David and everybody else that have replied to my problem. I have proceeded to uninstall the debian mailman3 packages and then endeavored to install everything with venv. I have implemented all of David's configuration suggestions which were very helpful. Now, mailman3 (core) appears to be working properly in that I can start/stop/restart/status the service. Also, the command "mailman info" from the venv prompt yields output which appears to be correct and the same as in the venv instructions. However, I have not been able to get mailman3-web working.
One general comment: Debian packages install files with a different owner than the pip install. You want to be sure all your directories and files are owned by the Mailman user ('mailman' on Debian).
All of my configuration files may be viewed here:
https://dickens.com/etc-mailman3/
The above files are basically a mirror of the contents of my /etc/mailman3 directory except that I have obfuscated sensitive info such as keys, passwords and email addresses.
Executing the mailman3-web service script with the command "systemctl start mailman3-web" starts everything without errors, however, the command "systemctl status mailman3-web" then generates the following output with errors:
● mailman3-web.service - GNU Mailman Web UI Loaded: loaded (/etc/mailman3/mailman3-web.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Tue 2022-01-25 17:10:30 EST; 6s ago Process: 1718478 ExecStart=/opt/mailman/venv/bin/uwsgi --ini /etc/mailman3/uwsgi.ini (code=exited, status=1/FAILURE) Main PID: 1718478 (code=exited, status=1/FAILURE) CPU: 12ms
Jan 25 17:10:29 slc.mailhub4u.com systemd[1]: Started GNU Mailman Web UI. Jan 25 17:10:30 slc.mailhub4u.com uwsgi[1718478]: [uWSGI] getting INI configuration from /etc/mailman3/uwsgi.ini Jan 25 17:10:30 slc.mailhub4u.com systemd[1]: mailman3-web.service: Main process exited, code=exited, status=1/FAILURE Jan 25 17:10:30 slc.mailhub4u.com systemd[1]: mailman3-web.service: Failed with result 'exit-code'.
Also, /var/log/syslog and "journalctl | grep mailman3-web" contain these two lines of errors:
Jan 25 14:00:46 slc systemd[1]: mailman3-web.service: Main process exited, code=exited, status=1/FAILURE Jan 25 14:00:46 slc systemd[1]: mailman3-web.service: Failed with result 'exit-code'.
Do you have anything further in any of the logs in /opt/mailman/web/logs ?
Also, other commands in the venv instructions do not work such as "mailman-web migrate" which produces volumes of error dialog starting with:
Traceback (most recent call last): File "/opt/mailman/venv/lib/python3.9/site-packages/django/template/utils.py", line 66, in __getitem__ return self._engines[alias] KeyError: 'django'
This one might be due to a bug I hit as well. There's a problem with mistune, one of the components. If that's the issue here, this should address it, run as the mailman user:
pip install mistune==2.0.0rc1 mailman-web migrate mailman-web compress mailman-web collectstatic mailman-web compilemessages
dn
David Newman wrote:
One general comment: Debian packages install files with a different owner than the pip install. You want to be sure all your directories and files are owned by the Mailman user ('mailman' on Debian).
Yes, I have checked the ownerships. The debian package installed everything in mailman3 as user "list" whereas I have now setup everything to run as user "mailman". As I previously mentioned, mailman3 (core) is running fine. In fact mailman3 has properly posted numerous messages to several of my mailing lists since I finished setting it up with pip. However, mailman3-web (or mailman-web) is not working, I think because of a permissions issue. The installation instructions specify that mailman3-web (or mailman-web) run as user=mailman and group=mailman. However, the debian installation specifies the data files as user=www-data and group=www-data which is what apache runs as. For instance, I have my static files stored in the /var/lib/mailman3/web/static directory and the previous Debian mailman3 version set all the ownerships and groups to www-data for /var/lib/mailman3/web, /var/lib/mailman3/web/static along with all of the directories and files in the static subdirectory. So, when I run "mailman-web collectstatic" it bombs on the following permissions error:
PermissionError: [Errno 13] Permission denied: '/var/lib/mailman3/web/static/hyperkitty/js/hyperkitty-common.js'
So, my question is as follows. Should I continue to run mailman3-web as user mailman and change the ownership of the /var/lib/mailman3/web directory to mailman or should I leave the permissions of the /var/lib/mailman3/web directory alone and have mailman3-web run as user and group www-data?
Do you have anything further in any of the logs in /opt/mailman/web/logs ?
Yes, everytime that I try to start mailman3-web it posts this to the log and which includes a permission error as the last line:
*** Starting uWSGI 2.0.20 (64bit) on [Wed Jan 26 17:00:13 2022] *** compiled with version: 10.2.1 20210110 on 24 January 2022 19:51:51 os: Linux-5.10.0-10-amd64 #1 SMP Debian 5.10.84-1 (2021-12-08) nodename: slc.mailhub4u.com machine: x86_64 clock source: unix pcre jit disabled detected number of CPU cores: 8 current working directory: / detected binary path: /opt/mailman/venv/bin/uwsgi chdir() to /usr/share/mailman3-web your processes number limit is 31530 your memory page size is 4096 bytes detected max file descriptor number: 1024 lock engine: pthread robust mutexes thunder lock: disabled (you can enable it with --thunder-lock) bind(): Permission denied [core/socket.c line 230]
This one might be due to a bug I hit as well. There's a problem with mistune, one of the components. If that's the issue here, this should address it, run as the mailman user: pip install mistune==2.0.0rc1 mailman-web migrate mailman-web compress mailman-web collectstatic mailman-web compilemessages
Thanks for that tip David! Reverting mistune back to version 2.0.0rc1 allowed "mailman-web migrate" to work. I think that the balance of my problems are permissions related to running mailman3-web (or mailman-web) as user mailman versus www-data. So, please let me know, based on your installation, if I should change the permissions for /var/lib/mailman3/web/ to user/group mailman or should I run mailman3-web as user/group www-data.
Also, any other advice from David or anyone else would be very much appreciated.
Thanks!
Gordon
On Jan 26, 2022, at 14:35, gordon@dickens.com wrote:
Please let me know, based on your installation, if I should change the permissions for /var/lib/mailman3/web/ to user/group mailman or should I run mailman3-web as user/group www-data.
With a pip installation on Debian 11, everything is owned by mailman.
dn
On 1/26/22 14:34, gordon@dickens.com wrote:
However, mailman3-web (or mailman-web) is not working, I think because of a permissions issue. The installation instructions specify that mailman3-web (or mailman-web) run as user=mailman and group=mailman. However, the debian installation specifies the data files as user=www-data and group=www-data which is what apache runs as. For instance, I have my static files stored in the /var/lib/mailman3/web/static directory and the previous Debian mailman3 version set all the ownerships and groups to www-data for /var/lib/mailman3/web, /var/lib/mailman3/web/static along with all of the directories and files in the static subdirectory. So, when I run "mailman-web collectstatic" it bombs on the following permissions error:
PermissionError: [Errno 13] Permission denied: '/var/lib/mailman3/web/static/hyperkitty/js/hyperkitty-common.js'
So, my question is as follows. Should I continue to run mailman3-web as user mailman and change the ownership of the /var/lib/mailman3/web directory to mailman or should I leave the permissions of the /var/lib/mailman3/web directory alone and have mailman3-web run as user and group www-data?
As David later said, owner and group of all Mailman's files and
directories should be mailman and what ever the wsgi interface is
(uwsgi, gunicorn, mod_wsgi) should run the wsgi process as user:group
mailman:mailman
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Hello everybody,
I successfully resolved the owner:group issue and I am no longer getting the permission errors. Basically, I had to set the ownership:group to mailman:www-data and make the group permissions rw for everything to work successfully with Apache and Mailman3.
As I mentioned earlier, I have mailman (core) now working, however, I cannot seem to get the web interface to work. I believe that I have gotten rid of all of the permissions issues and the mailman3-web service starts and stops without error. However, when I use a browser to go to the web interface, I get File Not Found errors in the browser window and my Apache2 logs say:
173.165.166.34 - root [04/Feb/2022:12:54:06 -0500] "GET /mailman3/postorius/lists/members.georgiajets.org/ HTTP/1.1" 404 282 173.165.166.34 - root [04/Feb/2022:12:54:06 -0500] "GET /favicon.ico HTTP/1.1" 200 318
So, I get File Not Found error 404 for the web page but the favicon.ico file appears to be found successfully. Therefore, I suspect that my apache configuration file for mailman3 may be at the source of my problems and specifically the alias for the static directory. Here is my apache configuration file:
Alias /mailman3/favicon.ico /var/lib/mailman3/web/static/postorius/img/favicon.ico Alias /mailman3/static /var/lib/mailman3/web/static
<Directory "/var/lib/mailman3/web/static"> Require all granted </Directory>
<IfModule mod_proxy_uwsgi.c> ProxyPass /mailman3/favicon.ico ! ProxyPass /mailman3/static ! # ProxyPass /mailman3 unix:/run/mailman3-web/uwsgi.sock|uwsgi://localhost </IfModule>
By the way, this is the Apache configuration file that the Debian package installed with my initial installation.
Does anybody see any problems in my Apache configuration file that could be causing my problems? Do you see or suspect any other issues?
Thanks,
Gordon
On 2/4/22 11:15, gordon@dickens.com wrote:
173.165.166.34 - root [04/Feb/2022:12:54:06 -0500] "GET /mailman3/postorius/lists/members.georgiajets.org/ HTTP/1.1" 404 282 173.165.166.34 - root [04/Feb/2022:12:54:06 -0500] "GET /favicon.ico HTTP/1.1" 200 318
So, I get File Not Found error 404 for the web page but the favicon.ico file appears to be found successfully. Therefore, I suspect that my apache configuration file for mailman3 may be at the source of my problems and specifically the alias for the static directory. Here is my apache configuration file:
It's not the alias for static
Alias /mailman3/favicon.ico /var/lib/mailman3/web/static/postorius/img/favicon.ico Alias /mailman3/static /var/lib/mailman3/web/static
<Directory "/var/lib/mailman3/web/static"> Require all granted </Directory>
<IfModule mod_proxy_uwsgi.c> ProxyPass /mailman3/favicon.ico ! ProxyPass /mailman3/static ! # ProxyPass /mailman3 unix:/run/mailman3-web/uwsgi.sock|uwsgi://localhost </IfModule>
It's the fact that
ProxyPass /mailman3 unix:/run/mailman3-web/uwsgi.sock|uwsgi://localhost
is commented out. This should not be commented assuming you are using uWSGI and it is listening on the socket /run/mailman3-web/uwsgi.sock
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Hi Mark,
Thanks very much for your prompt and helpful reply!
I made the change to the mailman3 configuration for Apache that you recommended since I am using uwsgi. Now, I am able to start/status/stop the mailman3-web service without errors. However, now, when I then attempt to access the mailman installation on my server with my browser, I get the following message in the browser window:
Internal Server Error
I am unable to find any errors in the mailman3 logs or in the Apache error logs. However the Apache access logs have the following:
173.165.166.34 - - [04/Feb/2022:17:06:20 -0500] "GET /mailman3 HTTP/1.1" 500 21
Do you have any idea as to what is going on now? Any suggestions for troubleshooting this?
Here is my /etc/mailman3/settings.py file where I have obfuscated sensitive info:
# Mailman Web configuration file. # /etc/mailman3/settings.py
from mailman_web.settings.base import * from mailman_web.settings.mailman import *
#: Default list of admins who receive the emails from error logging. ADMINS = ( ('Mailman Suite Admin', 'root@localhost'), )
# mysql database setup. DATABASES = { 'default': { 'ENGINE': 'django.db.backends.mysql', 'NAME': 'mailman3web', 'USER': 'mailman', # TODO: Replace this with the password. 'PASSWORD': 'my_password', 'HOST': 'localhost', 'PORT': '', # GLD - add utf8mb4 character set for emoticons to work. 'OPTIONS': {'charset': 'utf8mb4',}, } }
# 'collectstatic' command will copy all the static files here.
# Alias this location from your webserver to /static
STATIC_ROOT = '/var/lib/mailman3/web/static'
# Make sure that this directory is created or Django will fail on start. LOGGING['handlers']['file']['filename'] = '/var/log/mailman3/web/mailman-web.log'
#: See https://docs.djangoproject.com/en/dev/ref/settings/#allowed-hosts ALLOWED_HOSTS = [ "localhost", # Archiving API from Mailman, keep it. # "lists.your-domain.org", # Add here all production domains you have. ]
#: Current Django Site being served. This is used to customize the web host #: being used to serve the current website. For more details about Django #: site, see: https://docs.djangoproject.com/en/dev/ref/contrib/sites/ SITE_ID = 1
# Set this to a new secret value. SECRET_KEY = 'secret_key_value'
# Set this to match the api_key setting in # /opt/mailman/mm/mailman-hyperkitty.cfg (quoted here, not there). MAILMAN_ARCHIVER_KEY = 'archiver_key'
# The sender of emails from Django such as address confirmation requests. # Set this to a valid email address. DEFAULT_FROM_EMAIL = 'postmaster@tld.com'
# The sender of error messages from Django. Set this to a valid email # address. SERVER_EMAIL = 'postmaster@tld.com'
And here is my uwsgi.ini file:
[uwsgi] # Port on which uwsgi will be listening. uwsgi-socket = /run/mailman3-web/uwsgi.sock # http-socket = 0.0.0.0:8000 # http = 0.0.0.0:8000
# If running uwsgi from the virtual environment ... virtualenv = /opt/mailman/venv/
module=mailman_web.wsgi:application
# Set PYTHONPATH env = PYTHONPATH=/etc/mailman3/ # The default settings module. env = DJANGO_SETTINGS_MODULE=settings
#Enable threading for python # GLD - remove the following line from debian uwsgi.ini # enable-threads = true
# Move to the directory wher the django files are. # GLD - remove the following line from debian uwsgi.ini # chdir = /usr/share/mailman3-web
# Use the wsgi file provided with the django project. # GLD - remove the following line from debian uwsgi.ini #wsgi-file = wsgi.py
# Setup default number of processes and threads per process. master = true process = 2 threads = 2
# Drop privielges and don't run as root. # GLD uid = www-data gid = www-data # plugins = python3
# Setup the django_q related worker processes. # attach-daemon = python3 manage.py qcluster attach-daemon = /opt/mailman/venv/bin/mailman-web qcluster
# Setup hyperkitty's cron jobs. #unique-cron = -1 -1 -1 -1 -1 ./manage.py runjobs minutely #unique-cron = -15 -1 -1 -1 -1 ./manage.py runjobs quarter_hourly #unique-cron = 0 -1 -1 -1 -1 ./manage.py runjobs hourly #unique-cron = 0 0 -1 -1 -1 ./manage.py runjobs daily #unique-cron = 0 0 1 -1 -1 ./manage.py runjobs monthly #unique-cron = 0 0 -1 -1 0 ./manage.py runjobs weekly #unique-cron = 0 0 1 1 -1 ./manage.py runjobs yearly
# Setup the request log. req-logger = file:/var/log/mailman3/web/mailman-web.log
# Log cron seperately. #logger = cron file:/var/log/mailman3/web/mailman-web-cron.log #log-route = cron uwsgi-cron
# Log qcluster commands seperately. # GLD - Add the following two lines from mailman3 venv instructions. logger = qcluster file:/var/log/mailman3/web/mailman-web-qcluster.log log-route = qcluster uwsgi-daemons
# Last log and it logs the rest of the stuff. # GLD - Add the following line from mailman3 venv instructions. logger = file:/var/log/mailman3/web/mailman-web-error.log # GLD - remove the following line from debian uwsgi.ini # logto = /var/log/mailman3/web/mailman-web.log
On 2/4/22 14:54, gordon@dickens.com wrote:
Hi Mark,
Thanks very much for your prompt and helpful reply!
I made the change to the mailman3 configuration for Apache that you recommended since I am using uwsgi. Now, I am able to start/status/stop the mailman3-web service without errors. However, now, when I then attempt to access the mailman installation on my server with my browser, I get the following message in the browser window:
Internal Server Error
I am unable to find any errors in the mailman3 logs or in the Apache error logs. However the Apache access logs have the following:
173.165.166.34 - - [04/Feb/2022:17:06:20 -0500] "GET /mailman3 HTTP/1.1" 500 21
Do you have any idea as to what is going on now? Any suggestions for troubleshooting this?
Without more log info, I don't know what the issue is. Have you looked in /var/log/mailman3/mailman.log and /var/log/mailman3/web/mailman-web.log?
Possibly there is a uwsgi log somewhere in /var/log that might have something relevant.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Hi Mark,
I think that I found a hint as to the cause of my problems.
I changed the apache2 LogLevel to debug and the following was then generated in /var/log/apache2/ssl_error_log:
[Fri Feb 11 15:40:18.325675 2022] [core:debug] [pid 3982011] core.c(5280): (95)Operation not supported: [remote /var/lib/mailman3/web/static/uwsgi.sock:12150] AH00139: apr_socket_opt_set(APR_TCP_NODELAY)
Apparently, there appears to be a problem with uwsgi.sock.
My uwsgi.ini file has the following:
uwsgi-socket = /var/lib/mailman3/web/static/uwsgi.sock
Also, my apache.conf for mailman3 has the followig:
ProxyPass /mailman3 unix:/var/lib/mailman3/web/static/uwsgi.sock|uwsgi://localhost
So do you see a problem in my configuration files that would cause the Operation Not Supported error with the uwsgi.sock file.
Thanks,
Gordon
On 2/11/22 13:32, gordon@dickens.com wrote:
Hi Mark,
I think that I found a hint as to the cause of my problems.
I changed the apache2 LogLevel to debug and the following was then generated in /var/log/apache2/ssl_error_log:
[Fri Feb 11 15:40:18.325675 2022] [core:debug] [pid 3982011] core.c(5280): (95)Operation not supported: [remote /var/lib/mailman3/web/static/uwsgi.sock:12150] AH00139: apr_socket_opt_set(APR_TCP_NODELAY)
Apparently, there appears to be a problem with uwsgi.sock.
My uwsgi.ini file has the following:
uwsgi-socket = /var/lib/mailman3/web/static/uwsgi.sock
Also, my apache.conf for mailman3 has the followig:
ProxyPass /mailman3 unix:/var/lib/mailman3/web/static/uwsgi.sock|uwsgi://localhost
So do you see a problem in my configuration files that would cause the Operation Not Supported error with the uwsgi.sock file.
I don't use uWSGI and have no experience with it at all, however here are some things to consider:
For Apache to support uwsgi protocol you may need
a2enmod mod_proxy_uwsgi
However, I would suggest you use http rather than a unix socket.
In your uwsgi.ini file instead of
uwsgi-socket = /var/lib/mailman3/web/static/uwsgi.sock
which requires Apache support for uwsgi protocol, put
http-socket = 0.0.0.0:8000
which will tell uwsgi to listen on TCP port 8000 and in Apache, use
ProxyPass /mailman3 http://127.0.0.1:8000
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Hello Mark and everyone else,
I made significant progress in that I got the web interface working. The file permissions were wrong for the /var/log/mailman3/web directory in that the web directory needs to be writable by www-data (the apache user), Also, the aliases in the Apache configuration needed to be changed to the following:
Alias /favicon.ico /var/lib/mailman3/web/static/postorius/img/favicon.ico Alias /static/ /var/lib/mailman3/web/static/
Once I fixed these issues then I was able to successfully access the web interface and login.
Now, I am able to access the web interface and I am able to navigate to all of the main links on the top of the main page including: Postorius. Lists, Domains, Bans, System Information, Users and Archives.
My only remaining problem is that I cannot access the individual mailing lists. That is, when I navigate a list and click on the list name then my browser displays:
"Server error An error occurred while processing your request."
Concurrent with the above browser display, mailman3 sends an email to the system admin with the subject:
"[Django] ERROR (EXTERNAL IP): Internal Server Error: /mailman3/lists/list_local_name.domain.tld"
This email and my logs now complain about:
"Internal Server Error: /mailman3/lists/list_local_name.domain.tld/ MultipleObjectsReturned at /mailman3/lists/list_local_name.domain.tld/ get() returned more than one EmailAddress -- it returned 3!"
Also, I get the same errors after I create a new mailing list as well.
I hope that Mark and/or somebody here can offer up suggestions on what I need to do to fix this. Any suggestions would be very much appreciated!
Thanks,
Gordon
On 2/18/22 08:43, gordon@dickens.com wrote:
My only remaining problem is that I cannot access the individual mailing lists. That is, when I navigate a list and click on the list name then my browser displays:
"Server error An error occurred while processing your request."
Concurrent with the above browser display, mailman3 sends an email to the system admin with the subject:
"[Django] ERROR (EXTERNAL IP): Internal Server Error: /mailman3/lists/list_local_name.domain.tld"
This email and my logs now complain about:
"Internal Server Error: /mailman3/lists/list_local_name.domain.tld/ MultipleObjectsReturned at /mailman3/lists/list_local_name.domain.tld/ get() returned more than one EmailAddress -- it returned 3!"
Also, I get the same errors after I create a new mailing list as well.
I hope that Mark and/or somebody here can offer up suggestions on what I need to do to fix this. Any suggestions would be very much appreciated!
It seems like a database issue. I would like to help. What is the Traceback from that error?
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Thanks Mark!
Here is the Traceback from the error:
Traceback (most recent call last): File "/opt/mailman/venv/lib/python3.9/site-packages/django/core/handlers/exception.py", line 34, in inner response = get_response(request) File "/opt/mailman/venv/lib/python3.9/site-packages/django/core/handlers/base.py", line 115, in _get_response response = self.process_exception_by_middleware(e, request) File "/opt/mailman/venv/lib/python3.9/site-packages/django/core/handlers/base.py", line 113, in _get_response response = wrapped_callback(request, *callback_args, **callback_kwargs) File "/opt/mailman/venv/lib/python3.9/site-packages/django/views/generic/base.py", line 71, in view return self.dispatch(request, *args, **kwargs) File "/opt/mailman/venv/lib/python3.9/site-packages/postorius/views/generic.py", line 74, in dispatch return super(MailingListView, self).dispatch(request, *args, **kwargs) File "/opt/mailman/venv/lib/python3.9/site-packages/django/views/generic/base.py", line 97, in dispatch return handler(request, *args, **kwargs) File "/opt/mailman/venv/lib/python3.9/site-packages/postorius/views/list.py", line 326, in get primary_email = set_preferred(request.user, mm_user) File "/opt/mailman/venv/lib/python3.9/site-packages/postorius/utils.py", line 64, in set_preferred primary_email = EmailAddress.objects.get_primary(user) File "/opt/mailman/venv/lib/python3.9/site-packages/allauth/account/managers.py", line 30, in get_primary return self.get(user=user, primary=True) File "/opt/mailman/venv/lib/python3.9/site-packages/django/db/models/manager.py", line 82, in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs) File "/opt/mailman/venv/lib/python3.9/site-packages/django/db/models/query.py", line 419, in get raise self.model.MultipleObjectsReturned(
Exception Type: MultipleObjectsReturned at /mailman3/lists/testing.dickens.com/ Exception Value: get() returned more than one EmailAddress -- it returned 3!\
Thanks for your help!
Gordon
The entire error dialog that was emailed to me with the subject; "[Django] ERROR (EXTERNAL IP): Internal Server Error: /mailman3/lists/list_local_name.domain.tld", may be viewed at:
https://dickens.com/documents/mailman3_django_error.txt
FYI,
Gordon
On 2/18/22 11:41, gordon@dickens.com wrote:
The entire error dialog that was emailed to me with the subject; "[Django] ERROR (EXTERNAL IP): Internal Server Error: /mailman3/lists/list_local_name.domain.tld", may be viewed at:
The traceback which you already posted is the only info we need.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 2/18/22 11:28, gordon@dickens.com wrote:
Thanks Mark!
Here is the Traceback from the error:
Traceback (most recent call last): File "/opt/mailman/venv/lib/python3.9/site-packages/django/core/handlers/exception.py", line 34, in inner response = get_response(request) File "/opt/mailman/venv/lib/python3.9/site-packages/django/core/handlers/base.py", line 115, in _get_response response = self.process_exception_by_middleware(e, request) File "/opt/mailman/venv/lib/python3.9/site-packages/django/core/handlers/base.py", line 113, in _get_response response = wrapped_callback(request, *callback_args, **callback_kwargs) File "/opt/mailman/venv/lib/python3.9/site-packages/django/views/generic/base.py", line 71, in view return self.dispatch(request, *args, **kwargs) File "/opt/mailman/venv/lib/python3.9/site-packages/postorius/views/generic.py", line 74, in dispatch return super(MailingListView, self).dispatch(request, *args, **kwargs) File "/opt/mailman/venv/lib/python3.9/site-packages/django/views/generic/base.py", line 97, in dispatch return handler(request, *args, **kwargs) File "/opt/mailman/venv/lib/python3.9/site-packages/postorius/views/list.py", line 326, in get primary_email = set_preferred(request.user, mm_user) File "/opt/mailman/venv/lib/python3.9/site-packages/postorius/utils.py", line 64, in set_preferred primary_email = EmailAddress.objects.get_primary(user) File "/opt/mailman/venv/lib/python3.9/site-packages/allauth/account/managers.py", line 30, in get_primary return self.get(user=user, primary=True) File "/opt/mailman/venv/lib/python3.9/site-packages/django/db/models/manager.py", line 82, in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs) File "/opt/mailman/venv/lib/python3.9/site-packages/django/db/models/query.py", line 419, in get raise self.model.MultipleObjectsReturned(
Exception Type: MultipleObjectsReturned at /mailman3/lists/testing.dickens.com/ Exception Value: get() returned more than one EmailAddress -- it returned 3!\
Thank you for this info.
I've never seen this exact issue. What you have is a Django User with
more than one primary email address. It may be you. Go to the Django web
admin UI at
https://host2.mailhub4u.com/admin/account/emailaddress/?q=gordon where
you should see multiple email addresses for user gordon, three of
which are primary. Go to each address and uncheck primary for all but
one of them.
If it's not you, see the Mult_primary script at https://gitlab.com/mailman/django-mailman3/-/issues/53#note_842641913 to find who, but this script would need to be modified for other than a postgresql database.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Hi Mark,
That appears to have fixed it! Thanks very much!!!
I want to spend more time testing before I declare everything operational but it appears that everything is now working properly. I will let you know in a day or so if I find any other problems. Also, when I am done testing, I will post my configuration files for anyone else that is running Mailman3 with uWSGI, Exim, MySQL (mariadb) and Apache.
Thanks again!
Gordon Dickens
Hi Mark and everyone else,
Everything seems to be working great except none of the lists are being archived although they are configured to be archived. So, I assume that this is a configuration issue with hyperkitty. Here is some additional info:
.pck Files are being accumulated in /var/lib/mailman3/archives/hyperkitty/spool directory. I believe these are the emails waiting to be archived.
The following messages are in my mailman logs. First, in /var/log/mailman3/mailman.log:
Feb 24 16:01:33 2022 (1722225) Traceback (most recent call last): File "/opt/mailman/venv/lib/python3.9/site-packages/mailman_hyperkitty/__init__.py", line 151, in _archive_message url = self._send_message(mlist, msg) File "/opt/mailman/venv/lib/python3.9/site-packages/mailman_hyperkitty/__init__.py", line 207, in _send_message raise ValueError(result.text)
Then, in /var/log/mailman3/web/mailman-web-error.log:
Process Process-1:12: Traceback (most recent call last): File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/opt/mailman/venv/lib/python3.9/site-packages/django_q/cluster.py", line 356, in pusher task = SignedPackage.loads(task[1]) File "/opt/mailman/venv/lib/python3.9/site-packages/django_q/signing.py", line 25, in loads return signing.loads( File "/opt/mailman/venv/lib/python3.9/site-packages/django_q/core_signing.py", line 44, in loads return serializer().loads(data) File "/opt/mailman/venv/lib/python3.9/site-packages/django_q/signing.py", line 39, in loads return pickle.loads(data) AttributeError: Can't get attribute 'unlock_and_call' on <module 'hyperkitty.tasks' from '/opt/mailman/venv/lib/python3.9/site-packages/hyperkitty/tasks.py'> 21:23:57 [Q] ERROR reincarnated pusher Process-1:12 after sudden death 21:23:57 [Q] INFO Process-1:13 pushing tasks at 1734586
I am not confident that my base_url value is correct in mailman-hyperkitty.cfg. I have tried the following without success:
base_url: http://localhost/hyperkitty/ base_url: http://localhost/mailman3/hyperkitty/
Also, I have tried the following as well:
base_url: https://domain.tld/hyperkitty/ base_url: https://domain.tld/mailman3/hyperkitty/
Where domain.tld is my server's public domain name with the following in settings.py:
MAILMAN_ARCHIVER_FROM = ('127.0.0.1', 'IP address for domain.tld', '::1')
However, nothing has yet worked.
I have read that the base_url is supposed to be the external facing url of hyperkitty which I thought would be the public domain name but that hasn't worked either.
Please let me know if anybody has any ideas on how to get my archives working.
Thanks,
Gordon Dickens
Nevermind, I solved it with this:
base_url: http://127.0.0.1:8000/archives/
Now, the archives appear to be working perfectly.
FYI,
Gordon
I also recommend the installation of the virtualenv variant. It works very well, you can keep your existing database. Because the installation is complete separate from the Debian directories, you can keep the Debian installation until the virtualenv installation is completed.
participants (6)
-
David Newman -
eggert＠eehmke.de -
gordon＠dickens.com -
Mark Sapiro -
peter＠chubb.wattle.id.au -
Stephen J. Turnbull