On 1/6/22 3:25 PM, gordon@dickens.com wrote:

How do I uninstall the Debian version and then reinstall with the venv and web instructions so that all of my mailing lists and configuration data are preserved? I may elect to go that route if I can preserve all of my mailing list and data and you think that would be a quicker route to a solution. Is there a simple way to do this?

It's probably not "simple", but here's how I would do it.

First, follow https://docs.mailman3.org/en/latest/install/virtualenv.html to install Mailman in a virtualenv. You can use list as the Mailman user rather than mailman if you like.

The doc uses Postgreql as the database manager, but you want to use whatever the current DBM is because you will want to use the existing database which is where all the list configuration, membership and archives are.

One gotcha here is the instructions tell you to create /etc/mailman3/mailman.cfg and the Debian package already puts its mailman.cfg there. The Debian mailman.cfg may actually be appropriate although it puts things in different places, e.g. /var/lib/mailman3 instead of /opt/mailman/mm/var.

The Debian package puts its Django settings in /etc/mailman3/mailman-web.py and the virtualenv doc puts them in /etc/mailman3/settings.py. You need to reconcile the differences keeping in mind that the recommended settings.py imports most settings from the mailman-web project and only contains things you want to change locally.

The basic idea is you want the virtualenv install to use the existing databases and /var/lib/mailman3 stuff so you preserve all your existing list stuff.

It probably involves several gotchas that might be difficult to resolve if you are less than intimate with the details of a Mailman installation.

I actually spun up a small Ubuntu 20.04 droplet and installed the mailman3-full package to see what things looked like, but I destroyed it without examining the contents of some of the files or I might know more.

-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

gordon@dickens.com writes:

How do I uninstall the Debian version and then reinstall with the venv and web instructions so that all of my mailing lists and configuration data are preserved?

tar czf /tmp/mailmandata.tar.gz $PATH_TO_MAILMAN_DATA

Not sure where Debian puts it but I would guess /var/lib/mailman. You can also try apt-get --dry-run remove ... but I think that provides output at too coarse a level (packages, not files). Generally apt is pretty good about not removing files that are not in the package, though.

I may elect to go that route if I can preserve all of my mailing list and data and you think that would be a quicker route to a solution. Is there a simple way to do this?

Then you can do apt-get remove $MAILMAN_PACKAGES with confidence. Do not do apt-get purge ..., of course (although I suspect even that will leave your archives and mailing lists).

We still generally recommend installing from source, or possibly using the docker containers that Abhilash provides (usually quite promptly after a release). First, you're not dependent on a distro for new features, and Mailman 3 is still by no means feature-complete. (On the other hand, the distros are good about providing security upgrades, and often the easiest route for them is a version bump). Second, if you put stuff where we tell you to, we know where it is when you ask for advice. I'm a Debian user, but I don't use their packages for Mailman itself.

Steve

I agree that this is a Debian packaging issue and I agree with Mark that the underlying issue is the "sed:" error message. I have only had one reply to my Debian bug report. That reply told me what I had to do make apt-get usable again (since it was stuck on the mailman3 error) but that post did not address the "sed:" bug itself and nobody else has responded to the bug report. So, it looks like I am on my own as far as my Debian mailman3 installation is concerned.

So, here is what I plan to do:

1. apt-get remove mailman3 mailman3-web mailman3-full. This should not remove any configuration data or database files but I will run "apt-get --dry-run remove" first just to make sure.
2. Install mailman3 using the venv and web instructions as suggested in this thread. I plan to use mysql (mariadb) since my database was originally implemented with mysql.
3. I will poach as much of the configuration data as I can find from my original Debian mailman3 installation.

Thanks to everyone for your support and replys!

Gordon

Thanks to David and everybody else that have replied to my problem. I have proceeded to uninstall the debian mailman3 packages and then endeavored to install everything with venv. I have implemented all of David's configuration suggestions which were very helpful. Now, mailman3 (core) appears to be working properly in that I can start/stop/restart/status the service. Also, the command "mailman info" from the venv prompt yields output which appears to be correct and the same as in the venv instructions. However, I have not been able to get mailman3-web working.

All of my configuration files may be viewed here:

https://dickens.com/etc-mailman3/

The above files are basically a mirror of the contents of my /etc/mailman3 directory except that I have obfuscated sensitive info such as keys, passwords and email addresses.

Executing the mailman3-web service script with the command "systemctl start mailman3-web" starts everything without errors, however, the command "systemctl status mailman3-web" then generates the following output with errors:

● mailman3-web.service - GNU Mailman Web UI Loaded: loaded (/etc/mailman3/mailman3-web.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Tue 2022-01-25 17:10:30 EST; 6s ago Process: 1718478 ExecStart=/opt/mailman/venv/bin/uwsgi --ini /etc/mailman3/uwsgi.ini (code=exited, status=1/FAILURE) Main PID: 1718478 (code=exited, status=1/FAILURE) CPU: 12ms

Jan 25 17:10:29 slc.mailhub4u.com systemd[1]: Started GNU Mailman Web UI. Jan 25 17:10:30 slc.mailhub4u.com uwsgi[1718478]: [uWSGI] getting INI configuration from /etc/mailman3/uwsgi.ini Jan 25 17:10:30 slc.mailhub4u.com systemd[1]: mailman3-web.service: Main process exited, code=exited, status=1/FAILURE Jan 25 17:10:30 slc.mailhub4u.com systemd[1]: mailman3-web.service: Failed with result 'exit-code'.

Also, /var/log/syslog and "journalctl | grep mailman3-web" contain these two lines of errors:

Jan 25 14:00:46 slc systemd[1]: mailman3-web.service: Main process exited, code=exited, status=1/FAILURE Jan 25 14:00:46 slc systemd[1]: mailman3-web.service: Failed with result 'exit-code'.

Also, other commands in the venv instructions do not work such as "mailman-web migrate" which produces volumes of error dialog starting with:

Traceback (most recent call last): File "/opt/mailman/venv/lib/python3.9/site-packages/django/template/utils.py", line 66, in __getitem__ return self._engines[alias] KeyError: 'django'

Then these two lines are then followed by over 50 lines of django and python error messages.

I have not been able to find any hints or suggestions as to what I am doing wrong in my server's logs. FWIW, I have been working in the Unix/Linux command line environment for over 35 years and I have extensive LAMP, EXIM and Mailman2 experience. However, I have not ever worked with Django. I am usually able to resolve issues like this quickly, however, I am at a total loss as to what to do now with this mailman3 installation. I suspect that my unfamiliarity with Django may be my achilles heel.

So, I am hoping that somebody here will soon be able to offer suggestions that might help me get everything working properly. I have several organization's mailings lists on this server and I need to get this fixed ASAP especially since my mailman installation has been down now for almost 3 weeks. As you might imagine, this is causing me alot of grief with my mailman3 users. So, please let me know any suggestions that you folks may have to get my mailman3 installation working again.

Many Thanks,

Gordon

David Newman wrote:

One general comment: Debian packages install files with a different owner than the pip install. You want to be sure all your directories and files are owned by the Mailman user ('mailman' on Debian).

Yes, I have checked the ownerships. The debian package installed everything in mailman3 as user "list" whereas I have now setup everything to run as user "mailman". As I previously mentioned, mailman3 (core) is running fine. In fact mailman3 has properly posted numerous messages to several of my mailing lists since I finished setting it up with pip. However, mailman3-web (or mailman-web) is not working, I think because of a permissions issue. The installation instructions specify that mailman3-web (or mailman-web) run as user=mailman and group=mailman. However, the debian installation specifies the data files as user=www-data and group=www-data which is what apache runs as. For instance, I have my static files stored in the /var/lib/mailman3/web/static directory and the previous Debian mailman3 version set all the ownerships and groups to www-data for /var/lib/mailman3/web, /var/lib/mailman3/web/static along with all of the directories and files in the static subdirectory. So, when I run "mailman-web collectstatic" it bombs on the following permissions error:

PermissionError: [Errno 13] Permission denied: '/var/lib/mailman3/web/static/hyperkitty/js/hyperkitty-common.js'

So, my question is as follows. Should I continue to run mailman3-web as user mailman and change the ownership of the /var/lib/mailman3/web directory to mailman or should I leave the permissions of the /var/lib/mailman3/web directory alone and have mailman3-web run as user and group www-data?

Do you have anything further in any of the logs in /opt/mailman/web/logs ?

Yes, everytime that I try to start mailman3-web it posts this to the log and which includes a permission error as the last line:

*** Starting uWSGI 2.0.20 (64bit) on [Wed Jan 26 17:00:13 2022] *** compiled with version: 10.2.1 20210110 on 24 January 2022 19:51:51 os: Linux-5.10.0-10-amd64 #1 SMP Debian 5.10.84-1 (2021-12-08) nodename: slc.mailhub4u.com machine: x86_64 clock source: unix pcre jit disabled detected number of CPU cores: 8 current working directory: / detected binary path: /opt/mailman/venv/bin/uwsgi chdir() to /usr/share/mailman3-web your processes number limit is 31530 your memory page size is 4096 bytes detected max file descriptor number: 1024 lock engine: pthread robust mutexes thunder lock: disabled (you can enable it with --thunder-lock) bind(): Permission denied [core/socket.c line 230]

This one might be due to a bug I hit as well. There's a problem with mistune, one of the components. If that's the issue here, this should address it, run as the mailman user: pip install mistune==2.0.0rc1 mailman-web migrate mailman-web compress mailman-web collectstatic mailman-web compilemessages

Thanks for that tip David! Reverting mistune back to version 2.0.0rc1 allowed "mailman-web migrate" to work. I think that the balance of my problems are permissions related to running mailman3-web (or mailman-web) as user mailman versus www-data. So, please let me know, based on your installation, if I should change the permissions for /var/lib/mailman3/web/ to user/group mailman or should I run mailman3-web as user/group www-data.

Also, any other advice from David or anyone else would be very much appreciated.

Thanks!

Gordon

On 1/26/22 14:34, gordon@dickens.com wrote:

However, mailman3-web (or mailman-web) is not working, I think because of a permissions issue. The installation instructions specify that mailman3-web (or mailman-web) run as user=mailman and group=mailman. However, the debian installation specifies the data files as user=www-data and group=www-data which is what apache runs as. For instance, I have my static files stored in the /var/lib/mailman3/web/static directory and the previous Debian mailman3 version set all the ownerships and groups to www-data for /var/lib/mailman3/web, /var/lib/mailman3/web/static along with all of the directories and files in the static subdirectory. So, when I run "mailman-web collectstatic" it bombs on the following permissions error:

PermissionError: [Errno 13] Permission denied: '/var/lib/mailman3/web/static/hyperkitty/js/hyperkitty-common.js'

So, my question is as follows. Should I continue to run mailman3-web as user mailman and change the ownership of the /var/lib/mailman3/web directory to mailman or should I leave the permissions of the /var/lib/mailman3/web directory alone and have mailman3-web run as user and group www-data?

As David later said, owner and group of all Mailman's files and directories should be mailman and what ever the wsgi interface is (uwsgi, gunicorn, mod_wsgi) should run the wsgi process as user:group mailman:mailman

-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

On 2/4/22 11:15, gordon@dickens.com wrote:

173.165.166.34 - root [04/Feb/2022:12:54:06 -0500] "GET /mailman3/postorius/lists/members.georgiajets.org/ HTTP/1.1" 404 282 173.165.166.34 - root [04/Feb/2022:12:54:06 -0500] "GET /favicon.ico HTTP/1.1" 200 318

So, I get File Not Found error 404 for the web page but the favicon.ico file appears to be found successfully. Therefore, I suspect that my apache configuration file for mailman3 may be at the source of my problems and specifically the alias for the static directory. Here is my apache configuration file:

It's not the alias for static

Alias /mailman3/favicon.ico /var/lib/mailman3/web/static/postorius/img/favicon.ico Alias /mailman3/static /var/lib/mailman3/web/static

<Directory "/var/lib/mailman3/web/static"> Require all granted </Directory>

<IfModule mod_proxy_uwsgi.c> ProxyPass /mailman3/favicon.ico ! ProxyPass /mailman3/static ! # ProxyPass /mailman3 unix:/run/mailman3-web/uwsgi.sock|uwsgi://localhost </IfModule>

It's the fact that

ProxyPass /mailman3 unix:/run/mailman3-web/uwsgi.sock|uwsgi://localhost

is commented out. This should not be commented assuming you are using uWSGI and it is listening on the socket /run/mailman3-web/uwsgi.sock

-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

On 2/4/22 14:54, gordon@dickens.com wrote:

Hi Mark,

Thanks very much for your prompt and helpful reply!

I made the change to the mailman3 configuration for Apache that you recommended since I am using uwsgi. Now, I am able to start/status/stop the mailman3-web service without errors. However, now, when I then attempt to access the mailman installation on my server with my browser, I get the following message in the browser window:

Internal Server Error

I am unable to find any errors in the mailman3 logs or in the Apache error logs. However the Apache access logs have the following:

173.165.166.34 - - [04/Feb/2022:17:06:20 -0500] "GET /mailman3 HTTP/1.1" 500 21

Do you have any idea as to what is going on now? Any suggestions for troubleshooting this?

Without more log info, I don't know what the issue is. Have you looked in /var/log/mailman3/mailman.log and /var/log/mailman3/web/mailman-web.log?

Possibly there is a uwsgi log somewhere in /var/log that might have something relevant.

-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

On 2/11/22 13:32, gordon@dickens.com wrote:

Hi Mark,

I think that I found a hint as to the cause of my problems.

I changed the apache2 LogLevel to debug and the following was then generated in /var/log/apache2/ssl_error_log:

[Fri Feb 11 15:40:18.325675 2022] [core:debug] [pid 3982011] core.c(5280): (95)Operation not supported: [remote /var/lib/mailman3/web/static/uwsgi.sock:12150] AH00139: apr_socket_opt_set(APR_TCP_NODELAY)

Apparently, there appears to be a problem with uwsgi.sock.

My uwsgi.ini file has the following:

uwsgi-socket = /var/lib/mailman3/web/static/uwsgi.sock

Also, my apache.conf for mailman3 has the followig:

ProxyPass /mailman3 unix:/var/lib/mailman3/web/static/uwsgi.sock|uwsgi://localhost

So do you see a problem in my configuration files that would cause the Operation Not Supported error with the uwsgi.sock file.

I don't use uWSGI and have no experience with it at all, however here are some things to consider:

For Apache to support uwsgi protocol you may need

a2enmod mod_proxy_uwsgi

However, I would suggest you use http rather than a unix socket.

In your uwsgi.ini file instead of

uwsgi-socket = /var/lib/mailman3/web/static/uwsgi.sock

which requires Apache support for uwsgi protocol, put

http-socket = 0.0.0.0:8000

which will tell uwsgi to listen on TCP port 8000 and in Apache, use

ProxyPass /mailman3 http://127.0.0.1:8000

-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

On 2/18/22 08:43, gordon@dickens.com wrote:

My only remaining problem is that I cannot access the individual mailing lists. That is, when I navigate a list and click on the list name then my browser displays:

"Server error An error occurred while processing your request."

Concurrent with the above browser display, mailman3 sends an email to the system admin with the subject:

"[Django] ERROR (EXTERNAL IP): Internal Server Error: /mailman3/lists/list_local_name.domain.tld"

This email and my logs now complain about:

"Internal Server Error: /mailman3/lists/list_local_name.domain.tld/ MultipleObjectsReturned at /mailman3/lists/list_local_name.domain.tld/ get() returned more than one EmailAddress -- it returned 3!"

Also, I get the same errors after I create a new mailing list as well.

I hope that Mark and/or somebody here can offer up suggestions on what I need to do to fix this. Any suggestions would be very much appreciated!

It seems like a database issue. I would like to help. What is the Traceback from that error?

-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

The entire error dialog that was emailed to me with the subject; "[Django] ERROR (EXTERNAL IP): Internal Server Error: /mailman3/lists/list_local_name.domain.tld", may be viewed at:

https://dickens.com/documents/mailman3_django_error.txt

FYI,

Gordon

On 2/18/22 11:28, gordon@dickens.com wrote:

Thanks Mark!

Here is the Traceback from the error:

Traceback (most recent call last): File "/opt/mailman/venv/lib/python3.9/site-packages/django/core/handlers/exception.py", line 34, in inner response = get_response(request) File "/opt/mailman/venv/lib/python3.9/site-packages/django/core/handlers/base.py", line 115, in _get_response response = self.process_exception_by_middleware(e, request) File "/opt/mailman/venv/lib/python3.9/site-packages/django/core/handlers/base.py", line 113, in _get_response response = wrapped_callback(request, *callback_args, **callback_kwargs) File "/opt/mailman/venv/lib/python3.9/site-packages/django/views/generic/base.py", line 71, in view return self.dispatch(request, *args, **kwargs) File "/opt/mailman/venv/lib/python3.9/site-packages/postorius/views/generic.py", line 74, in dispatch return super(MailingListView, self).dispatch(request, *args, **kwargs) File "/opt/mailman/venv/lib/python3.9/site-packages/django/views/generic/base.py", line 97, in dispatch return handler(request, *args, **kwargs) File "/opt/mailman/venv/lib/python3.9/site-packages/postorius/views/list.py", line 326, in get primary_email = set_preferred(request.user, mm_user) File "/opt/mailman/venv/lib/python3.9/site-packages/postorius/utils.py", line 64, in set_preferred primary_email = EmailAddress.objects.get_primary(user) File "/opt/mailman/venv/lib/python3.9/site-packages/allauth/account/managers.py", line 30, in get_primary return self.get(user=user, primary=True) File "/opt/mailman/venv/lib/python3.9/site-packages/django/db/models/manager.py", line 82, in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs) File "/opt/mailman/venv/lib/python3.9/site-packages/django/db/models/query.py", line 419, in get raise self.model.MultipleObjectsReturned(

Exception Type: MultipleObjectsReturned at /mailman3/lists/testing.dickens.com/ Exception Value: get() returned more than one EmailAddress -- it returned 3!\

Thank you for this info.

I've never seen this exact issue. What you have is a Django User with more than one primary email address. It may be you. Go to the Django web admin UI at https://host2.mailhub4u.com/admin/account/emailaddress/?q=gordon where you should see multiple email addresses for user gordon, three of which are primary. Go to each address and uncheck primary for all but one of them.

If it's not you, see the Mult_primary script at https://gitlab.com/mailman/django-mailman3/-/issues/53#note_842641913 to find who, but this script would need to be modified for other than a postgresql database.

-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

Hi Mark and everyone else,

Everything seems to be working great except none of the lists are being archived although they are configured to be archived. So, I assume that this is a configuration issue with hyperkitty. Here is some additional info:

.pck Files are being accumulated in /var/lib/mailman3/archives/hyperkitty/spool directory. I believe these are the emails waiting to be archived.

The following messages are in my mailman logs. First, in /var/log/mailman3/mailman.log:

Feb 24 16:01:33 2022 (1722225) Traceback (most recent call last): File "/opt/mailman/venv/lib/python3.9/site-packages/mailman_hyperkitty/__init__.py", line 151, in _archive_message url = self._send_message(mlist, msg) File "/opt/mailman/venv/lib/python3.9/site-packages/mailman_hyperkitty/__init__.py", line 207, in _send_message raise ValueError(result.text)

Then, in /var/log/mailman3/web/mailman-web-error.log:

Process Process-1:12: Traceback (most recent call last): File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/opt/mailman/venv/lib/python3.9/site-packages/django_q/cluster.py", line 356, in pusher task = SignedPackage.loads(task[1]) File "/opt/mailman/venv/lib/python3.9/site-packages/django_q/signing.py", line 25, in loads return signing.loads( File "/opt/mailman/venv/lib/python3.9/site-packages/django_q/core_signing.py", line 44, in loads return serializer().loads(data) File "/opt/mailman/venv/lib/python3.9/site-packages/django_q/signing.py", line 39, in loads return pickle.loads(data) AttributeError: Can't get attribute 'unlock_and_call' on <module 'hyperkitty.tasks' from '/opt/mailman/venv/lib/python3.9/site-packages/hyperkitty/tasks.py'> 21:23:57 [Q] ERROR reincarnated pusher Process-1:12 after sudden death 21:23:57 [Q] INFO Process-1:13 pushing tasks at 1734586

I am not confident that my base_url value is correct in mailman-hyperkitty.cfg. I have tried the following without success:

base_url: http://localhost/hyperkitty/ base_url: http://localhost/mailman3/hyperkitty/

Also, I have tried the following as well:

base_url: https://domain.tld/hyperkitty/ base_url: https://domain.tld/mailman3/hyperkitty/

Where domain.tld is my server's public domain name with the following in settings.py:

MAILMAN_ARCHIVER_FROM = ('127.0.0.1', 'IP address for domain.tld', '::1')

However, nothing has yet worked.

I have read that the base_url is supposed to be the external facing url of hyperkitty which I thought would be the public domain name but that hasn't worked either.

Please let me know if anybody has any ideas on how to get my archives working.

Thanks,

Gordon Dickens