Lessons learned and a couple little gotcha for newbies
I just finished, with great help from Mark Sapiro on the list, a new installation of mailman3.
I used the instructions at https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/docs/instal...
and
https://docs.mailman3.org/en/latest/install/virtualenv.html
There were a couple of gotchas that I didn't see well documented.
- I got the error
django.template.library.InvalidTemplateLibrary: Invalid template library specified. ImportError raised when trying to load 'hyperkitty.templatetags.decorate': cannot import name 'escape_html' from 'mistune.scanner' (/opt/mailman/venv/lib/python3.9/site- packages/mistune/scanner.py)
when I ran 'mailman-web migrate'
The fix was trivial: I just added the lines:
from mistune.util import escape_html from mistune.util import escape_url
to /opt/mailman/venv/lib/python3.9/site-packages/mistune/scanner.py
It's not hard to search for those definitions, but if someone doesn't know a smattering of python, it might not be obvious.
- Similarly, migrate (using postgres) also gave the error:
File "/opt/mailman/venv/lib/python3.9/site- packages/django/db/backends/postgresql/utils.py", line 6, in utc_tzinfo_factory raise AssertionError("database connection isn't set to UTC")
I was able to find a discusson of this one at
https://stackoverflow.com/questions/68024060/assertionerror-database-connect...
which had me downgrade to an older version of psycopg2-binary:
pip install psycopg2-binary==2.8.6
There might be a different fix, but this worked.
- When I ran the test "uwsgi --ini /etc/mailman3/uwsgi.ini" I got the error:
ModuleNotFoundError: No module named 'settings'
This was also trivially fixed by adding the path:
export PYTHONPATH=$PYTHONPATH:/etc/mailman3
in /opt/mailman/.bashrc
- Sometimes when I rebooted the machine, when I ran 'systemctl start mailman3' I'd get the error that mailman wouldn't start because the lock was stll there. I never could figure out where the lock was, so I worked around it by replacing
ExecStart=/opt/mailman/venv/bin/mailman start
with ExecStart=/opt/mailman/venv/bin/mailman start --force
in /etc/systemd/system/mailman3.service
I'm sure there's something bad about doing that, but it works for me, so far.
- I had a problem with mail to outside people not being delivered while mail to the local host was delivered. It turned out that this was because in my mailman.cfg I had "smtp_host:my.listserver.tld when it should have been localhost. The error was "relay access denied." To fix this, I had to add my.listserver.tld to "mynetworks" in /etc/postfix/main.cf, ie change:
mynetworks = 127.0.0.0/8 10.0.0.0/24
to
mynetworks = 127.0.0.0/8 10.0.0.0/24 111.222.333.444 <--- my ip address
However, Mark Sapiro (a God) told me this was because I had changed a line in the [mta] section of mailman.cfg. I had changed
smtp_host: localhost
to
smtp_host: mydomain.tld
Here's the thing. The reason I did this was because the instructions at https://docs.mailman3.org/en/latest/config-core.html
give this as their model:
[mta] incoming: mailman.mta.postfix.LMTP outgoing: mailman.mta.deliver.deliver lmtp_host: mail.example.com lmtp_port: 8024 smtp_host: mail.example.com <---- not localhost smtp_port: 25
The instructions are good, but there are a few places where one size does not fit all, I guess.
- A second error in delivery to outside addresses resulted in an "Access denied" error. A search on the internet revealed that this had to do with (for me) smtpd_recipient_restrictions in postfix. See, for instance: https://know.mailsbestfriend.com/postfix_error_relay_access_denied_noqueue_r...
Changing this fixed things.
I'm not sure, but I think this shows up mostly with installations that include dovecot.
- If you are using virtual domains in postfix, you need to use the "unusual postfix configuration" in https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/docs/mta.ht...
virtual_alias_maps = hash:/path-to-mailman/var/data/postfix_vmap
You need to manually set the local virtual host for mailman3. Somehow, I got the impression from the installation instructions that this was done automagically to the default x.mydomain.tld if your domain is mydomain.tld. It doesn't.
Follow the instructions verbatim, and follow them in the order provided. I installed postfix and nginx out of order, and it did OK, but once you get into the python virtual environment, it is easy to miss something if you don't do it in order.
Jot down lots of notes. This is one of the more complicated installations I've done, mostly because it involved configuring a zillion packages to talk to each other: Postfix, nginx, postorius, hyperkitty, django, and others (I'd never heard of "whoosh" or "sass" for instance). All of these can be unforgiving with configuration errors.
The default installation instructions assumed a number of packages I'd never used. I'm an apache/dovecot/mysql kind of person, and I had to switch to nginx/postgres to use the instructions as the written defaults. I'm sure I could have used apache and mysql, but it would mean trying to figure out what didn't work without following the installations verbatim. I'd never even heard of postorius, hyperkitty, or django.
So, it was easy to get confused a bit. Taking notes of changes I made was important in being able to back out of disastrous choices. Plus, once I'd installed it on one machine, I sat down and installed it on another machine, and it was very useful to have written down how I fixed an issue when it came up again -- and I didn't remember exactly what I had done.
- Make multiple sequential backups of configuration files. I went down a couple of paths I shouldn't have gone down. For instance, with the vmap issue, I got it in my head that it was a postfix configuration issue, not a mailman issue. So I kept changing things in postfix. Once I was told that it was in *mailman* I had to set the virtual host, I needed to back out of about 8 tweaks I'd made to main.cf in /etc/postfix. If I hadn't been making main.cf.backup.01, main.cf.backup.02, etc. I would have had to spend another few hours trying to remember what inappropriate changes I had made.
This is particularly true for me because I am not a great typist, and when I try to retype something to back out, there's a reasonable chance I'll mistype something and it still won't work. With these backups, I don't have both manually back out of things *and* proof my typing again and again. It's a lot easier to delete backups once things are working than to try to remember what you did late at night after a couple of bourbons.
- Choose one set of installation instructions and follow it. Don't try to mix instructions from different tutorials unless you really know what you are doing. I didn't. For instance, there's a different set of mailman3 installation instructions at:
https://wiki.list.org/DOC/Howto_Install_Mailman3_On_Debian10
It does things a little differently. It's useful to look at that when you are debugging why something's not working, but you can get into some trouble trying to do both installations at the same time -- which I found myself doing when something wasn't working. It didn't help.
billo
On 12/28/21 11:56 AM, William Oliver wrote:
- I got the error
django.template.library.InvalidTemplateLibrary: Invalid template library specified. ImportError raised when trying to load 'hyperkitty.templatetags.decorate': cannot import name 'escape_html' from 'mistune.scanner' (/opt/mailman/venv/lib/python3.9/site- packages/mistune/scanner.py)
when I ran 'mailman-web migrate'
The fix was trivial: I just added the lines:
from mistune.util import escape_html from mistune.util import escape_url to /opt/mailman/venv/lib/python3.9/site-packages/mistune/scanner.py
This is fixed at https://gitlab.com/mailman/hyperkitty/-/merge_requests/379. Unfortunately, that fix is not in the latest release version. We will need to address that as mistune 2.0.0 moved those functions from their prior location in mistune 2.0.0rc1
- Similarly, migrate (using postgres) also gave the error:
File "/opt/mailman/venv/lib/python3.9/site- packages/django/db/backends/postgresql/utils.py", line 6, in utc_tzinfo_factory raise AssertionError("database connection isn't set to UTC")
I was able to find a discusson of this one at
https://stackoverflow.com/questions/68024060/assertionerror-database-connect...
which had me downgrade to an older version of psycopg2-binary:
pip install psycopg2-binary==2.8.6
There might be a different fix, but this worked.
This should be fixed in psycopg2-binary, but it isn't. The work around is what you did and I'll update the doc.
- When I ran the test "uwsgi --ini /etc/mailman3/uwsgi.ini" I got the error:
ModuleNotFoundError: No module named 'settings'
This was also trivially fixed by adding the path:
export PYTHONPATH=$PYTHONPATH:/etc/mailman3
in /opt/mailman/.bashrc
The recommended uwsgi.ini includes
pythonpath = /etc/mailman3/
according to https://uwsgi-docs.readthedocs.io/en/latest/Options.html?highlight=pythonpat... this should work. Did you have that? If you did and it doesn't work, perhaps adding
env = PYTHONPATH=/etc/mailman3
would work.
- Sometimes when I rebooted the machine, when I ran 'systemctl start mailman3' I'd get the error that mailman wouldn't start because the lock was stll there. I never could figure out where the lock was, so I worked around it by replacing
ExecStart=/opt/mailman/venv/bin/mailman start
with ExecStart=/opt/mailman/venv/bin/mailman start --force
in /etc/systemd/system/mailman3.service
I'm sure there's something bad about doing that, but it works for me, so far.
The lock is /opt/mailman/mm/var/locks/master.lck*
The problem with using --force unconditionally is you can start a second instance when mailman core is already running and that causes other issues.
- I had a problem with mail to outside people not being delivered while mail to the local host was delivered. It turned out that this was because in my mailman.cfg I had "smtp_host:my.listserver.tld when it should have been localhost. The error was "relay access denied." To fix this, I had to add my.listserver.tld to "mynetworks" in /etc/postfix/main.cf, ie change:
mynetworks = 127.0.0.0/8 10.0.0.0/24
to
mynetworks = 127.0.0.0/8 10.0.0.0/24 111.222.333.444 <--- my ip address
However, Mark Sapiro (a God) told me this was because I had changed a line in the [mta] section of mailman.cfg. I had changed
smtp_host: localhost
to
smtp_host: mydomain.tld
Here's the thing. The reason I did this was because the instructions at https://docs.mailman3.org/en/latest/config-core.html
give this as their model:
[mta] incoming: mailman.mta.postfix.LMTP outgoing: mailman.mta.deliver.deliver lmtp_host: mail.example.com lmtp_port: 8024 smtp_host: mail.example.com <---- not localhost smtp_port: 25
I'll add some clarification to the doc.
The instructions are good, but there are a few places where one size does not fit all, I guess.
- A second error in delivery to outside addresses resulted in an "Access denied" error. A search on the internet revealed that this had to do with (for me) smtpd_recipient_restrictions in postfix. See, for instance: https://know.mailsbestfriend.com/postfix_error_relay_access_denied_noqueue_r...
Changing this fixed things.
I'm not sure, but I think this shows up mostly with installations that include dovecot.
This is a Postfix configuration issue.
<big snip>
Thanks for your feedback.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On Tue, 2021-12-28 at 17:50 -0800, Mark Sapiro wrote:
The recommended uwsgi.ini includes
pythonpath = /etc/mailman3/
according to https://uwsgi-docs.readthedocs.io/en/latest/Options.html?highlight=pythonpat... this should work. Did you have that? If you did and it doesn't work, perhaps adding
env = PYTHONPATH=/etc/mailman3
Yes, I had the first, and does not work. The second entry works.
This is on debian 11/nginx/postgres/postfix system.
Thanks,
billo
On 12/28/21 5:58 PM, William Oliver wrote:
On Tue, 2021-12-28 at 17:50 -0800, Mark Sapiro wrote:
The recommended uwsgi.ini includes
pythonpath = /etc/mailman3/
according to https://uwsgi-docs.readthedocs.io/en/latest/Options.html?highlight=pythonpat...
this should work. Did you have that? If you did and it doesn't work, perhaps adding
env = PYTHONPATH=/etc/mailman3
Yes, I had the first, and does not work. The second entry works.
I updated the doc to use
env = PYTHONPATH=/etc/mailman3
I don't know enough about uwsgi to know for sure, but the issue may be that setting pythonpath in a running python process doesn't work - it's too late as the path is already set in python's initialization.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On Tue, 2021-12-28 at 20:05 -0800, Mark Sapiro wrote:
On 12/28/21 5:58 PM, William Oliver wrote:
On Tue, 2021-12-28 at 17:50 -0800, Mark Sapiro wrote:
The recommended uwsgi.ini includes
pythonpath = /etc/mailman3/
according to https://uwsgi-docs.readthedocs.io/en/latest/Options.html?highlight=pythonpat... this should work. Did you have that? If you did and it doesn't work, perhaps adding
env = PYTHONPATH=/etc/mailman3
Yes, I had the first, and does not work. The second entry works.
I updated the doc to use
env = PYTHONPATH=/etc/mailman3
I don't know enough about uwsgi to know for sure, but the issue may be that setting pythonpath in a running python process doesn't work - it's too late as the path is already set in python's initialization.
You might add that it may be necessary to do it in .bashrc for linux folk -- I got it to work but as I've demonstrated, I can sometimes make mistakes and leave detritus around. I may not have successfully cleared my previous setting. That being said, I think it works.
I am, I am afraid not a details-oriented guy, which is a bit of a drawback for someone doing system admin...
hi William, please clarify what you mean by this step 8. You need to manually set the local virtual host for mailman3 Thank you
On 4/13/22 05:23, Paul Ssekamatte via Mailman-users wrote:
hi William, please clarify what you mean by this step 8. You need to manually set the local virtual host for mailman3 Thank you
He means that if you are going to use the alias domain feature discussed
at
https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/docs/mta.ht...
you need to actually set the value you choose for the alias_domain in
the Postorius domains view or via REST or mailman shell
.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
participants (3)
-
Mark Sapiro
-
Paul Ssekamatte
-
William Oliver