Hi All
I'm getting errors when a custom "Welcome message" template is in place [list:user:notice:welcome].
Case: Subscribing a new user via the admin "Mass subscribe".
Although the new user is successfully subscribed, a welcome message is not sent and the following error message displays on the webpage.
=== HTTP Error 400: HTTPConnectionPool(host='localhost', port=8000): Read timed out. (read timeout=5)
And the site-owner receives an email with ...
==== Internal Server Error: /mailman3/lists/ DoesNotExist at /mailman3/lists/ Site matching query does not exist. Request Method: GET Request URL: http://<REDACTED-PUBLIC-IPv4-ADDRESS>/mailman3/lists/ Django Version: 4.2.11 Python Executable: /opt/mailman/venv/bin/python3 Python Version: 3.11.2 ... etc ...
Regarding the second line in the email - my thinking is that it should be looking in "/opt/mailman/mm/var/lists" and not "/mailman3/lists/".
If that's right, where would I look to change that directive? If not, any other clues would be greatly appreciated.
The error doesn't occur if the is no custom "Welcome message" template - ie. the user is sent the default welcome message.
$ mailman info GNU Mailman 3.3.9 (Tom Sawyer) Python 3.11.2 (main, Mar 13 2023, 12:18:29) [GCC 12.2.0] config file: /etc/mailman3/mailman.cfg REST root url: http://localhost:8001/3.1/
On 4/21/24 19:15, Mark wrote:
And the site-owner receives an email with ...
==== Internal Server Error: /mailman3/lists/ DoesNotExist at /mailman3/lists/ Site matching query does not exist. Request Method: GET Request URL: http://<REDACTED-PUBLIC-IPv4-ADDRESS>/mailman3/lists/
The above means you have to add <REDACTED-PUBLIC-IPv4-ADDRESS> to the list of ALLOWED_HOSTS in your Django settings, but see below.
Django Version: 4.2.11 Python Executable: /opt/mailman/venv/bin/python3 Python Version: 3.11.2 ... etc ...
Regarding the second line in the email - my thinking is that it should be looking in "/opt/mailman/mm/var/lists" and not "/mailman3/lists/".
It's talking about the path in the URL, not a path in the file system.
The error doesn't occur if the is no custom "Welcome message" template - ie. the user is sent the default welcome message.
I think you must be setting the custom template via Postorius and your Django setting for POSTORIUS_TEMPLATE_BASE_URL is wrong. It should be set to the scheme and host you use to access Postorius. I.e, if you normally access Postorius via something like https://example.com/mailman3 set
POSTORIUS_TEMPLATE_BASE_URL = 'https://example.com'
It is probably now set as
POSTORIUS_TEMPLATE_BASE_URL = 'http://<REDACTED-PUBLIC-IPv4-ADDRESS>'
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 2024-04-22 14:07, Mark Sapiro wrote:
I think you must be setting the custom template via Postorius and your Django setting for POSTORIUS_TEMPLATE_BASE_URL is wrong. It should be set to the scheme and host you use to access Postorius. I.e, if you normally access Postorius via something like https://example.com/mailman3 set
POSTORIUS_TEMPLATE_BASE_URL = 'https://example.com'
It is probably now set as
POSTORIUS_TEMPLATE_BASE_URL = 'http://<REDACTED-PUBLIC-IPv4-ADDRESS>'
================================================================================
There was no line POSTORIUS_TEMPLATE_BASE_URL in /etc/mailman3/settings.py
So I've now added the line: POSTORIUS_TEMPLATE_BASE_URL = 'https://lists.mydomain.com'
Checked in settings.py that:
- "lists.mydomain.com" is in ALLOWED_HOSTS; and
- "https://lists.mydomain.com" is in CSRF_TRUSTED_ORIGINS
Restarted everything...
rm /opt/mailman/mm/var/locks/* systemctl restart mailman3 systemctl restart mailmanweb systemctl daemon-reload systemctl restart qcluster
# pstree systemd─┬─
├─gunicorn───gunicorn
├─mailman-web───mailman-web───4*[mailman-web]
├─nginx───2*[nginx]
├─postgres───18*[postgres]
├─python3─┬─11*[python3]
│ ├─python3───{python3}
│ └─python3───2*[python3]I deleted the custom Welcome template via the WebUI (Postorius), and checked it was removed from the database (PostGres mailman.template).
Then, as site-owner, creating a new custom Welcome template (again via the WebUI) I can see it correctly added to the database in mailman.template.uri
Previously it was writing ...
So far so good. Nothing showing in mailmanweb.log yet.
Then, still as site-owner using Postorius, I mass-subscribed a new member (pre confirm, pre approve, pre verify all ticked).
The error below displays on the webpage, the member is subscribed, but no custom welcome message emailed out.
=================================== HTTP Error 400: HTTPSConnectionPool(host='lists.mydomain.com', port=443): Read timed out. (read timeout=5)
And the syslog shows a ton of issues, starting with:
===================================
2024-04-22T23:39:27.536002+00:00 lists mailman[1317]: Apr 22 23:39:27
2024 (1317) HTTPSConnectionPool(host='<LISTS.MYDOMAIN.COM>', port=443):
Read timed out. (read timeout=5)
2024-04-22T23:39:27.537648+00:00 lists mailman[1317]: Traceback (most
recent call last):
2024-04-22T23:39:27.537681+00:00 lists mailman[1317]: File
"/opt/mailman/venv/lib/python3.11/site-packages/urllib3/connectionpool.py",
line 536, in _make_request
2024-04-22T23:39:27.537700+00:00 lists mailman[1317]: response =
conn.getresponse()
2024-04-22T23:39:27.537717+00:00 lists mailman[1317]:
^^^^^^^^^^^^^^^^^^
2024-04-22T23:39:27.537740+00:00 lists mailman[1317]: File
"/opt/mailman/venv/lib/python3.11/site-packages/urllib3/connection.py",
line 461, in getresponse
2024-04-22T23:39:27.537757+00:00 lists mailman[1317]:
httplib_response = super().getresponse()
2024-04-22T23:39:27.537772+00:00 lists mailman[1317]:
^^^^^^^^^^^^^^^^^^^^^
2024-04-22T23:39:27.537790+00:00 lists mailman[1317]: File
"/usr/lib/python3.11/http/client.py", line 1374, in getresponse
I removed the custom welcome message and the newly subscribed members. The default "You've been unsubscribed" notice got emailed out okay.
Then mass subscribing a member, the default welcome message gets emailed out okay and no error displayed on the webpage.
I have no idea what the report in syslog is saying, or where to look next. Any help appreciated.
On 4/22/24 17:06, Mark wrote:
There was no line POSTORIUS_TEMPLATE_BASE_URL in /etc/mailman3/settings.py
So presumably you were getting the default
POSTORIUS_TEMPLATE_BASE_URL = 'http://localhost:8000'
from mailman-web/settings/mailman.py because of
from mailman_web.settings.mailman import *
in your settings.py.
Then, as site-owner, creating a new custom Welcome template (again via the WebUI) I can see it correctly added to the database in mailman.template.uri
- https://lists.mydomain.com/mailman3/api/templates/list/lists.mydomain.com/li...
And what happens if you go to that URL in a web browser? Also what do you get when going to https://lists.mydomain.com/mailman3?
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 2024-04-23 11:33, Mark Sapiro wrote:
On 4/22/24 17:06, Mark wrote:
There was no line POSTORIUS_TEMPLATE_BASE_URL in /etc/mailman3/settings.py
So presumably you were getting the default
POSTORIUS_TEMPLATE_BASE_URL = 'http://localhost:8000'
from mailman-web/settings/mailman.py because of from mailman_web.settings.mailman import * in your settings.py.
Yes.
# Mailman Web configuration file. # /etc/mailman3/settings.py # Get the default settings. from mailman_web.settings.base import * from mailman_web.settings.mailman import * ...
Then, as site-owner, creating a new custom Welcome template (again via the WebUI) I can see it correctly added to the database in mailman.template.uri
- https://lists.mydomain.com/mailman3/api/templates/list/lists.mydomain.com/li...
That URI should have been;
https://lists.mydomain.com/mailman3/api/templates/list/mytestlist.lists.mydo...
If I go to the above URL, the text of the list's Custom welcome message displays on an otherwise empty webpage.
And what happens if you go to that URL in a web browser? Also what do you get when going to https://lists.mydomain.com/mailman3?
I get the Postorius page "Mailing Lists" that displays the table of lists.
On 4/22/24 19:23, Mark wrote:
On 2024-04-23 11:33, Mark Sapiro wrote:
On 4/22/24 17:06, Mark wrote:
Then, as site-owner, creating a new custom Welcome template (again via the WebUI) I can see it correctly added to the database in mailman.template.uri
- https://lists.mydomain.com/mailman3/api/templates/list/lists.mydomain.com/li...
That URI should have been;
https://lists.mydomain.com/mailman3/api/templates/list/mytestlist.lists.mydo...
I think you are saying the what you reported originally was not correct and the URL in the template table was in fact the latter URL.
If I go to the above URL, the text of the list's Custom welcome message displays on an otherwise empty webpage.
And that's what is expected. So you can go to that URL and get the template, but when Mailman Core tries to do the same thing, it gets a timeout. Is there some firewall or other security software blocking this access?
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 2024-04-23 11:33, Mark Sapiro wrote:
That URI should have been;
https://lists.mydomain.com/mailman3/api/templates/list/mytestlist.lists.mydo...
I think you are saying the what you reported originally was not correct and the URL in the template table was in fact the latter URL.
I meant when I typed the URL in my first email I had unintentionally omitted the name of the list (mytestlist) when I redacted the real domain name to 'lists.mydomain.com'.
...emplates/list/lists.mydomain.com/list:user:noti...
I should have written
...emplates/list/mytestlist.lists.mydomain.com/list:user:noti...
If I go to the above URL, the text of the list's Custom welcome message displays on an otherwise empty webpage.
And that's what is expected. So you can go to that URL and get the template, but when Mailman Core tries to do the same thing, it gets a timeout. Is there some firewall or other security software blocking this access?
#ufw disable Firewall stopped and disabled on system startup
Then restarted everything...
# ufw status verbose Status: inactive
And still getting the error in the Postorius webpage ...
HTTP Error 400: HTTPSConnectionPool(host='lists.mydomain.com', port=443): Read timed out. (read timeout=5)
On 4/22/24 21:29, Mark wrote:
And still getting the error in the Postorius webpage ...
HTTP Error 400: HTTPSConnectionPool(host='lists.mydomain.com', port=443): Read timed out. (read timeout=5)
What do you get if you do
sudo -u mailman curl
https://lists.mydomain.com/mailman3/api/templates/list/mytestlist.lists.mydomain.com/list:user:notice:welcome
with the template set in Postorius?
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 2024-04-24 03:58, Mark Sapiro wrote:
What do you get if you do
sudo -u mailman curl https://lists.mydomain.com/mailman3/api/templates/list/mytestlist.lists.mydomain.com/list:user:notice:welcomewith the template set in Postorius?
The welcome message is returned.
root@hel:~# sudo -u mailman curl https://LISTS.MYDOMAIN.COM/mailman3/api/templates/list/MYTESTLIST.LISTS.MYDO... Test Welcome message for new subscribers.root@hel:~#
On 4/23/24 14:59, Mark wrote:
The welcome message is returned.
root@hel:~# sudo -u mailman curl https://LISTS.MYDOMAIN.COM/mailman3/api/templates/list/MYTESTLIST.LISTS.MYDO... Test Welcome message for new subscribers.root@hel:~#
I don't know what the issue is. curl, running as user mailman, is able to get the url, but Mailman core, presumably running as user mailman, times out trying to get the url. I can't explain why.
Try the following in mailman shell running as user mailman
(venv) $ mailman shell
Welcome to the GNU Mailman shell
Use commit() to commit changes.
Use abort() to discard changes since the last commit.
Exit with ctrl+D does an implicit commit() but exit() does not.
>>> import requests
>>> r = requests.get(
...
'https://LISTS.MYDOMAIN.COM/mailman3/api/templates/list/MYTESTLIST.LISTS.MYDOMAIN.COM/list:user:notice:welcome',
... timeout=5)
>>> t.text
Note the URL above is all on one line.
This is exactly what Mailman core is doing to get the template assuming
the uri in the template` table entry is the argument of the
requests.get().
If this times out, try increasing the timeout= value.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 2024-04-24 10:38, Mark Sapiro wrote:
Try the following in
mailman shellrunning as user mailman(venv) $ mailman shell Welcome to the GNU Mailman shell Use commit() to commit changes. Use abort() to discard changes since the last commit. Exit with ctrl+D does an implicit commit() but exit() does not. >>> import requests >>> r = requests.get( ... 'https://LISTS.MYDOMAIN.COM/mailman3/api/templates/list/MYTESTLIST.LISTS.MYDOMAIN.COM/list:user:notice:welcome', ... timeout=5) >>> t.textNote the URL above is all on one line. This is exactly what Mailman core is doing to get the template assuming the
uriin the template` table entry is the argument of the requests.get().If this times out, try increasing the timeout= value.
I crossed emails with you on my last post.
I reset the POSTORIUS_TEMPLATE_BASE_URL (in settings.py) back to the domain name, restarted everything, created a new Welcome template (using Postorius) and ...
import requests r = requests.get('https://MYDOMAIN.COM/mailman3/api/templates/list/MYTESTLIST.MYDOMAIN.COM/lis...', timeout=5) r.text 'Another test Welcome message for new members.'
... so all well there. And mailman is running mailman core ...
# cd /opt/mailman/venv/bin/ # ls -la ... -rwxr-xr-x 1 mailman mailman 230 Jan 26 04:00 gunicorn -rwxr-xr-x 1 mailman mailman 231 Jan 26 04:00 mailman -rwxr-xr-x 1 mailman mailman 230 Mar 25 22:18 mailman-web ...
On 4/23/24 18:12, Mark wrote:
import requests r = requests.get('https://MYDOMAIN.COM/mailman3/api/templates/list/MYTESTLIST.MYDOMAIN.COM/lis...', timeout=5) r.text 'Another test Welcome message for new members.'
Did you delete and recreate the template after changing POSTORIUS_TEMPLATE_BASE_URL? If not, the template table still has the old URI.
... so all well there. And mailman is running mailman core ...
# cd /opt/mailman/venv/bin/ # ls -la ... -rwxr-xr-x 1 mailman mailman 230 Jan 26 04:00 gunicorn -rwxr-xr-x 1 mailman mailman 231 Jan 26 04:00 mailman -rwxr-xr-x 1 mailman mailman 230 Mar 25 22:18 mailman-web ...
That only shows the user and group that owns the files, not whose running them.
What does ps -fwwu mailman|grep runner show?
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Did you delete and recreate the template after changing POSTORIUS_TEMPLATE_BASE_URL? If not, the template table still has the old URI.
Yes. Every test is: delete template and users (via Postorius), make any config changes, restart everything, then new template and new user (via Postorius).
What does
ps -fwwu mailman|grep runnershow?
# ps -fwwu mailman|grep runner mailman 375726 375716 0 02:00 ? 00:00:01 /opt/mailman/venv/bin/python3 /opt/mailman/venv/bin/runner -C /etc/mailman3/mailman.cfg --runner=archive:0:1 mailman 375727 375716 0 02:00 ? 00:00:01 /opt/mailman/venv/bin/python3 /opt/mailman/venv/bin/runner -C /etc/mailman3/mailman.cfg --runner=bounces:0:1 mailman 375728 375716 0 02:00 ? 00:00:01 /opt/mailman/venv/bin/python3 /opt/mailman/venv/bin/runner -C /etc/mailman3/mailman.cfg --runner=command:0:1 mailman 375729 375716 0 02:00 ? 00:00:01 /opt/mailman/venv/bin/python3 /opt/mailman/venv/bin/runner -C /etc/mailman3/mailman.cfg --runner=in:0:1 mailman 375730 375716 0 02:00 ? 00:00:01 /opt/mailman/venv/bin/python3 /opt/mailman/venv/bin/runner -C /etc/mailman3/mailman.cfg --runner=lmtp:0:1 mailman 375731 375716 0 02:00 ? 00:00:01 /opt/mailman/venv/bin/python3 /opt/mailman/venv/bin/runner -C /etc/mailman3/mailman.cfg --runner=nntp:0:1 mailman 375732 375716 0 02:00 ? 00:00:01 /opt/mailman/venv/bin/python3 /opt/mailman/venv/bin/runner -C /etc/mailman3/mailman.cfg --runner=out:0:1 mailman 375733 375716 0 02:00 ? 00:00:01 /opt/mailman/venv/bin/python3 /opt/mailman/venv/bin/runner -C /etc/mailman3/mailman.cfg --runner=pipeline:0:1 mailman 375734 375716 0 02:00 ? 00:00:01 /opt/mailman/venv/bin/python3 /opt/mailman/venv/bin/runner -C /etc/mailman3/mailman.cfg --runner=rest:0:1 mailman 375735 375716 0 02:00 ? 00:00:00 /opt/mailman/venv/bin/python3 /opt/mailman/venv/bin/runner -C /etc/mailman3/mailman.cfg --runner=retry:0:1 mailman 375736 375716 0 02:00 ? 00:00:00 /opt/mailman/venv/bin/python3 /opt/mailman/venv/bin/runner -C /etc/mailman3/mailman.cfg --runner=task:0:1 mailman 375737 375716 0 02:00 ? 00:00:01 /opt/mailman/venv/bin/python3 /opt/mailman/venv/bin/runner -C /etc/mailman3/mailman.cfg --runner=virgin:0:1 mailman 375738 375716 0 02:00 ? 00:00:01 /opt/mailman/venv/bin/python3 /opt/mailman/venv/bin/runner -C /etc/mailman3/mailman.cfg --runner=digest:0:1 mailman 375790 375734 0 02:00 ? 00:00:00 /opt/mailman/venv/bin/python3 /opt/mailman/venv/bin/runner -C /etc/mailman3/mailman.cfg --runner=rest:0:1 mailman 375791 375734 0 02:00 ? 00:00:00 /opt/mailman/venv/bin/python3 /opt/mailman/venv/bin/runner -C /etc/mailman3/mailman.cfg --runner=rest:0:1
On 4/23/24 20:36, Mark wrote:
Did you delete and recreate the template after changing POSTORIUS_TEMPLATE_BASE_URL? If not, the template table still has the old URI.
Yes. Every test is: delete template and users (via Postorius), make any config changes, restart everything, then new template and new user (via Postorius).
OK
What does
ps -fwwu mailman|grep runnershow?# ps -fwwu mailman|grep runner mailman 375726 375716 0 02:00 ? 00:00:01 /opt/mailman/venv/bin/python3 /opt/mailman/venv/bin/runner -C /etc/mailman3/mailman.cfg --runner=archive:0:1 mailman 375727 375716 0 02:00 ? 00:00:01 /opt/mailman/venv/bin/python3 /opt/mailman/venv/bin/runner -C /etc/mailman3/mailman.cfg --runner=bounces:0:1 mailman 375728 375716 0 02:00 ? 00:00:01 /opt/mailman/venv/bin/python3 /opt/mailman/venv/bin/runner -C /etc/mailman3/mailman.cfg --runner=command:0:1 mailman 375729 375716 0 02:00 ? 00:00:01 /opt/mailman/venv/bin/python3 /opt/mailman/venv/bin/runner -C /etc/mailman3/mailman.cfg --runner=in:0:1 mailman 375730 375716 0 02:00 ? 00:00:01 /opt/mailman/venv/bin/python3 /opt/mailman/venv/bin/runner -C /etc/mailman3/mailman.cfg --runner=lmtp:0:1 mailman 375731 375716 0 02:00 ? 00:00:01 /opt/mailman/venv/bin/python3 /opt/mailman/venv/bin/runner -C /etc/mailman3/mailman.cfg --runner=nntp:0:1 mailman 375732 375716 0 02:00 ? 00:00:01 /opt/mailman/venv/bin/python3 /opt/mailman/venv/bin/runner -C /etc/mailman3/mailman.cfg --runner=out:0:1 mailman 375733 375716 0 02:00 ? 00:00:01 /opt/mailman/venv/bin/python3 /opt/mailman/venv/bin/runner -C /etc/mailman3/mailman.cfg --runner=pipeline:0:1 mailman 375734 375716 0 02:00 ? 00:00:01 /opt/mailman/venv/bin/python3 /opt/mailman/venv/bin/runner -C /etc/mailman3/mailman.cfg --runner=rest:0:1 mailman 375735 375716 0 02:00 ? 00:00:00 /opt/mailman/venv/bin/python3 /opt/mailman/venv/bin/runner -C /etc/mailman3/mailman.cfg --runner=retry:0:1 mailman 375736 375716 0 02:00 ? 00:00:00 /opt/mailman/venv/bin/python3 /opt/mailman/venv/bin/runner -C /etc/mailman3/mailman.cfg --runner=task:0:1 mailman 375737 375716 0 02:00 ? 00:00:01 /opt/mailman/venv/bin/python3 /opt/mailman/venv/bin/runner -C /etc/mailman3/mailman.cfg --runner=virgin:0:1 mailman 375738 375716 0 02:00 ? 00:00:01 /opt/mailman/venv/bin/python3 /opt/mailman/venv/bin/runner -C /etc/mailman3/mailman.cfg --runner=digest:0:1 mailman 375790 375734 0 02:00 ? 00:00:00 /opt/mailman/venv/bin/python3 /opt/mailman/venv/bin/runner -C /etc/mailman3/mailman.cfg --runner=rest:0:1 mailman 375791 375734 0 02:00 ? 00:00:00 /opt/mailman/venv/bin/python3 /opt/mailman/venv/bin/runner -C /etc/mailman3/mailman.cfg --runner=rest:0:1
So all the runners are running as user mailman. That's good. What exactly is the issue now. Is it retrievals timing out or an empty welcome message?
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 2024-04-24 14:18, Mark Sapiro wrote:
So all the runners are running as user mailman. That's good. What exactly is the issue now. Is it retrievals timing out or an empty welcome message?
The issue is:
A. With POSTORIUS_TEMPLATE_BASE_URL = 'http://<REDACTED-PUBLIC-IPv4-ADDRESS>', it sends an empty Welcome message when a new member is added.
B. With POSTORIUS_TEMPLATE_BASE_URL = 'https://MYDOMAIN.COM', no Welcome message gets sent and a timeout error appears on the webpage, although the new member gets added.
On 4/23/24 23:20, Mark wrote:
The issue is:
A. With POSTORIUS_TEMPLATE_BASE_URL = 'http://<REDACTED-PUBLIC-IPv4-ADDRESS>', it sends an empty Welcome message when a new member is added.
B. With POSTORIUS_TEMPLATE_BASE_URL = 'https://MYDOMAIN.COM', no Welcome message gets sent and a timeout error appears on the webpage, although the new member gets added.
I think the difference is http vs https, not the IP vs domain name.
Also, with either of these, you are addressing the web server and not the wsgi server. The wsgi server is presumable listening on port 8000 and in the beginning when you had the default
POSTORIUS_TEMPLATE_BASE_URL = 'http://localhost:8000'
you also got the timeout error and you also got an email to the site owner with 'Site matching query does not exist.' for 'Request URL: http://<REDACTED-PUBLIC-IPv4-ADDRESS>/mailman3/lists/'
This leads me to think 'localhost' resolves to <REDACTED-PUBLIC-IPv4-ADDRESS> instead of 127.0.0.1. Also, <REDACTED-PUBLIC-IPv4-ADDRESS> is not in ALLOWED_HOSTS. Have you set ALLOWED_HOSTS in your settings.py? It should be something like
ALLOWED_HOSTS = [
"localhost", # Archiving API from Mailman, keep it.
"127.0.0.1",
"<REDACTED-PUBLIC-IPv4-ADDRESS>",
"MYDOMAIN.COM",
]
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 2024-04-25 01:37, Mark Sapiro wrote:
I think the difference is http vs https, not the IP vs domain name.
Also, with either of these, you are addressing the web server and not the wsgi server. The wsgi server is presumable listening on port 8000 and in the beginning when you had the default
POSTORIUS_TEMPLATE_BASE_URL = 'http://localhost:8000'
With this default, the template URI is in the format of
http://localhost:8000/mailman3/api/templates/list/mytestlist.mydomain.com/li...
Looking at this URI in lynx (on the same server as Mailman) it returns the text of the custom Welcome template, as does curl and wget.
And it's the same if 127.0.0.1 is used in place of localhost -- ie.
http://127.0.0.1:8000/mailman3/api/templates/list/mytestlist.mydomain.com/li...
And the same result if the domain name is used with https -- ie.
https://mydomain.com/mailman3/api/templates/list/mytestlist.mydomain.com/lis...
So, the template is all good, it can be accessed by Lynx, but the part of the Mailman suite that sends out the custom Welcome message cannot access it and spits out the error.
Where are the custom templates stored? Is there a permissions issue perhaps? A grep for the text of my Welcome message turns up zero results.
# grep -Rnw '/' -e 'Test two million for welcome'
By the way, in Lynx when the template's URI uses the public ip adress -- ie.
http://my-public-ip-address/mailman3/api/templates/list/mytestlist.mydomain....
... it returns a "404 page not found" in Lynx.
Given that this is pretty much an out-of-the-box installation (venv installation as per the docs) and everything else works perfectly I'm thinking surely others have encountered this. Or is it just me?
It's completely done my head in, so if anyone can offer a glimmer of light I'll grab it.
<REDACTED-PUBLIC-IPv4-ADDRESS> instead of 127.0.0.1. Also, Have you set ALLOWED_HOSTS in your settings.py? It should be something like
ALLOWED_HOSTS = [ "localhost", # Archiving API from Mailman, keep it. "127.0.0.1", "<REDACTED-PUBLIC-IPv4-ADDRESS>", "MYDOMAIN.COM", ]
Yes.
On 4/25/24 23:07, Mark wrote:
So, the template is all good, it can be accessed by Lynx, but the part of the Mailman suite that sends out the custom Welcome message cannot access it and spits out the error.
Where are the custom templates stored? Is there a permissions issue perhaps? A grep for the text of my Welcome message turns up zero results.
They are stored in the postorius_emailtemplate table in the mailmanweb database.
And yes, I think there is some permissions issue, but it is not simple. It probably involves apparmor or some other security manager. If it were simple, the mailman shell test I suggested at https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/message/... would fail.
By the way, in Lynx when the template's URI uses the public ip adress -- ie.
http://my-public-ip-address/mailman3/api/templates/list/mytestlist.mydomain....
... it returns a "404 page not found" in Lynx.
Which is probably why that setting sent the welcome with empty content.
Given that this is pretty much an out-of-the-box installation (venv installation as per the docs) and everything else works perfectly I'm thinking surely others have encountered this. Or is it just me?
I just did a test on my development system and I see the same timeout error, so it's not just you. I'll work on this.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mark Sapiro wrote:
I just did a test on my development system and I see the same timeout error, so it's not just you. I'll work on this.
This is very strange. In my test, I see this in the apache log:
127.0.0.1 - - [26/Apr/2024:08:59:25 -0700] "GET /mailman3/api/templates/list/test.msapiro.net/list:user:notice:welcome HTTP/1.1" 200 4215 "-" "python-requests/2.28.1"
I.e., apache gets the request and serves the template, and then 5 seconds later I see
Apr 26 08:59:30 2024 (2592) deque: Traceback (most recent call last): ... socket.timeout: The read operation timed out
I also note that this timeout only occurs when it is a subscription via
REST. I.e. subscription via email or mailman addmembers sends the
proper welcome message.
So it has something to do with the retrieval being done by a rest runner, but I still don't know what the underlying issue is.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 4/26/24 17:04, Mark Sapiro wrote:
I just did a test on my development system and I see the same timeout error, so it's not just you. I'll work on this. ... I also note that this timeout only occurs when it is a subscription via REST. I.e. subscription via email or
mailman addmemberssends theMark Sapiro wrote: proper welcome message.
So it has something to do with the retrieval being done by a rest runner, but I still don't know what the underlying issue is.
More strangeness. I've now done a test on one of the production servers I manage, and there is no error. Both the production server and the development server are running the identical Mailman code installed in a venv, they both have the same version of urllib3 and similar Python versions, 3.9.16 on the development server and 3.9.10 on the production server. Both servers are running under Ubuntu 20.04.6. The only other difference I see is on the production server everything runs as user mailman and on the development server everything runs as user mark.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 2024-04-28 01:32, Mark Sapiro wrote:
I also note that this timeout only occurs when it is a subscription via REST. I.e. subscription via email or
mailman addmemberssends the proper welcome message.So it has something to do with the retrieval being done by a rest runner, but I still don't know what the underlying issue is.
I created a text file as a custom welcome message and chowned it to mailman:mailman, then edited the database record (template.uri) from
http://localhost:8000/mailman3/api/templates/list/mytestlist.mydomain.com/li... TO file:///home/mark/welcome.txt
This delivered the custom welcome message to the user when subscribed via Mass Subscribe.
I'm not sure what this tells me other than a call can be made to a file but not the database.
More strangeness. I've now done a test on one of the production servers I manage, and there is no error. Both the production server and the development server are running the identical Mailman code installed in a venv, they both have the same version of urllib3 and similar Python versions, 3.9.16 on the development server and 3.9.10 on the production server. Both servers are running under Ubuntu 20.04.6. The only other difference I see is on the production server everything runs as user mailman and on the development server everything runs as user mark.
Mine is Debian 12 with Python 3.11.2. Could a newer Python version really cause this amount of grief?
On 4/27/24 18:44, Mark wrote:
I created a text file as a custom welcome message and chowned it to mailman:mailman, then edited the database record (template.uri) from
http://localhost:8000/mailman3/api/templates/list/mytestlist.mydomain.com/li... TO file:///home/mark/welcome.txt
This delivered the custom welcome message to the user when subscribed via Mass Subscribe.
I'm not sure what this tells me other than a call can be made to a file but not the database.
The retrieval of a file does not involve calling requests.get() to retrieve the URL via http which is where the issue occurs.
There's another way to set templates in the file system. If there is no entry in the template table matching the requested template name and context the file system is searched according to the algorithm described at <https://gitlab.com/mailman/mailman/-/blob/master/src/mailman/utilities/i18n....>. $template_dir in that description is Mailman's var/templates/ directory.
So for example, to create a lists specific english language welcome template for the list.example.com list you could just put the template in var/templates/lists/list.example.com/en/list:user:notice:welcome.txt or to make it sitewide, in var/templates/site/en/list:user:notice:welcome.txt
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
I created a text file as a custom welcome message ...
The retrieval of a file does not involve calling requests.get() to retrieve the URL via http which is where the issue occurs.
...
So for example, to create a lists specific english language welcome template for the list.example.com list you could just put the template in var/templates/lists/list.example.com/en/list:user:notice:welcome.txt or to make it sitewide, in var/templates/site/en/list:user:notice:welcome.txt
Thank you, that is indeed useful info, and as the site-owner I'll use it. But it's of no help to a list-owner who wants to create a custom template, because they have to use Postorius.
So, I'm thinking (in broad brush strokes) that if I'm stuck with this "http glitch" I could write a script to:
- extract the text of the custom message from the database,
- write that message text to a file
- update the templates.uri entries to replace the http://localhost... with file:///path/to/file
- cron the script to run every few minutes.
It screams of utter clunk I know...
On 4/27/24 08:32, Mark Sapiro wrote:
More strangeness. I've now done a test on one of the production servers I manage, and there is no error. Both the production server and the development server are running the identical Mailman code installed in a venv, they both have the same version of urllib3 and similar Python versions, 3.9.16 on the development server and 3.9.10 on the production server. Both servers are running under Ubuntu 20.04.6. The only other difference I see is on the production server everything runs as user mailman and on the development server everything runs as user mark.
I have now reinstalled Mailman from scratch on the development server in a Python 3.12.3 venv using the heads of the gitlab branches and getting the current versions of all the dependencies and this installation gets no error. So while I still have an older venv that produces this error, I cannot duplicate the error in a new install.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 2024-05-01 03:12, Mark Sapiro wrote:
I have now reinstalled Mailman from scratch on the development server in a Python 3.12.3 venv using the heads of the gitlab branches and getting the current versions of all the dependencies and this installation gets no error. So while I still have an older venv that produces this error, I cannot duplicate the error in a new install.
In the new installation, are the template.uri (in the mailman DB) records written as http://localhost:8000/... ?
On 4/30/24 17:54, Mark wrote:
In the new installation, are the template.uri (in the mailman DB) records written as http://localhost:8000/... ?
No. In both the installation that worked and the one that didn't, they were https://msapiro.net/..., but I don't think it would have mattered.
When I reinstalled, I kept the same database, I only rebuilt the venv.
As I have said at https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/message/..., this is all strange. My logs from the failure show the web server getting the request and serving the template with a 200 status and then after 5 seconds which is the specified timeout a requests.exceptions.ReadTimeout error is logged in mailman.log.
The exact same code is executed whether by a rest runner gunicorn worker or via some other Mailman core runner, but only the rest runner fails. Further, a reinstalled venv on the failing server has no error, and another server which has a venv identical to the failing one except for Python micro version also has no error.
I can't begin to understand what the underlying cause might be.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 2024-05-01 13:35, Mark Sapiro wrote:
On 4/30/24 17:54, Mark wrote:
In the new installation, are the template.uri (in the mailman DB) records written as http://localhost:8000/... ?
No. In both the installation that worked and the one that didn't, they were https://msapiro.net/..., but I don't think it would have mattered.
When I reinstalled, I kept the same database, I only rebuilt the venv.
As I have said at https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/message/..., this is all strange. My logs from the failure show the web server getting the request and serving the template with a 200 status and then after 5 seconds which is the specified timeout a requests.exceptions.ReadTimeout error is logged in mailman.log.
The exact same code is executed whether by a rest runner gunicorn worker or via some other Mailman core runner, but only the rest runner fails. Further, a reinstalled venv on the failing server has no error, and another server which has a venv identical to the failing one except for Python micro version also has no error.
I can't begin to understand what the underlying cause might be.
Thank you Mark. I'll give rebuilding venv a crack. In the meantime I've cobbled together a script -- https://postmark.io/scripts/ -- and it's got things going for the moment.
.
On 2024-04-24 03:58, Mark Sapiro wrote:
What do you get if you do
sudo -u mailman curl https://lists.mydomain.com/mailman3/api/templates/list/mytestlist.lists.mydomain.com/list:user:notice:welcomewith the template set in Postorius?
The welcome message is returned.
root@hel:~# sudo -u mailman curl https://LISTS.MYDOMAIN.COM/mailman3/api/templates/list/MYTESTLIST.LISTS.MYDO... Test Welcome message for new subscribers.root@hel:~#
Making progress...
I edited settings file again, replacing the domain name with the IP address ()
POSTORIUS_TEMPLATE_BASE_URL = 'http://12.34.56.78' (no SSL)
The URI in the template table now has this.
http://12.34.56.78/mailman3/api/templates/list/MYTESTLIST.MYDOMAIN.COM/list:...
Now, when Mass subscribing a new member, a Welcome email is sent to the new member with the correct Subject and From, but the message is blank.
On 4/23/24 17:51, Mark wrote:
On 2024-04-24 03:58, Mark Sapiro wrote:
What do you get if you do
sudo -u mailman curl https://lists.mydomain.com/mailman3/api/templates/list/mytestlist.lists.mydomain.com/list:user:notice:welcomewith the template set in Postorius?
The welcome message is returned.
root@hel:~# sudo -u mailman curl https://LISTS.MYDOMAIN.COM/mailman3/api/templates/list/MYTESTLIST.LISTS.MYDO... Test Welcome message for new subscribers.root@hel:~#
Making progress...
I edited settings file again, replacing the domain name with the IP address ()
POSTORIUS_TEMPLATE_BASE_URL = 'http://12.34.56.78' (no SSL)
That should be no different than http://host_name where host_name's
DNS A record is 12.34.56.78.
The URI in the template table now has this.
http://12.34.56.78/mailman3/api/templates/list/MYTESTLIST.MYDOMAIN.COM/list:...
And if you go there in a browser, what do you get?
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
The URI in the template table now has this.
http://12.34.56.78/mailman3/api/templates/list/MYTESTLIST.MYDOMAIN.COM/list:...
And if you go there in a browser, what do you get?
That returns a "Page not found". Likewise with curl.
participants (2)
-
Mark -
Mark Sapiro