Hello together,
I migrated a number of lists from mailman2 to mailman3 using the import functionality. Everthing works fine and the new lists are in use. Currently I´m using the maxking-docker-images with postorius: Mailman Core Version GNU Mailman 3.3.3 (Tom Sawyer) Mailman Core API Version 3.1 Mailman Core Python Version 3.8.5 (default, Feb 19 2021, 09:27:16) [GCC 9.3.0]
After this migration I started a refactoring with the list-styles specially with message-footers. Already the old lists in mailman2 were confused in some configuration details an I was clear to import this "misconfiguration" and clean it up afterwards. But this was much more difficult than expected: some lists are using a footer (from the mailman2-config) and there is no record in the template-section visible. Further I found no configuration item which points to this "old" footer.
I found the file [code] /usr/lib/python3.8/site-packages/mailman/templates/de/list:member:generic:footer.txt [/code] which includes the old footer: [code] _______________________________________________ $display_name Mailingliste -- $listname Zur Abmeldung von dieser Mailingliste senden Sie eine Nachricht an ${short_listname}-leave@${domain} [/code]
but no configuration item to disable this behaviour. I created a new template list:member:regular:footer for the list to override the footer. But the old footer is still used - in detail: the resulting message sent to all subcribers of the list contains only the footer and the content sent to the list is an attachement of that "footer-message". This is the basic reason why I want to change it.
In the logfiles I found an error and a warning: [code] ERROR 2021-05-24 16:22:22,895 1648 django.security.DisallowedHost Invalid HTTP_HOST header: 'tgc_mailman_web:8000'. The domain name provided is not valid according to RFC 1034/1035. WARNING 2021-05-24 16:22:22,896 1648 django.request Bad Request: /postorius/api/templates/list/homepage.gruppe.tgcamberg1848.de/list:member:regular:footer [/code]
Obviously I have a problem with the hostname. I set [code] POSTORIUS_TEMPLATE_BASE_URL=http://tgc_mailman_web:8000 [/code] where tgc_mailman_web is my container name. This URL ist stored in the DB-table "template" for the name=list:member:regular:footer. A call to my fqdn or to localhost [code] wget http://localhost:8000/postorius/api/templates/list/homepage.gruppe.tgcamberg... [/code] will return the footer itself - this works fine. Therefore, should i set POSTORIUS_TEMPLATE_BASE_URL to http://localhost:8000?
I think, resolving this issue will use the newly configured footer. But I started with the intention to remove alle footers due to the attachement situation described above.
Does anybody have an idea how to change the configuration of my list? Are these problems probably related to the import? Currently we are using more then 25 lists with a lot of users. Manually recreating all lists is not possible an could only be a test for a single list.
Best regards, chrclaus
On 5/24/21 10:43 AM, christopher.claus@tgcamberg1848.de wrote:
I found the file [code] /usr/lib/python3.8/site-packages/mailman/templates/de/list:member:generic:footer.txt [/code] which includes the old footer: [code] _______________________________________________ $display_name Mailingliste -- $listname Zur Abmeldung von dieser Mailingliste senden Sie eine Nachricht an ${short_listname}-leave@${domain} [/code]
but no configuration item to disable this behaviour.
That is the default footer template that's part of the Mailman distribution. That template is overridden by those in Mailman's var/templates/ directory. Mailmnan's import21 process created footers template in var/templates/lists/<LIST_ID>/list:member:regular:footer.txt and var/templates/lists/<LIST_ID>/list:member:digest:footer.txt
I created a new template list:member:regular:footer for the list to override the footer. But the old footer is still used
where did you put it?
in detail: the resulting message sent to all subcribers of the list contains only the footer and the content sent to the list is an attachement of that "footer-message". This is the basic reason why I want to change it.
In the logfiles I found an error and a warning: [code] ERROR 2021-05-24 16:22:22,895 1648 django.security.DisallowedHost Invalid HTTP_HOST header: 'tgc_mailman_web:8000'. The domain name provided is not valid according to RFC 1034/1035. WARNING 2021-05-24 16:22:22,896 1648 django.request Bad Request: /postorius/api/templates/list/homepage.gruppe.tgcamberg1848.de/list:member:regular:footer [/code]
It appears you created the new template with Postorius, but there is an issue for Postorius templates. Possibly your Django setting for POSTORIUS_TEMPLATE_BASE_URL needs to be adjusted.
Obviously I have a problem with the hostname. I set [code] POSTORIUS_TEMPLATE_BASE_URL=http://tgc_mailman_web:8000 [/code] where tgc_mailman_web is my container name. This URL ist stored in the DB-table "template" for the name=list:member:regular:footer. A call to my fqdn or to localhost [code] wget http://localhost:8000/postorius/api/templates/list/homepage.gruppe.tgcamberg... [/code] will return the footer itself - this works fine. Therefore, should i set POSTORIUS_TEMPLATE_BASE_URL to http://localhost:8000?
If that works, yes.
I think, resolving this issue will use the newly configured footer. But I started with the intention to remove alle footers due to the attachement situation described above.
Does anybody have an idea how to change the configuration of my list? Are these problems probably related to the import? Currently we are using more then 25 lists with a lot of users. Manually recreating all lists is not possible an could only be a test for a single list.
The issue you are seeing has only to do with the fact that the footer is non-empty. It is not related to footer content. It is because you are posting other than text/plain messages and the MUAs you are viewing the mail with are confused by the footer being a separate MIME part. See the FAQ article at https://wiki.list.org/x/4030707 for discussion of this issue in MM 2.1 which is also relevant to MM 3.
To disable adding footers entirely, sitewide, create an empty file, readable by Mailman at Mailman's
var/templates/site/de/list:member:generic:footer.txt
More about templates and search rules is at templates/de/list:member:generic:footer.txt and https://gitlab.com/mailman/mailman/-/blob/master/src/mailman/utilities/i18n....
Mark Sapiro wrote:
ERROR 2021-05-24 16:22:22,895 1648 django.security.DisallowedHost Invalid HTTP_HOST header: 'tgc_mailman_web:8000'. The domain name provided is not valid according to RFC 1034/1035. WARNING 2021-05-24 16:22:22,896 1648 django.request Bad Request: /postorius/api/templates/list/homepage.gruppe.tgcamberg1848.de/list:member:regular:footer It appears you created the new template with Postorius, but there is an
issue for Postorius templates. Possibly your Django setting for POSTORIUS_TEMPLATE_BASE_URL needs to be adjusted.
Obviously I have a problem with the hostname. I set [code] POSTORIUS_TEMPLATE_BASE_URL=http://tgc_mailman_web:8000 [/code] where tgc_mailman_web is my container name. This URL ist stored in the DB-table "template" for the name=list:member:regular:footer. A call to my fqdn or to localhost [code] wget http://localhost:8000/postorius/api/templates/list/homepage.gruppe.tgcamberg... [/code] will return the footer itself - this works fine. Therefore, should i set POSTORIUS_TEMPLATE_BASE_URL to http://localhost:8000? If that works, yes.
Hi Mark,
thanks for your reply and excuse my delay. I was very busy in the last weeks but made some tests. Newly created templates with POSTORIUS_TEMPLATE_BASE_URL=http://localhost:8000 did not work. I changed it to my fqdn - at the moment I am able to access the template via this URl (e.G. http://lists.example.de:8000/postorius/api/templates/list/homepage.lists.exa...). But if I try to send an email I got the following error:
Jul 09 16:00:37 2021 (27) Uncaught runner exception: HTTPConnectionPool(host='lists.example.de', port=8000): Max retries exceeded with url: /postorius/api/templates/list/homepage.lists.example.de/list:admin:action:post (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe80eaaec40>: Failed to establish a new connection: [Errno 111] Connection refused')) Jul 09 16:00:37 2021 (27) Traceback (most recent call last): File "/usr/lib/python3.8/site-packages/urllib3/connection.py", line 159, in _new_conn conn = connection.create_connection( File "/usr/lib/python3.8/site-packages/urllib3/util/connection.py", line 84, in create_connection raise err File "/usr/lib/python3.8/site-packages/urllib3/util/connection.py", line 74, in create_connection sock.connect(sa) ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/usr/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen httplib_response = self._make_request( File "/usr/lib/python3.8/site-packages/urllib3/connectionpool.py", line 392, in _make_request conn.request(method, url, **httplib_request_kw) File "/usr/lib/python3.8/http/client.py", line 1255, in request self._send_request(method, url, body, headers, encode_chunked) File "/usr/lib/python3.8/http/client.py", line 1301, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/usr/lib/python3.8/http/client.py", line 1250, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/lib/python3.8/http/client.py", line 1010, in _send_output self.send(msg) File "/usr/lib/python3.8/http/client.py", line 950, in send self.connect() File "/usr/lib/python3.8/site-packages/urllib3/connection.py", line 187, in connect conn = self._new_conn() File "/usr/lib/python3.8/site-packages/urllib3/connection.py", line 171, in _new_conn raise NewConnectionError( urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fe80eaaec40>: Failed to establish a new connection: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/usr/lib/python3.8/site-packages/requests/adapters.py", line 439, in send resp = conn.urlopen( File "/usr/lib/python3.8/site-packages/urllib3/connectionpool.py", line 724, in urlopen retries = retries.increment( File "/usr/lib/python3.8/site-packages/urllib3/util/retry.py", line 439, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='lists.example.de', port=8000): Max retries exceeded with url: /postorius/api/templates/list/lists.example.de/list:admin:action:post (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe80eaaec40>: Failed to establish a new connection: [Errno 111] Connection refused'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/usr/lib/python3.8/site-packages/mailman/core/runner.py", line 173, in _one_iteration self._process_one_file(msg, msgdata) File "/usr/lib/python3.8/site-packages/mailman/core/runner.py", line 266, in _process_one_file keepqueued = self._dispose(mlist, msg, msgdata) File "/usr/lib/python3.8/site-packages/mailman/runners/incoming.py", line 79, in _dispose process(mlist, msg, msgdata, start_chain) File "/usr/lib/python3.8/site-packages/mailman/core/chains.py", line 79, in process link.function(mlist, msg, msgdata) File "/usr/lib/python3.8/site-packages/mailman/chains/hold.py", line 232, in _process template = getUtility(ITemplateLoader).get( File "/usr/lib/python3.8/site-packages/mailman/model/template.py", line 188, in get contents = getUtility(ITemplateManager).get( File "/usr/lib/python3.8/site-packages/mailman/database/transaction.py", line 85, in wrapper return function(args[0], config.db.store, *args[1:], **kws) File "/usr/lib/python3.8/site-packages/mailman/model/template.py", line 110, in get contents = protocols.get(actual_uri, **auth) File "/usr/lib/python3.8/site-packages/mailman/utilities/protocols.py", line 38, in get response = requests.get(url, timeout=REQUEST_TIMEOUT, **kws) File "/usr/lib/python3.8/site-packages/requests/api.py", line 76, in get return request('get', url, params=params, **kwargs) File "/usr/lib/python3.8/site-packages/requests/api.py", line 61, in request return session.request(method=method, url=url, **kwargs) File "/usr/lib/python3.8/site-packages/requests/sessions.py", line 530, in request resp = self.send(prep, **send_kwargs) File "/usr/lib/python3.8/site-packages/requests/sessions.py", line 643, in send r = adapter.send(request, **kwargs) File "/usr/lib/python3.8/site-packages/requests/adapters.py", line 516, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPConnectionPool(host='lists.example.de', port=8000): Max retries exceeded with url: /postorius/api/templates/list/lists.example.de/list:admin:action:post (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe80eaaec40>: Failed to establish a new connection: [Errno 111] Connection refused'))
I set POSTORIUS_TEMPLATE_BASE_URL=lists.example.de:8000 and I have no idea, why the connection is refused. Do you see any configuration error d or have another idea?
Best regards, chrclaus
Hi together,
does anybody has an idea for the problem "ConnectionRefusedError: [Errno 111] Connection refused"?
Best regards, chrclaus
christopher.claus@tgcamberg1848.de writes:
Hi together,
does anybody has an idea for the problem "ConnectionRefusedError: [Errno 111] Connection refused"?
Does this have anything to do with the subject?
As for ideas, no, no ideas without more information. Like what is trying to connect to what and why, is this a new install or a sudden change in a working install, if the latter, were there other changes to the system (OS or package upgrade), etc.
Hi Stephen,
please excuse my short question. This was a follow-up to my initial post. I got the "connection refused" after applying footer and header messages and after making changes to the template-url. I will formulate my questions more precise in the future.
best regards, chrclaus
On Jul 9, 2021, at 9:15 AM, christopher.claus@tgcamberg1848.de wrote:
Mark Sapiro wrote:
ERROR 2021-05-24 16:22:22,895 1648 django.security.DisallowedHost Invalid HTTP_HOST header: 'tgc_mailman_web:8000'. The domain name provided is not valid according to RFC 1034/1035. WARNING 2021-05-24 16:22:22,896 1648 django.request Bad Request: /postorius/api/templates/list/homepage.gruppe.tgcamberg1848.de/list:member:regular:footer It appears you created the new template with Postorius, but there is an
issue for Postorius templates. Possibly your Django setting for POSTORIUS_TEMPLATE_BASE_URL needs to be adjusted.
Obviously I have a problem with the hostname. I set [code] POSTORIUS_TEMPLATE_BASE_URL=http://tgc_mailman_web:8000 [/code] where tgc_mailman_web is my container name. This URL ist stored in the DB-table "template" for the name=list:member:regular:footer. A call to my fqdn or to localhost [code] wget http://localhost:8000/postorius/api/templates/list/homepage.gruppe.tgcamberg... [/code] will return the footer itself - this works fine. Therefore, should i set POSTORIUS_TEMPLATE_BASE_URL to http://localhost:8000? If that works, yes.
Hi Mark,
thanks for your reply and excuse my delay. I was very busy in the last weeks but made some tests. Newly created templates with POSTORIUS_TEMPLATE_BASE_URL=http://localhost:8000 did not work. I changed it to my fqdn - at the moment I am able to access the template via this URl (e.G. http://lists.example.de:8000/postorius/api/templates/list/homepage.lists.exa...). But if I try to send an email I got the following error:
You want the URL at which Postorius (postorius) is available from Core (mailman-core container). The default for that in the container images is set to http://mailman-web:8000. Does that not work for you?
If you want to verify, you want to exec into the Core’s container (docker exec -it mailman-core bash) and see if you can curl the URL from inside there. If you are using the default docker-compose-postorius.yaml then there is a /etc/hosts entry created by docker for the mailman-web container, hence http://mailman-web:8000 works there (but it won’t work on the host, since docker doesn’t modify the host’s /etc/hosts).
There might also be a bug in play here which would cause such connection refused errors when subscribing/unsubscribing when there was a template configured for the list. Is that the case for you or are you seeing always when trying to send the email (which would imply that the Web container simply isn’t reachable from Core).
Hi Abhilash,
excuse my delay, i was out-of-order in the last weeks.
You want the URL at which Postorius (postorius) is available from Core (mailman-core container). The default for that in the container images is set to http://mailman-web:8000. Does that not work for you?
I changed the container-names (but not the hostnames) an probably i set the POSTORIUS_TEMPLATE_BASE_URL to the container-name and not to the hostname.
If you want to verify, you want to exec into the Core’s container (docker exec -it mailman-core bash) and see if you can curl the URL from inside there. If you are using the default docker-compose-postorius.yaml then there is a /etc/hosts entry created by docker for the mailman-web container, hence http://mailman-web:8000 works there (but it won’t work on the host, since docker doesn’t modify the host’s /etc/hosts).
I modified the compose-file since i have a separate container for a database and i use a seperate network for all my other container. The consequence was, that i do not have a fix subnet. I tried to change all occurences of ip-adresses to hostnames. Therefore i stripped these parts in the compose-file. Currently, there is not entry for mailman-web in the hosts file of mailman-core. I did not find any place in your config and scripts, were this file is written. At which place, this should be done? An interesting point: mailman_core has an entry to itself (with own ip) in /etc/hosts and mailman_web also. I think, this is not correct, isnt it? But nevertheless a curl of the URL from core to web works without error message.
There might also be a bug in play here which would cause such connection refused errors when subscribing/unsubscribing when there was a template configured for the list. Is that the case for you or are you seeing always when trying to send the email (which would imply that the Web container simply isn’t reachable from Core).
I thought about the situation, that all my lists where migrated from mailman 2. Many of the have footer- and header-messages, automatic replies which are not shown in the template section of the list. Is a recreation in mailman3 a good option in this situation?
Best regards, chrclaus
Hi Abhilash, Hi Mark,
I upgraded my docker-image to maxking/docker-mailman 0.4.1 and reset my whole configuration. After applying my individual config-items step-by-step I got no errors anymore. Probably I have had a misconfiguration by trying to switch from IP-addresses to hostnames in docker-image v0.3. Thank you for the update of the docker-image to v0.4 - good job!
Best regards, chrclaus
-
Abhilash Raj -
christopher.claus＠tgcamberg1848.de -
Mark Sapiro -
Stephen J. Turnbull