Re: Held messages: Long time waiating for response of the Mailman API
by Stephan Krinetzki
Stephen J. Turnbull wrote:
> Krinetzki, Stephan writes:
> > Hi everyone,
> > any news here? We are running with Pyton 3.6 - maybe a Python
> > Upgrade can speed Mailman up?
> > It would help if you provide the original description.
> I see you're running Apache + uswgi. uwsgi has been pretty finicky,
> and most of the developers use gunicorn as the WSGI host. I suggest
> you ask on the uswgi channels as well. (I don't think that's a
> *better* place to ask, just a relevant place.)
Then we should maybe move to gunicorn, if this is the better way (or at least better supported way)
> You wrote:
> > If we try to approve them via Postorius, the message is delivered
> > to the list immediately, but the success message in Postorius takes
> > a long time - on average the process hangs for three minutes until
> > the message is no longer displayed.
> > You jump to the conclusion that the REST API is slow, but there's zero
> evidence to localize to the REST API. If your browser isn't running
> on the same host as the webserver it could even be some kind of
> network timeout (reverse DNS?) completely unrelated to Mailman
> (unlikely, but can't be ruled out with the information provided).
>
> What is in the various logs? Mailman core, Django, your backend
> database, the webserver, the WSGI hosts all may be writing to
> logs, especially if there's some kind of time out or other error
> condition.
After we enabled the "slow query log", we got in the postgresql database the following output:
2022-01-25 12:49:41.538 CET [16126] LOG: unexpected EOF on client connection with an open transaction
2022-01-25 15:34:36.047 CET [16864] LOG: duration: 30760.028 ms statement: DELETE FROM pendedkeyvalue WHERE pendedkeyvalue.id = XXX
2022-01-25 15:34:36.047 CET [3900] LOG: duration: 9372.471 ms statement: DELETE FROM pendedkeyvalue WHERE pendedkeyvalue.id = XXX
2022-01-25 16:42:07.243 CET [3900] LOG: unexpected EOF on client connection with an open transaction
Pretty sounds like a database problem?
The other logs are silent after we set the following values in the gunicorn.cfg in the mailman home:
[gunicorn]
# All the options that can be put here are available at
# http://docs.gunicorn.org/en/stable/settings.html#settings
# No of seconds to wait before killing the worker processes.
graceful_timeout = 30
# Timeout for the requests.
timeout = 360
sometimes the log (mailmansuite.log) show us this error
ERROR 2022-01-25 18:55:55,037 2599 django.request Service Unavailable: /postorius/lists/games.lists.rwth-aachen.de/held_messages
ERROR 2022-01-25 18:58:24,325 2588 postorius Mailman REST API not available
Traceback (most recent call last):
File "/opt/mailman/mailman-venv/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/opt/mailman/mailman-venv/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 384, in _make_request
six.raise_from(e, None)
File "<string>", line 2, in raise_from
File "/opt/mailman/mailman-venv/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 380, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib64/python3.6/http/client.py", line 1346, in getresponse
response.begin()
File "/usr/lib64/python3.6/http/client.py", line 307, in begin
version, status, reason = self._read_status()
File "/usr/lib64/python3.6/http/client.py", line 276, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/mailman/mailman-venv/lib64/python3.6/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/opt/mailman/mailman-venv/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen
_stacktrace=sys.exc_info()[2])
File "/opt/mailman/mailman-venv/lib64/python3.6/site-packages/urllib3/util/retry.py", line 367, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/opt/mailman/mailman-venv/lib64/python3.6/site-packages/urllib3/packages/six.py", line 685, in reraise
raise value.with_traceback(tb)
File "/opt/mailman/mailman-venv/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/opt/mailman/mailman-venv/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 384, in _make_request
six.raise_from(e, None)
File "<string>", line 2, in raise_from
File "/opt/mailman/mailman-venv/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 380, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib64/python3.6/http/client.py", line 1346, in getresponse
response.begin()
File "/usr/lib64/python3.6/http/client.py", line 307, in begin
version, status, reason = self._read_status()
File "/usr/lib64/python3.6/http/client.py", line 276, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/mailman/mailman-venv/lib64/python3.6/site-packages/mailmanclient/restbase/connection.py", line 145, in call
response = request(**params, auth=self.auth)
File "/opt/mailman/mailman-venv/lib64/python3.6/site-packages/requests/api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "/opt/mailman/mailman-venv/lib64/python3.6/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/opt/mailman/mailman-venv/lib64/python3.6/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/opt/mailman/mailman-venv/lib64/python3.6/site-packages/requests/adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/mailman/mailman-venv/lib64/python3.6/site-packages/django/core/handlers/base.py", line 113, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/opt/mailman/mailman-venv/lib64/python3.6/site-packages/django/contrib/auth/decorators.py", line 21, in _wrapped_view
return view_func(request, *args, **kwargs)
File "/opt/mailman/mailman-venv/lib64/python3.6/site-packages/postorius/auth/decorators.py", line 63, in wrapper
return fn(*args, **kwargs)
File "/opt/mailman/mailman-venv/lib64/python3.6/site-packages/postorius/views/list.py", line 698, in list_moderation
_perform_action(message_ids, mailing_list.accept_message)
File "/opt/mailman/mailman-venv/lib64/python3.6/site-packages/postorius/views/list.py", line 685, in _perform_action
action(message_id)
File "/opt/mailman/mailman-venv/lib64/python3.6/site-packages/mailmanclient/restobjects/mailinglist.py", line 364, in accept_message
return self.moderate_message(request_id, 'accept')
File "/opt/mailman/mailman-venv/lib64/python3.6/site-packages/mailmanclient/restobjects/mailinglist.py", line 334, in moderate_message
path, data, 'POST')
File "/opt/mailman/mailman-venv/lib64/python3.6/site-packages/mailmanclient/restbase/connection.py", line 169, in call
'Could not connect to Mailman API: ', repr(e))
mailmanclient.restbase.connection.MailmanConnectionError: ('Could not connect to Mailman API: ', "ConnectionError(ProtocolError('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',)),)")
* The venv is in /opt/mailman/mailman-venv
* The settings are in /opt/mailman/mailman-venv/mailman-suite/mailman-suite_project
content of mailman.cfg:
# AUTOMATICALLY GENERATED BY MAILMAN ON 2016-09-12 14:35:56
#
# This is your GNU Mailman 3 configuration file. You can edit this file to
# configure Mailman to your needs, and Mailman will never overwrite it.
# Additional configuration information is (for now) available in the
# schema.cfg file <http://tinyurl.com/cm5rtqe> and the base mailman.cfg file
# <http://tinyurl.com/dx9b8eg>.
#
# For example, uncomment the following lines to run Mailman in developer mode.
#
# [devmode]
# enabled: yes
# recipient: your.address(a)your.domain
[mailman]
site_owner: mailman(a)lists.example.com
layout: here
pending_request_life: 3d
#layout: fhs
#[general]
#hostname = 'lists.example.com'
# The default language for this server.
default_language: de
[paths.here]
var_dir: /opt/mailman/var
log_dir: /var/log/mailman/mailman-logs
[database]
class: mailman.database.postgresql.PostgreSQLDatabase
url: postgres://mailman:<PASS>@127.0.0.1/mailman
debug: no
[archiver.hyperkitty]
class: mailman_hyperkitty.Archiver
enable: yes
configuration: /opt/mailman/mailman-suite/mailman-suite_project/mailman-hyperkitty.cfg
#[archiver.mail_archive]
#enable: yes
#[plugin.example]
#class: example.hooks.ExamplePlugin
#enabled: yes
#[archiver.prototype]
#enable: yes
[webservice]
# The hostname at which admin web service resources are exposed.
hostname: localhost
# The port at which the admin web service resources are exposed.
port: 8000
# Whether or not requests to the web service are secured through SSL.
use_https: no
# Whether or not to show tracebacks in an HTTP response for a request that
# raised an exception.
show_tracebacks: yes
# The API version number for the current (highest) API.
api_version: 3.1
# The administrative username.
admin_user: mailmanapi
# The administrative password.
admin_pass: <PASS>
workers: 4
configuration: /opt/mailman/gunicorn.cfg
#[language.master]
[language.de]
description: German
#charset: iso-8859-1
charset: utf-8
enabled: yes
[language.en]
description: English
charset: utf-8
enabled: yes
[mta]
incoming: mailman.mta.postfix.LMTP
outgoing: mailman.mta.deliver.deliver
lmtp_host: 127.0.0.1
lmtp_port: 8024
smtp_host: 127.0.0.1
smtp_port: 25
configuration: /opt/mailman/mailman-venv/lib/python3.6/site-packages/mailman/config/postfix.cfg
verp_probes: yes
[logging.root]
level: debug
path: mailman.log
[logging.archiver]
level: warn
path: archiver.log
[logging.bounce]
path: bounce.log
level: warn
[logging.config]
level: debug
path: mailman.log
[logging.database]
level: warn
path: database.log
[logging.debug]
path: debug.log
level: debug
[logging.error]
level: debug
path: error.log
[logging.fromusenet]
level: warn
path: mailman.log
[logging.http]
level: warn
path: httpd.log
[logging.locks]
level: warn
path: locks.log
[logging.mischief]
level: warn
path: mailman.log
[logging.plugins]
path: mailman.log
level: warn
[logging.runner]
level: debug
path: runner.log
[logging.smtp]
path: smtp.log
level: debug
[logging.subscribe]
path: subscribe.log
level: warn
[logging.vette]
path: vette.log
level: warn
# Some list posts and mail to the -owner address may contain DomainKey or
# DomainKeys Identified Mail (DKIM) signature headers <http://www.dkim.org/>.
# Various list transformations to the message such as adding a list header or
# footer or scrubbing attachments or even reply-to munging can break these
# signatures. It is generally felt that these signatures have value, even if
# broken and even if the outgoing message is resigned. However, some sites
# may wish to remove these headers by setting this to 'yes'.
remove_dkim_headers: yes
[bounces]
# How often should the bounce runner process queued detected bounces?
register_bounces_every: 15m
contenten of settings.py
# -*- coding: utf-8 -*-
# Copyright (C) 1998-2016 by the Free Software Foundation, Inc.
#
# This file is part of Mailman Suite.
#
# Mailman Suite is free sofware: you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by the Free
# Software Foundation, either version 3 of the License, or (at your option)
# any later version.
#
# Mailman Suite is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
# or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
# for more details.
# You should have received a copy of the GNU General Public License along
# with Mailman Suite. If not, see <http://www.gnu.org/licenses/>.
"""
Django Settings for Mailman Suite (hyperkitty + postorius)
For more information on this file, see
https://docs.djangoproject.com/en/1.8/topics/settings/
For the full list of settings and their values, see
https://docs.djangoproject.com/en/1.8/ref/settings/
"""
# Build paths inside the project like this: os.path.join(BASE_DIR, ...)
import os
BASE_DIR = os.path.dirname(os.path.abspath(__file__))
# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = 'hdeu37xh3291kf'
# SECURITY WARNING: don't run with debug turned on in production!
DEBUG = False
ADMINS = (
('Mailman Admin', 'mailman(a)lists.example.com'))
SITE_ID = 1
# Hosts/domain names that are valid for this site; required if DEBUG is False
# See https://docs.djangoproject.com/en/1.8/ref/settings/#allowed-hosts
ALLOWED_HOSTS = [
"lists.example.com", "example.com", "localhost", "127.0.0.1", "XXX.XXX.XXX.XXX", "YYY.YYY.YYY.YYY" # Archiving API from Mailman, keep it.
# "lists.your-domain.org",
# Add here all production URLs you may have.
]
# Mailman API credentials
MAILMAN_REST_API_URL = 'http://localhost:8000'
MAILMAN_REST_API_USER = 'mailmanapi'
MAILMAN_REST_API_PASS = '<PASS>'
MAILMAN_ARCHIVER_KEY = '<ARCHIVER_KEY>'
MAILMAN_ARCHIVER_FROM = ('XXX.XXX.XXX.XXX', 'localhost', '127.0.0.1', '::1')
# Application definition
INSTALLED_APPS = (
'hyperkitty',
'postorius',
'django_mailman3',
# Uncomment the next line to enable the admin:
'django.contrib.admin',
# Uncomment the next line to enable admin documentation:
# 'django.contrib.admindocs',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.sites',
'django.contrib.messages',
'django.contrib.staticfiles',
'rest_framework',
'django_gravatar',
'compressor',
'haystack',
'django_extensions',
'django_q',
'allauth',
'allauth.account',
'allauth.socialaccount',
#'django_mailman3.lib.auth.fedora',
#'allauth.socialaccount.providers.openid',
#'allauth.socialaccount.providers.github',
#'allauth.socialaccount.providers.gitlab',
#'allauth.socialaccount.providers.google',
## 'allauth.socialaccount.providers.facebook',
#'allauth.socialaccount.providers.twitter',
#'allauth.socialaccount.providers.stackexchange',
)
MIDDLEWARE = (
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.middleware.locale.LocaleMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware',
'django.middleware.security.SecurityMiddleware',
'django_mailman3.middleware.TimezoneMiddleware',
'postorius.middleware.PostoriusMiddleware',
)
ROOT_URLCONF = 'urls'
TEMPLATES = [
{
'BACKEND': 'django.template.backends.django.DjangoTemplates',
'DIRS': [],
'APP_DIRS': True,
'OPTIONS': {
'context_processors': [
'django.template.context_processors.debug',
'django.template.context_processors.i18n',
'django.template.context_processors.media',
'django.template.context_processors.static',
'django.template.context_processors.tz',
'django.template.context_processors.csrf',
'django.template.context_processors.request',
'django.contrib.auth.context_processors.auth',
'django.contrib.messages.context_processors.messages',
'django_mailman3.context_processors.common',
'hyperkitty.context_processors.common',
'postorius.context_processors.postorius',
],
},
},
]
WSGI_APPLICATION = 'wsgi.application'
# Database
# https://docs.djangoproject.com/en/1.8/ref/settings/#databases
DATABASES = {
#'default': {
# Use 'sqlite3', 'postgresql_psycopg2', 'mysql', 'sqlite3' or 'oracle'.
#'ENGINE': 'django.db.backends.sqlite3',
# DB name or path to database file if using sqlite3.
#'NAME': os.path.join(BASE_DIR, 'mailmansuite.db'),
## The following settings are not used with sqlite3:
#'USER': 'mailmansuite',
#'PASSWORD': 'mmpass',
# HOST: empty for localhost through domain sockets or '127.0.0.1' for
# localhost through TCP.
#'HOST': '',
# PORT: set to empty string for default.
#'PORT': '',
#}
# Example for PostgreSQL (recommanded for production):
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': 'mailman',
'USER': 'mailman',
'PASSWORD': '<PASS>',
'HOST': 'localhost',
}
}
# Maintain type of autogenerated keys going forward
# https://docs.djangoproject.com/en/3.2/releases/3.2/#customizing-type-of-aut…
DEFAULT_AUTO_FIELD = 'django.db.models.AutoField'
# If you're behind a proxy, use the X-Forwarded-Host header
# See https://docs.djangoproject.com/en/1.8/ref/settings/#use-x-forwarded-host
# USE_X_FORWARDED_HOST = True
# And if your proxy does your SSL encoding for you, set SECURE_PROXY_SSL_HEADER
# https://docs.djangoproject.com/en/1.8/ref/settings/#secure-proxy-ssl-header
# SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')
# SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_SCHEME', 'https')
# Other security settings
# SECURE_SSL_REDIRECT = True
# If you set SECURE_SSL_REDIRECT to True, make sure the SECURE_REDIRECT_EXEMPT
# contains at least this line:
# SECURE_REDIRECT_EXEMPT = [
# "archives/api/mailman/.*", # Request from Mailman.
# ]
# SESSION_COOKIE_SECURE = True
# SECURE_CONTENT_TYPE_NOSNIFF = True
# SECURE_BROWSER_XSS_FILTER = True
# CSRF_COOKIE_SECURE = True
# CSRF_COOKIE_HTTPONLY = True
# X_FRAME_OPTIONS = 'DENY'
# Password validation
# https://docs.djangoproject.com/en/1.9/ref/settings/#auth-password-validators
AUTH_PASSWORD_VALIDATORS = [
{
'NAME':
'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',
},
{
'NAME':
'django.contrib.auth.password_validation.MinimumLengthValidator',
},
{
'NAME':
'django.contrib.auth.password_validation.CommonPasswordValidator',
},
{
'NAME':
'django.contrib.auth.password_validation.NumericPasswordValidator',
},
]
# Internationalization
# https://docs.djangoproject.com/en/1.8/topics/i18n/
#LANGUAGE_CODE = 'en-us'
LANGUAGE_CODE = 'de-de'
#TIME_ZONE = 'UTC'
TIME_ZONE = 'Europe/Berlin'
USE_I18N = True
USE_L10N = True
USE_TZ = True
# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/1.8/howto/static-files/
# Absolute path to the directory static files should be collected to.
# Don't put anything in this directory yourself; store your static files
# in apps' "static/" subdirectories and in STATICFILES_DIRS.
# Example: "/var/www/example.com/static/"
STATIC_ROOT = os.path.join(BASE_DIR, 'static')
# URL prefix for static files.
# Example: "http://example.com/static/", "http://static.example.com/"
STATIC_URL = '/static/'
# Additional locations of static files
STATICFILES_DIRS = (
# Put strings here, like "/home/html/static" or "C:/www/django/static".
# Always use forward slashes, even on Windows.
# Don't forget to use absolute paths, not relative paths.
# BASE_DIR + '/static/',
)
# List of finder classes that know how to find static files in
# various locations.
STATICFILES_FINDERS = (
'django.contrib.staticfiles.finders.FileSystemFinder',
'django.contrib.staticfiles.finders.AppDirectoriesFinder',
# 'django.contrib.staticfiles.finders.DefaultStorageFinder',
'compressor.finders.CompressorFinder',
)
# Django 1.6+ defaults to a JSON serializer, but it won't work with
# django-openid, see
# https://bugs.launchpad.net/django-openid-auth/+bug/1252826
SESSION_SERIALIZER = 'django.contrib.sessions.serializers.PickleSerializer'
LOGIN_URL = 'account_login'
LOGIN_REDIRECT_URL = 'list_index'
LOGOUT_URL = 'account_logout'
# If you enable internal authentication, this is the address that the emails
# will appear to be coming from. Make sure you set a valid domain name,
# otherwise the emails may get rejected.
# https://docs.djangoproject.com/en/1.8/ref/settings/#default-from-email
# DEFAULT_FROM_EMAIL = "mailing-lists(a)you-domain.org"
DEFAULT_FROM_EMAIL = 'mailman(a)mailman.example.com'
# If you enable email reporting for error messages, this is where those emails
# will appear to be coming from. Make sure you set a valid domain name,
# otherwise the emails may get rejected.
# https://docs.djangoproject.com/en/1.8/ref/settings/#std:setting-SERVER_EMAIL
# SERVER_EMAIL = 'root(a)your-domain.org'
SERVER_EMAIL = 'mailman(a)mailman.example.com'
# Change this when you have a real email backend
EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend'
EMAIL_HOST = 'localhost'
EMAIL_PORT = 25
# Compatibility with Bootstrap 3
from django.contrib.messages import constants as messages # flake8: noqa
MESSAGE_TAGS = {
messages.ERROR: 'danger'
}
#
# Social auth
#
AUTHENTICATION_BACKENDS = (
'django.contrib.auth.backends.ModelBackend',
'allauth.account.auth_backends.AuthenticationBackend',
)
# Django Allauth
ACCOUNT_AUTHENTICATION_METHOD = "username_email"
ACCOUNT_EMAIL_REQUIRED = True
ACCOUNT_EMAIL_VERIFICATION = "mandatory"
# You probably want https in production, but this is a dev setup file
ACCOUNT_DEFAULT_HTTP_PROTOCOL = "https"
ACCOUNT_UNIQUE_EMAIL = True
#SOCIALACCOUNT_PROVIDERS = {
# 'openid': {
# 'SERVERS': [
# dict(id='yahoo',
# name='Yahoo',
# openid_url='http://me.yahoo.com'),
# ],
# },
# 'google': {
# 'SCOPE': ['profile', 'email'],
# 'AUTH_PARAMS': {'access_type': 'online'},
# },
# 'facebook': {
# 'METHOD': 'oauth2',
# 'SCOPE': ['email'],
# 'FIELDS': [
# 'email',
# 'name',
# 'first_name',
# 'last_name',
# 'locale',
# 'timezone',
# ],
# 'VERSION': 'v2.4',
# },
#}
#
# Gravatar
# https://github.com/twaddington/django-gravatar
#
# Gravatar base url.
# GRAVATAR_URL = 'http://cdn.libravatar.org/'
# Gravatar base secure https url.
# GRAVATAR_SECURE_URL = 'https://seccdn.libravatar.org/'
# Gravatar size in pixels.
# GRAVATAR_DEFAULT_SIZE = '80'
# An image url or one of the following: 'mm', 'identicon', 'monsterid',
# 'wavatar', 'retro'.
# GRAVATAR_DEFAULT_IMAGE = 'mm'
# One of the following: 'g', 'pg', 'r', 'x'.
# GRAVATAR_DEFAULT_RATING = 'g'
# True to use https by default, False for plain http.
# GRAVATAR_DEFAULT_SECURE = True
#
# django-compressor
# https://pypi.python.org/pypi/django_compressor
#
COMPRESS_PRECOMPILERS = (
('text/less', 'lessc {infile} {outfile}'),
('text/x-scss', 'sassc -t compressed {infile} {outfile}'),
('text/x-sass', 'sassc -t compressed {infile} {outfile}'),
)
# On a production setup, setting COMPRESS_OFFLINE to True will bring a
# significant performance improvement, as CSS files will not need to be
# recompiled on each requests. It means running an additional "compress"
# management command after each code upgrade.
# http://django-compressor.readthedocs.io/en/latest/usage/#offline-compression
# COMPRESS_OFFLINE = True
# Needed for debug mode
# INTERNAL_IPS = ('127.0.0.1',)
#
# Full-text search engine
#
HAYSTACK_CONNECTIONS = {
'default': {
'ENGINE': 'haystack.backends.whoosh_backend.WhooshEngine',
'PATH': os.path.join(BASE_DIR, "fulltext_index"),
# You can also use the Xapian engine, it's faster and more accurate,
# but requires another library.
# http://django-haystack.readthedocs.io/en/v2.4.1/installing_search_engines.h…
# Example configuration for Xapian:
#'ENGINE': 'xapian_backend.XapianEngine'
},
}
#
# Asynchronous tasks
#
Q_CLUSTER = {
'timeout': 300,
'save_limit': 100,
'orm': 'default',
'retry': 360,
}
# A sample logging configuration. The only tangible logging
# performed by this configuration is to send an email to
# the site admins on every HTTP 500 error when DEBUG=False.
# See http://docs.djangoproject.com/en/dev/topics/logging for
# more details on how to customize your logging configuration.
LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'filters': {
'require_debug_false': {
'()': 'django.utils.log.RequireDebugFalse'
}
},
'handlers': {
'mail_admins': {
'level': 'INFO',
'filters': ['require_debug_false'],
'class': 'django.utils.log.AdminEmailHandler'
},
'file':{
'level': 'WARN',
#'class': 'logging.handlers.RotatingFileHandler',
'class': 'logging.handlers.WatchedFileHandler',
'filename': os.path.join(BASE_DIR, 'logs', 'mailmansuite.log'),
'formatter': 'verbose',
},
'console': {
'class': 'logging.StreamHandler',
'formatter': 'simple',
},
},
'loggers': {
'django.request': {
'handlers': ['mail_admins', 'file'],
'level': 'DEBUG',
'propagate': True,
},
'django': {
'handlers': ['file'],
'level': 'INFO',
'propagate': True,
},
'hyperkitty': {
'handlers': ['file'],
'level': 'DEBUG',
'propagate': True,
},
'postorius': {
'handlers': ['console', 'file'],
'level': 'INFO',
},
},
'formatters': {
'verbose': {
'format': '%(levelname)s %(asctime)s %(process)d %(name)s %(message)s'
},
'simple': {
'format': '%(levelname)s %(message)s'
},
},
#'root': {
# 'handlers': ['file'],
# 'level': 'INFO',
#},
}
# Using the cache infrastructure can significantly improve performance on a
# production setup. This is an example with a local Memcached server.
#CACHES = {
# 'default': {
# 'BACKEND': 'django.core.cache.backends.memcached.PyLibMCCache',
# 'LOCATION': '127.0.0.1:11211',
# }
#}
# When DEBUG is True, don't actually send emails to the SMTP server, just store
# them in a directory. This way you won't accidentally spam your mailing-lists
# while you're fiddling with the code.
#if DEBUG == False:
# EMAIL_BACKEND = 'django.core.mail.backends.filebased.EmailBackend'
# EMAIL_FILE_PATH = os.path.join(BASE_DIR, 'emails')
#
# HyperKitty-specific
#
# Only display mailing-lists from the same virtual host as the webserver
FILTER_VHOST = False
POSTORIUS_TEMPLATE_BASE_URL = 'https://lists.example.com'
try:
from settings_local import *
except ImportError:
pass
> Are all the relevant components running on the same host? Are
> there other web services running under Apache and/or WSGI?
Nope, the virtual machine is dedicated to Mailman3. Mailman3 as venv, uwsgi as Webservice, Apache as Webserver, Postfix as Mailserver and PostgreSQL 10 als Database, all on the same host
> What are the relevant configurations (Apache, uswgi, gunicorn,
> Mailman, Postorius, Django, database, DNS A/AAAA/CNAME)? This
> would be anything like port numbers and host names or IP
> addresses, user names and passwords (we don't need to know these,
> but you should provide obfuscated versions so we can check syntax,
> and of course you should check for typos).
>
> Steve
Should be provided on the top of the mail.
At moment i think its a slow database. Because of the slow database, the Mailman API is slow and so on. We are trying now to solve the database part with postgresqltuner - maybe we will see more details then.
Stephan
4 years, 9 months
Re: My list has been switched from public to private archives, but they are still visible
by Enrique Terrazas
Thank you Abhilash,
I appreciate your prompt response. What about the once private archives becoming public after setting a list to “Do not archive this list”?
Regards,
Enrique
On Sep 19, 2019, at 1:38 PM, Abhilash Raj <maxking(a)asynchronous.in<mailto:maxking@asynchronous.in>> wrote:
On Thu, Sep 19, 2019, at 11:15 AM, Enrique Terrazas wrote:
I am having a very similar issue. Here is what I have found:
There are basically three archive settings we can select via the
postorious web front end for archives. These can be found in: Settings
Archiving and they are:
Public archives
Private archives
Do not archive this list
This is interface of Mailman to set the archives.
In the Django administrative interface we can find those
same(?/similar?) settings here: Main Page > Hyperkitty > Mailing lists
Mailing List object(x) > Archive policy:
never
private
public
This is basically a hotline to the database. You can change these values,
but they would be overriden with the values that are in Mailman Core on
the daily cron job run.
I would suggest not to tinker with values there since you could leave
database in a corrupted state. Since there is an ORM layer there, chances
are less but still a possibility :)
I’m not sure which of the two takes precedence. My guess is postorius
since that is a list owner accessible setting and the Django
administrative interface would typically not be accessible to list
owners. We have noticed the following odd behavior, not sure if it is a
bug. For existing lists with private archives, when switching to “Do
not archive this list” in postorious, the desired effect of no longer
archiving the list is instantaneous. However, after the daily cron job
that syncs with Mailman Core happens, the existing archives now become
publicly accessible. Is this behavior based on the assumption that the
list will no longer be archived and will therefore be purged?
Source of truth is Mailman Core's database. Postorius is a front-end to
change that and owners have permissions to do that.
Hyperkitty only syncs back that information daily. I have a fix in the
master which makes such that this information is synced synchronously
and is available immediately to Hyperkitty.
For now, you could either run the "python manage.py runjobs --daily" or
wait for a day if that is possible.
Enrique
On Sep 19, 2019, at 1:08 PM, Abhilash Raj <maxking(a)asynchronous.in<mailto:maxking@asynchronous.in>> wrote:
On Thu, Sep 19, 2019, at 3:31 AM, Paul Arenson wrote:
Can anyone help with this?
I set my list to private archives today. Up until then, it was pubic.
And yet, if you go to the list page, you will see that if you click
archives (top left), you can see them. (Filled with a bunch of tests
posts for now)
This change takes a bit of time to sync up Hyperkitty. THere is a daily
cron job which syncs this information from Mailman Core.
I have made a change which should fix this problem and the process would
become synchronous.
You can run `python manage.py runjobs --daily` command to force sync
that information.
https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Flist.toky…
Am I misunderstanding what a private archive is?
Not really, your expectation is right.
Is something wrong with my setup (which is hosted)?
Thank you.
_______________________________________________
Mailman-users mailing list -- mailman-users(a)mailman3.org<mailto:mailman-users@mailman3.org>
To unsubscribe send an email to mailman-users-leave(a)mailman3.org<mailto:mailman-users-leave@mailman3.org>
https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.mai…
--
thanks,
Abhilash Raj (maxking)
_______________________________________________
Mailman-users mailing list -- mailman-users(a)mailman3.org<mailto:mailman-users@mailman3.org>
To unsubscribe send an email to mailman-users-leave(a)mailman3.org<mailto:mailman-users-leave@mailman3.org>
https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.mai…
--
thanks,
Abhilash Raj (maxking)
_______________________________________________
Mailman-users mailing list -- mailman-users(a)mailman3.org<mailto:mailman-users@mailman3.org>
To unsubscribe send an email to mailman-users-leave(a)mailman3.org<mailto:mailman-users-leave@mailman3.org>
https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.mai…
7 years, 1 month
Re: Held messages page not rendering the message
by Odhiambo Washington
On Sun, Mar 19, 2023 at 1:12 PM <mike-mailman(a)earth.li> wrote:
> Thanks - may be on to something.
>
> The sources tab shows everything ok except for
> /mailman3/static/postorius/js/script.js for which it says "Unable to fetch
> script source"
> However the network tab shows it successfully retrieving it.
>
> /var/lib/mailman3/web/static/postorius/js/script.js exists and contains
> only comments
>
Is anything showing up in your webserver logs that is related to this? Just
maybe.
I have tested on my setup (although I use a virtualenv setup) and I see as
a click on a held message corresponds to a log entry in my webserver log.
I do get a pop-up like this:
```
https://imgur.com/a/esxmDsK
```
And that is the same across all my browsers. My Desktop is running
Winblows11 though.
--
Best regards,
Odhiambo WASHINGTON,
Nairobi,KE
+254 7 3200 0004/+254 7 2274 3223
"Oh, the cruft.", egrep -v '^$|^.*#' ¯\_(ツ)_/¯ :-)
3 years, 7 months
Re: Issues with @gmail.com addresses
by Allan Hansen
Once this gets handled, Mohsen, Google has another issue with list servers:
If a subscriber who uses GMail posts a message to a list, then that subscriber will not get his/her own message from the server - Google eats it. All other subscribers will get it, but not the author. My subscribers run into this all the time and are up in arms over it. I tell them to complain to Google.
Yours,
Allan
On 9/5/23, 13:18, "Mark Sapiro" <mark(a)msapiro.net <mailto:mark@msapiro.net>> wrote:
On 9/5/23 12:59, Mohsen Masoudfar wrote:
> Hi,
>
> since 8/1/2023 I get complains from users with @gmail.com addresses, that they are not getting any emails.
> I checked the logs and I see that the emails are going out:
>
> to=<xxxx(a)gmail.com <mailto:xxxx@gmail.com>>, relay=email-smtp.us-east-1.amazonaws.com[35.169.171.20]:587, delay=0.46, delays=0.03/0.06/0.13/0.24, dsn=2.0.0, status=sent (250 Ok xxx )
>
> But it seems the users are getting email delivered to them.
> I’ve advised them to check their spam, promotions, and social folders and to add the list address to their email safe-senders lists, but still no success.
> Is there any way to help this list members?
Even though gmail publishes a DMARC policy of none, it has recently
started enforcing a stricter policy for mail From: the gmail.com domain.
You need to apply DMARC mitigations to mail From: the gmail.com domain.
There are two ways to do this.
One way is to set DMARC Mitigations -> DMARC Mitigate unconditionally to
Yes (and set DMARC mitigation action appropriately)
The other way requires upgrading Postorius and Mailman core to the heads
of the GitLab branches (core=3.3.9b1, postorius=1.3.9b1). Doing this
enables a DMARC Addresses setting which can be set to `^.*(a)gmail\.com$`
to apply DMARC mitigatations to mail From: the gmail.com domain
regardless of it's published policy.
--
Mark Sapiro <mark(a)msapiro.net <mailto:mark@msapiro.net>> The highway is for gamblers,
San Francisco Bay Area, California better use your sense - B. Dylan
_______________________________________________
Mailman-users mailing list -- mailman-users(a)mailman3.org <mailto:mailman-users@mailman3.org>
To unsubscribe send an email to mailman-users-leave(a)mailman3.org <mailto:mailman-users-leave@mailman3.org>
https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/ <https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/>
Archived at: https://lists.mailman3.org/archives/list/mailman-users@mailman3.org <mailto:mailman-users@mailman3.org>/message/WZY5IC3OVHVBHIGDT4TE3EWSSGMF6CYI/
This message sent to hansen(a)rc.org <mailto:hansen@rc.org>
3 years, 2 months
including a template failed
by Wolfgang Bock
Hi,
I created a new mailinglist and got during the test-process the following
errormessage in my mailman.log after:
...
Sep 19 22:55:08 2020 (1433) ACCEPT:
<000001d68ec7$2553e6e0$6ffbb4a0$(a)domain-01.de>
Errors:
Sep 19 22:55:09 2020 (1436) Cannot connect to SMTP server localhost on port
25
HTTPSConnectionPool(host='webmail.domain-02.de', port=443): Max retries
exceeded with url:
/mailman3/postorius/api/templates/list/mylist.lists.domain-03.de/list:member
:regular:footer (Caused by NewConnectionError('<urllib3.connec
tion.VerifiedHTTPSConnection object at 0x7fc8380f1748>: Failed to establish
a new connection: [Errno -2] Name or service not known'))
...
domain02.de isn't involved in mailman3 either as a user domain nor as a
list-domain.
After deleting the template
mailman3/postorius/api/templates/list/mylist.lists.domain-03.de/list:member:
regular:footer the mail to the list is processed without problems:
Sep 19 22:56:22 2020 (1436) <000001d68ec7$2553e6e0$6ffbb4a0$(a)domain-01.de>
smtp to mylist(a)list.domain-03.de for 1 recips, completed in
0.21389174461364746 seconds
Sep 19 22:56:22 2020 (1436) <000001d68ec7$2553e6e0$6ffbb4a0$(a)wbock.de> post
to mylist(a)list.domain-03.de from user(a)domain-01.de, 1331 bytes
Please help to solve this problem?
Thanks and regards
Wolfgang
6 years, 1 month
Bulk changes to delivery mode and moderation action
by David Newman
I fat-fingered a command in the Postorius UI and accidentally
unsubcribed all members of a list. Oops.
Have added them back using Mass operations/Mass subscribe but this
raised three questions:
1. In Users/Members, moderation action for the newly resubscribed users
shows as "None". Is there a method, either in the UI or the command
line, to change this to "Default processing" for all users?
2. Although Delivery Mode shows as "Regular" for all users in
Users/Members list, clicking on a member's name shows all options such
as delivery status, delivery mode, etc., as undefined.
Similar question: How to set subscriber options for all members in one go?
3. As a subscriber to this list, I received email notification that I'd
been unsubscribed. Where is the control for enabling/disabling
subscription notifications?
Settings/Member Policy/Un-subscription Policy is set to Confirm, but I
don't think that's relevant here since I unsubscribed users as the
Django superuser.
Thanks.
dn, chastened
4 years, 9 months
Re: Accept these non-members
by Lars Bjørndal
On Mon, Mar 09, 2026 at 02:40:29AM +0900, Stephen J. Turnbull wrote:
> Lars Bjørndal writes:
>
> > All addresses, both addresses that I know has sent messages to the
> > list and others, are marked as MemberRole.nonmember> Action.defer
>
> Post from these nonmembers go through normal checks (spam, for
> example) before being distributed.
>
> > When I view non-members in postorius, I get some differences:
> >
> > Some ar marked: Regular List default, and others Regular Default
> > processing.
>
> "List default" means "this user has not made a choice, so Mailman will
> check what is 'normal' for the list in the list's configuration".
>
> In detail, Postorius is intended to allow users and list
> administrators to *modify* configurations. At delivery time, these
> configurations "cascade", typically in this way:
>
> - if this subscription has a setting, apply it, and return; if not
> - if the address this subscription delivers to has a setting, apply
> it, and return; if not
> - if the user who owns that address has a setting, apply it, and
> return; if not
> - if the list that is subscribed to has a setting, apply it, and
> return;
> - otherwise, use the fallback setting from the source code.
>
> In Postorius, the "no setting" option is displayed in such a way that
> the person considering changing the setting knows what the next level
> of the cascade is. "List default" means "this user has not made a
> choice, so Mailman will check what is 'normal' for the list in the
> list's configuration".
>
> "Default processing" is a regular option, that can be applied to this
> post. It means "check if the user is allowed to post at all, then
> check for reasons (eg, spam) why this post might not be OK". Usually
> "list normal" is also "Default processing", but it might be "Hold for
> moderator", for example.
>
> On the other hand, Mark's script will return the option that will be
> applied using the logic of the cascade.
Which script?
> > The list is imported from Mm 2.1 a long time ago. I don't know if
> > that matters.
>
> I don't think it does.
>
> > Hm, this is a mystery to me. Is there othere places I can search
> > for addresses that are aloud to send to the list?
>
> I don't understand what you want. The "mailman shell interaction"
> that Mark posted earlier gives you exactly the list of members who can
> post to the specified list. If some are "trusted users" who don't go
> through the spam checker, etc, they are displayed with Action.accept.
> Other posters are displayed with Action.defer. But both will
> automatically be distributed as long as they appear to be legit posts
> (not spam, etc).
As menitoned in my first post, this is an announce-only list.
In the Settings for this list, under Message Acceptance, the Default action to take when a member posts to the list is set to (X) Reject (with notification). The same goes for non-members. In the same Tab, Accept these non-members edit field is empty. Because one of the subscribed addresses to this list is another mailing list, all messages goes both to this announce-only list and to the subscribed list. So to the announce list, there are a few non-members that are allowed to post, e.g. bypass the default actions, which is set to Reject.
Hope this clarify my situation a bit. I don't see any place where to find the list of allowed addresses to post.
Thanks,
Lars
7 months, 4 weeks
Re: Mailman Suite installation on CentOS 7
by Mark Sapiro
On 11/8/21 4:18 AM, Kimmo L wrote:
> Hello!
>
> I want to install fresh Mailman, which is combined with hyperkitty+postorius. After checking documentation i found, that Mailman Suite is perfect for me.
> But i have some misunderstandings about installation. I only found this part in documentation, which explains how to clone and start using Mailman Suite (https://docs.mailman3.org/en/latest/devsetup.html#run-the-mailman-suite-com…)
>
> But am i correct, that before doing so, i need to install mailman core, mailman client, django-mailman3 ?
> Or what would be the correct steps to build up my first Mailman Suite on CentOS?
The docs you want to look at are
https://docs.mailman3.org/en/latest/#the-pre-installation-guide and
https://docs.mailman3.org/en/latest/install/install.html.
There is also information at https://wiki.list.org/x/17891998 and in
docs linked from thefirst paragraph thereof.
--
Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers,
San Francisco Bay Area, California better use your sense - B. Dylan
4 years, 11 months
Re: Removing the footer from mailing list messages
by Mark Sapiro
On 12/7/25 14:00, MegaBrutal via Mailman-users wrote:
>
> What I already tried: define a custom template for
> list:member:regular:footer. The result was that the mailing list stopped
> operating altogether, no messages passed through until I deleted this
> template. After I deleted it, all delayed e-mails were forwarded at once.
How did you create the footer? If you created a file at Mailman's
var/templates/whatever/list:member:regular:footer.txt, it probably was
not readable by the Mailman user.
If you created it via Postorius there is probably and issue with your
setting for POSTORIUS_TEMPLATE_BASE_URL.
> What would be the right way to remove the footer?
For all lists
sudo -u mailman cp /dev/null
var/templates/site/en/list:member:regular:footer.txt
For a single list
sudo -u mailman cp /dev/null
var/templates/lists/LIST-ID/en/list:member:regular:footer.txt
--
Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers,
San Francisco Bay Area, California better use your sense - B. Dylan
10 months, 4 weeks
Re: New year releases!
by Torge Riedel
Dears,
I like to know and discuss ideas to get the update process more stable!
Today I updated core and web of mailman3 and ran into all the issues you can have:
- Dependencies are not updated properly
- In case you use "--force-reinstall" Django gets also updated in web and you need to update the urls.py file accordingly
- Enhancement of settings file required (DEFAULT_AUTO_FIELD = ...)
Finally I got it working, but there is always the fear for breaking something on update.
I know it is open source and no one gets paid for the work. But I think there is space for improvement here.
Kind regards
Torge
Am 04.01.23 um 10:59 schrieb Abhilash Raj:
> Hello Everyone,
>
> Happy new year!
>
> I am pleased to announce several new releases for GNU Mailman project.
>
> 1. Mailman Core 3.3.8 [1]
> 2. Postorius 1.3.8 [2]
> 3. Hyperkitty 1.3.7 [3]
> 4. Mailmanclient 3.3.5 [4]
> 5. Django-mailman3 1.3.9 [5]
> 6. Mailman-hyperkitty 1.2.1 [6]
>
> All the releases are published to PyPI and you can find links to all the PyPI projects:
>
> https://docs.mailman3.org/en/latest/
>
> A full list of change logs are available below:
>
> [1]: https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/docs/NEWS.…
> [2]: https://docs.mailman3.org/projects/postorius/en/latest/news.html#news-1-3-8
> [3]: https://docs.mailman3.org/projects/hyperkitty/en/latest/news.html#news-1-3-7
> [4]: https://docs.mailman3.org/projects/mailmanclient/en/latest/src/mailmanclien…
> [5]: https://gitlab.com/mailman/django-mailman3/-/blob/master/README.rst
> [6]: https://gitlab.com/mailman/mailman-hyperkitty/-/blob/master/README.rst
>
> Finally, the minimum Python version required is 3.7, and all the components now support Python 3.7 to 3.11.
>
> Thanks to all the contributors for making this release possible!
>
> If you find any issues, you can report them at the respective bug tracker for the projects. The links to issue trackers are available
> here: https://docs.mailman3.org/en/latest/
>
> Installation documentation is available at
>
> https://docs.mailman3.org/en/latest/install/install.html
>
> and if you are upgrading an existing virtualenv install you can refer
>
> https://docs.mailman3.org/en/latest/upgrade-3.2.html
>
>
>
> _______________________________________________
> Mailman-users mailing list -- mailman-users(a)mailman3.org
> To unsubscribe send an email to mailman-users-leave(a)mailman3.org
> https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
> Archived at: https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/message…
>
> This message sent to torgeriedel(a)gmx.de
3 years, 8 months